LiteLLM

Every provider brings its own SDK, auth pattern, request format and error types, so code that talks to several models fills up with special cases. LiteLLM collapses all of it into one interface in the OpenAI format, covering more than 100 providers.

It works two ways. As a Python SDK you import a completion function and switch models by changing a string, and as a proxy server you run a gateway the whole team points at, using any OpenAI client with a different base address.

The gateway is where the operational features live.

• Virtual keys: issue per-team credentials without handing out the underlying provider keys.
• Spend tracking: attribute cost across keys and models from the admin dashboard.
• Load balancing: spread traffic across deployments and fall back when a provider fails.
• Guardrails: apply policy checks to requests as they pass through the gateway.
• Broad endpoints: chat completions, responses, embeddings, images, audio, batches and rerank.

Beyond model calls it invokes A2A agents from LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore and Pydantic AI. The project reports 8ms P95 latency at 1k requests per second, publishes deploy templates for Render, Railway, AWS and Google Cloud, and counts Netflix among its adopters.