Open Source OpenRouter Alternatives

Every provider brings its own SDK, auth pattern, request format and error types, so code that talks to several models fills up with special cases. LiteLLM collapses all of it into one interface in the OpenAI format, covering more than 100 providers.

It works two ways. As a Python SDK you import a completion function and switch models by changing a string, and as a proxy server you run a gateway the whole team points at, using any OpenAI client with a different base address.

The gateway is where the operational features live.

• Virtual keys: issue per-team credentials without handing out the underlying provider keys.
• Spend tracking: attribute cost across keys and models from the admin dashboard.
• Load balancing: spread traffic across deployments and fall back when a provider fails.
• Guardrails: apply policy checks to requests as they pass through the gateway.
• Broad endpoints: chat completions, responses, embeddings, images, audio, batches and rerank.

Beyond model calls it invokes A2A agents from LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore and Pydantic AI. The project reports 8ms P95 latency at 1k requests per second, publishes deploy templates for Render, Railway, AWS and Google Cloud, and counts Netflix among its adopters.

LocalAI runs models on hardware you already own and speaks the APIs your code already calls. Text, vision, voice, image and video all sit behind a single endpoint, and a GPU is optional rather than assumed, which removes the usual reason for sending private data to somebody else's server.

The design is a small core rather than a bundle. Each backend wraps an established engine such as llama.cpp, vLLM, whisper.cpp, stable-diffusion or MLX in its own image, pulled only when a model needs it, so you install nothing you do not use. LocalAI detects your GPU capability and fetches the matching backend on its own.

Several things follow from that structure.

• Drop-in API compatibility: the OpenAI, Anthropic and ElevenLabs APIs are served across every backend.
• Any hardware: NVIDIA, AMD, Intel, Apple Silicon, Vulkan and CPU-only each have their own container tag.
• Model sources: run one from the gallery, Hugging Face, the Ollama registry, a YAML config or any OCI registry.
• Multi-user ready: API key authentication, per-user quotas and role-based access come built in.
• Built-in agents: autonomous agents with tool use, RAG, MCP and skills ship with the engine.
• Terminal agent: a chat client reads your files and runs commands, asking approval before anything changes state.

macOS has a disk image download, unsigned by Apple, that needs a quarantine attribute cleared after install; everywhere else a single docker run starts the server.

The AI Gateway routes requests to more than sixteen hundred language, vision, audio and image models behind one API, so changing provider stops being a code change. It is small, around 122kb, adds under a millisecond of latency, and the project reports more than ten billion tokens passing through it every day.

Running it locally is a single npx command, after which the gateway serves on a local port and a console shows every request that went through. Calls can come from the Portkey client, the OpenAI SDKs, or framework integrations for LangChain, LlamaIndex, Autogen and CrewAI, all using the familiar OpenAI signature.

Behavior is set with configs, small objects attached to a client.

• Reliable routing: fallbacks to another provider or model on chosen errors, up to five automatic retries with exponential backoff, weighted load balancing and granular timeouts.
• Guardrails: more than forty prebuilt checks on inputs and outputs, with room to bring your own or use partner checks.
• Cost management: simple and semantic caching, plus analytics covering request volume, latency, cost and error rates.
• MCP Gateway: one authentication layer, access control and full logging for Model Context Protocol servers across an organization.

Deployment guides cover Docker, Node.js, Cloudflare Workers and Replit as well as the hosted service, while the enterprise build adds organization management, governance and private deployments on AWS, Azure, GCP, OpenShift and Kubernetes. Gateway 2.0, which merges the enterprise core into the open source project, is available as a pre-release branch.

Bifrost places a single API in front of every model provider an application might reach for. More than 23 of them, including OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Cohere, Mistral, Groq, and Ollama, sit behind one OpenAI-compatible endpoint, so changing provider stops being a code change.

Starting it takes one command through npx or Docker, after which a built-in web interface handles configuration, live monitoring, and analytics. Providers can also be set up dynamically, through the API, or from files, and existing SDK code usually needs one line changed to point at the gateway.

The gateway layer is where the reliability and cost work happens.

• Automatic failover: requests move between providers and models when one of them fails.
• Load balancing: traffic is distributed across multiple API keys and providers.
• Semantic caching: responses are reused when a new request is close enough in meaning, cutting cost and latency.
• Governance: usage tracking, rate limiting, virtual keys, and budgets scoped to teams and customers.
• Observability: native Prometheus metrics, distributed tracing, and request logging.
• MCP tools: models call external tools such as filesystem access, web search, and databases.

The codebase is modular, separating core provider implementations, a framework layer for configuration, log, and vector storage, HTTP transports, and the web UI. Text, images, audio, and streaming all pass through the same interface. Private deployments add clustering, guardrails, an MCP gateway, and adaptive load balancing for teams running at scale.

LLM Gateway sits between your applications and the model providers they call, so adding or swapping a provider stops being an application change. It routes to OpenAI, Anthropic, Google Vertex AI and others, holds every provider key in one place, and records what each call consumed and cost.

Because the gateway speaks the OpenAI API format, existing clients migrate by pointing at a different base address rather than rewriting request code. The repository is a monorepo holding a Next.js dashboard, a Hono backend, the gateway service itself, a documentation site, a Drizzle ORM schema with migrations, and shared model and provider definitions.

The parts a team actually uses break down as follows.

• Unified interface: one request and response shape works across every connected provider.
• Key management: provider credentials live in a single place instead of scattered through services.
• Usage analytics: requests, tokens, response times, and costs are tracked across all interactions.
• Performance monitoring: models are compared on speed and cost-effectiveness before you commit to one.
• Self-hosting: a unified Docker image bundles Postgres and Redis and starts from two generated secrets.

Self-hosters should use Docker-managed volumes rather than bind-mounting a host directory to the Postgres data path, since initialization inside the container needs to set permissions there and can fail depending on host filesystem and ownership. A hosted account is available for teams who want an API key immediately. The core is AGPLv3, while commercial features in the enterprise directory need a separate license.