Open Source Cloudflare AI Gateway Alternatives

Every provider brings its own SDK, auth pattern, request format and error types, so code that talks to several models fills up with special cases. LiteLLM collapses all of it into one interface in the OpenAI format, covering more than 100 providers.

It works two ways. As a Python SDK you import a completion function and switch models by changing a string, and as a proxy server you run a gateway the whole team points at, using any OpenAI client with a different base address.

The gateway is where the operational features live.

• Virtual keys: issue per-team credentials without handing out the underlying provider keys.
• Spend tracking: attribute cost across keys and models from the admin dashboard.
• Load balancing: spread traffic across deployments and fall back when a provider fails.
• Guardrails: apply policy checks to requests as they pass through the gateway.
• Broad endpoints: chat completions, responses, embeddings, images, audio, batches and rerank.

Beyond model calls it invokes A2A agents from LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore and Pydantic AI. The project reports 8ms P95 latency at 1k requests per second, publishes deploy templates for Render, Railway, AWS and Google Cloud, and counts Netflix among its adopters.

The AI Gateway routes requests to more than sixteen hundred language, vision, audio and image models behind one API, so changing provider stops being a code change. It is small, around 122kb, adds under a millisecond of latency, and the project reports more than ten billion tokens passing through it every day.

Running it locally is a single npx command, after which the gateway serves on a local port and a console shows every request that went through. Calls can come from the Portkey client, the OpenAI SDKs, or framework integrations for LangChain, LlamaIndex, Autogen and CrewAI, all using the familiar OpenAI signature.

Behavior is set with configs, small objects attached to a client.

• Reliable routing: fallbacks to another provider or model on chosen errors, up to five automatic retries with exponential backoff, weighted load balancing and granular timeouts.
• Guardrails: more than forty prebuilt checks on inputs and outputs, with room to bring your own or use partner checks.
• Cost management: simple and semantic caching, plus analytics covering request volume, latency, cost and error rates.
• MCP Gateway: one authentication layer, access control and full logging for Model Context Protocol servers across an organization.

Deployment guides cover Docker, Node.js, Cloudflare Workers and Replit as well as the hosted service, while the enterprise build adds organization management, governance and private deployments on AWS, Azure, GCP, OpenShift and Kubernetes. Gateway 2.0, which merges the enterprise core into the open source project, is available as a pre-release branch.

Bifrost places a single API in front of every model provider an application might reach for. More than 23 of them, including OpenAI, Anthropic, AWS Bedrock, Google Vertex, Azure, Cohere, Mistral, Groq, and Ollama, sit behind one OpenAI-compatible endpoint, so changing provider stops being a code change.

Starting it takes one command through npx or Docker, after which a built-in web interface handles configuration, live monitoring, and analytics. Providers can also be set up dynamically, through the API, or from files, and existing SDK code usually needs one line changed to point at the gateway.

The gateway layer is where the reliability and cost work happens.

• Automatic failover: requests move between providers and models when one of them fails.
• Load balancing: traffic is distributed across multiple API keys and providers.
• Semantic caching: responses are reused when a new request is close enough in meaning, cutting cost and latency.
• Governance: usage tracking, rate limiting, virtual keys, and budgets scoped to teams and customers.
• Observability: native Prometheus metrics, distributed tracing, and request logging.
• MCP tools: models call external tools such as filesystem access, web search, and databases.

The codebase is modular, separating core provider implementations, a framework layer for configuration, log, and vector storage, HTTP transports, and the web UI. Text, images, audio, and streaming all pass through the same interface. Private deployments add clustering, guardrails, an MCP gateway, and adaptive load balancing for teams running at scale.

Envoy AI Gateway handles request traffic from application clients to generative AI services, using Envoy Gateway as its foundation. It gives an organization one place to put authentication, routing and rate limiting for model calls, instead of scattering provider keys through every service that wants to talk to a language model.

The recommended shape is a two-tier gateway. The tier one gateway is the centralized entry point, handling authentication, top-level routing and global rate limiting. The tier two gateway sits in front of a self-hosted model serving cluster and gives fine-grained control over access to those models.

A few pieces define how the project is used.

• Two-tier pattern: a central entry gateway in front of gateways guarding self-hosted model clusters.
• Endpoint picker: routing support aimed at optimizing inference on those self-hosted clusters.
• Hosted providers: OpenAI, Azure OpenAI, Google Gemini, Vertex AI, AWS Bedrock, Anthropic and Mistral.
• Independent model hosts: Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok and the Tetrate Agent Router Service.
• Quickstart: a getting-started guide that brings the gateway up in a few steps, with a concepts section covering the architecture and its resources.

The project grew out of a proposal to use Envoy Gateway as a cloud native gateway for large language models. It follows the CNCF code of conduct, holds a community meeting every Monday with a public agenda, and coordinates in a dedicated channel of the Envoy Slack workspace, so contributions arrive through the usual pull request path.

LLM Gateway sits between your applications and the model providers they call, so adding or swapping a provider stops being an application change. It routes to OpenAI, Anthropic, Google Vertex AI and others, holds every provider key in one place, and records what each call consumed and cost.

Because the gateway speaks the OpenAI API format, existing clients migrate by pointing at a different base address rather than rewriting request code. The repository is a monorepo holding a Next.js dashboard, a Hono backend, the gateway service itself, a documentation site, a Drizzle ORM schema with migrations, and shared model and provider definitions.

The parts a team actually uses break down as follows.

• Unified interface: one request and response shape works across every connected provider.
• Key management: provider credentials live in a single place instead of scattered through services.
• Usage analytics: requests, tokens, response times, and costs are tracked across all interactions.
• Performance monitoring: models are compared on speed and cost-effectiveness before you commit to one.
• Self-hosting: a unified Docker image bundles Postgres and Redis and starts from two generated secrets.

Self-hosters should use Docker-managed volumes rather than bind-mounting a host directory to the Postgres data path, since initialization inside the container needs to set permissions there and can fail depending on host filesystem and ownership. A hosted account is available for teams who want an API key immediately. The core is AGPLv3, while commercial features in the enterprise directory need a separate license.

Grepture Proxy stands between an application and an AI provider and cleans the traffic on the way through. It detects and redacts personal data, blocks content you do not want leaving the building, and swaps sensitive JSON fields for tokens, all before the request reaches the model.

Each request carries a target header naming the upstream API, so one proxy can front several providers. It authenticates the caller, applies input rules, forwards the request, applies output rules to the reply, restores anything it tokenized and returns the result. Buffered and streaming responses both work, and token restoration spans streamed chunks. Rules sit in a JSON file that reloads when it changes or on SIGHUP, each with conditions, a target of input, output or both, and a sampling rate.

Actions are the part you configure.

• Redact PII: regex patterns catch email addresses, phone numbers, SSNs, credit cards, IP addresses, postal addresses and dates of birth.
• Find and replace: literal or regex substitution rewrites matching text.
• Tokenize: JSON fields become tokens, with the originals kept for restoration on the way back.
• Block request: the call stops with a status code and message you choose.
• Log only: the request is tagged for logging and passes through unchanged.

The proxy runs on Bun, listens on port 4001 and builds into a Docker image with the rules file mounted in. An optional API key restricts who may use it, and bodies over 50KB offload to S3-compatible storage during the log flush, after the client has its response.

Shipping an AI feature in an iOS app usually means shipping the provider key inside the binary, where anyone willing to look can take it. Proxed.AI puts a backend in front of that call: the key stays on the server, requests are checked before they are forwarded, and model responses come back in a shape the app can rely on.

The project is a TypeScript monorepo running on Bun, with React and Next.js for the landing page and portal, Hono for the API, Supabase for authentication, database and storage, and shadcn components on Tailwind CSS. The web pieces and the API deploy to Vercel, and Resend, GitHub Actions and Polar handle the service around them.

The features are aimed squarely at mobile work.

• Key management: provider keys live behind the proxy instead of in the app bundle.
• DeviceCheck verification: Apple's device attestation blocks requests that did not come from a real installation.
• Structured responses: JSON response schemas are defined in a visual builder and enforced on model output.
• Observability: usage, rate limits and model selection are monitored and adjusted while the app is live.
• Quick integration: the documented setup is meant to take under ten minutes with no extra code in the app.

Documentation is still being written and lives on the project's docs site. The license is AGPL-3.0 for non-commercial use, so a commercial deployment needs different terms from the maintainers.