Envoy AI Gateway

Envoy AI Gateway handles request traffic from application clients to generative AI services, using Envoy Gateway as its foundation. It gives an organization one place to put authentication, routing and rate limiting for model calls, instead of scattering provider keys through every service that wants to talk to a language model.

The recommended shape is a two-tier gateway. The tier one gateway is the centralized entry point, handling authentication, top-level routing and global rate limiting. The tier two gateway sits in front of a self-hosted model serving cluster and gives fine-grained control over access to those models.

A few pieces define how the project is used.

• Two-tier pattern: a central entry gateway in front of gateways guarding self-hosted model clusters.
• Endpoint picker: routing support aimed at optimizing inference on those self-hosted clusters.
• Hosted providers: OpenAI, Azure OpenAI, Google Gemini, Vertex AI, AWS Bedrock, Anthropic and Mistral.
• Independent model hosts: Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok and the Tetrate Agent Router Service.
• Quickstart: a getting-started guide that brings the gateway up in a few steps, with a concepts section covering the architecture and its resources.

The project grew out of a proposal to use Envoy Gateway as a cloud native gateway for large language models. It follows the CNCF code of conduct, holds a community meeting every Monday with a public agenda, and coordinates in a dedicated channel of the Envoy Slack workspace, so contributions arrive through the usual pull request path.