LangWatch

LangWatch is a platform for LLM evaluation and AI agent testing, covering the same system before release and once it is live. It exists so a team stops assembling its own regression harness, simulation runner and tracing stack separately and then gluing them together.

Tracing is OpenTelemetry and OTLP native, which keeps it agnostic about frameworks and model providers. The working loop is a single circuit rather than a pile of tools: trace, build a dataset, evaluate, optimize prompts and models, then test again.

The platform breaks into a few parts.

• Agent simulations: run realistic scenarios against the full stack, tools, state, user simulator and judge, and see where an agent breaks down to each decision.
• AI gateway: an OpenAI and Anthropic compatible proxy with virtual keys, hierarchical budgets, inline guardrails and automatic fallback across providers, shipped as a separate Go binary and Helm sub-chart.
• Annotations and queues: domain experts label edge cases and review runs beside the engineers.
• Prompt management: prompts live in Git through the GitHub integration, and prompt versions link back to the traces they produced.
• MCP server: reach LangWatch from Claude Desktop and other MCP clients.

Local setup needs only Node.js. One command installs Postgres, Redis, ClickHouse, the gateway binary and the assistant runtime into a single directory, writes an env file with generated secrets and starts everything. The code is open-core, an Apache 2.0 floor with an enterprise extension. Docker Compose, Kubernetes with Helm and cloud-specific on-premises setups cover self-hosting, and a hybrid mode keeps data in your own environment.