Open Source Braintrust Alternatives

Langfuse is an open source LLM engineering platform for teams that develop, monitor, evaluate and debug AI applications together. It replaces the improvised mix of log files and spreadsheets most projects fall back on when they need to work out why a particular model call went wrong.

Instrumentation sends traces of LLM calls and the logic around them, retrieval, embedding and agent actions, so a complex run or a whole user session can be inspected step by step. Prompt management is centralized with version control and collaborative iteration, and caching on both server and client means changing a prompt adds no latency to the running application.

Four further pieces close the development loop.

• Evaluations: LLM-as-a-judge, code evaluators, user feedback, manual labeling and custom pipelines through the API.
• Datasets: test sets and benchmarks for pre-deployment testing and structured experiments.
• Playground: prompts and model configurations are iterated directly, reached from a bad result seen in a trace.
• API access: an OpenAPI spec, a Postman collection and typed Python and JavaScript SDKs power bespoke workflows.

Integrations cover the two SDKs, a drop-in replacement for the OpenAI SDK, callback handlers for LangChain, LlamaIndex and Haystack, and LiteLLM for reaching a hundred or more models. Self-hosting takes about five minutes locally with Docker Compose, runs on a single virtual machine, or goes to Kubernetes with Helm as the preferred production route. A managed cloud with a free tier exists for teams who would rather skip all of that.

Arize Phoenix is an open source AI observability platform built for experimentation, evaluation, and troubleshooting. It removes the guesswork from LLM applications, where a bad answer can come from a prompt, a retrieval step, or a model call, and nothing in an ordinary log tells you which.

Instrumentation is OpenTelemetry based through the OpenInference project, which keeps Phoenix vendor and language agnostic. It ships with support for frameworks such as LangGraph, CrewAI, LlamaIndex, DSPy and the OpenAI Agents SDK, and for providers including OpenAI, Anthropic, Google GenAI, AWS Bedrock and LiteLLM.

The platform is organized around a handful of workflows.

• Tracing: capture your application runtime and inspect every span sitting behind an answer.
• Evaluation: use LLMs to benchmark response and retrieval quality against criteria you define.
• Datasets and experiments: version examples, then track how prompt, model and retrieval changes score.
• Playground: compare models, adjust parameters, and replay traced calls while tuning a prompt.
• Prompt management: version, tag and test prompt changes instead of editing them in place.

A remote MCP server exposes traces, datasets and experiments to clients such as Claude Code and Cursor, and a setup command in the CLI wires instrumentation into an existing project by detecting its framework and provider.

Phoenix installs with pip or conda and runs locally, in a container, or on a cluster through published Docker images and a Helm chart. It is self hosted throughout, so traces stay inside your own infrastructure.

OpenLLMetry extends OpenTelemetry to cover LLM applications, so model calls, vector database queries and agent frameworks appear in the observability stack a team already runs. Nothing has to move to a separate vendor dashboard built only for AI.

The repository holds standard OpenTelemetry instrumentations for LLM providers and vector databases, plus a Traceloop SDK that shortens the setup. Getting started is a pip install and a single init call in your code. Teams already instrumented with OpenTelemetry can skip the SDK and add the instrumentations directly.

Coverage spans the layers an LLM request passes through.

• Model providers: instrument OpenAI, Anthropic, Bedrock, Vertex AI, Cohere, Mistral, Groq, Ollama, HuggingFace and more.
• Vector databases: cover Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB and Marqo.
• Frameworks: trace LangChain, LlamaIndex, LangGraph, CrewAI, Haystack, LiteLLM and OpenAI Agents.
• Protocol support: instruments the Model Context Protocol alongside the frameworks that use it.
• Destinations: export to Datadog, Honeycomb, Grafana, New Relic, Splunk, Sentry, SigNoz, Dynatrace and around two dozen more.

Everything OpenTelemetry already instruments, a database or an outbound API call, keeps working next to these extensions. Traceloop maintains the project under the Apache 2.0 license and has removed telemetry collection from both the SDK and the instrumentations. The semantic conventions it introduced are now part of OpenTelemetry itself, and a JavaScript and TypeScript version lives in a sibling repository.

Helicone sits between an application and the model providers it calls, turning opaque LLM traffic into something inspectable. Changing the base URL in an OpenAI client is enough to start logging every request, and the same gateway reaches more than a hundred models behind one API key.

Beyond logging, the gateway handles routing and automatic fallbacks when a provider fails. Integrations cover OpenAI, Anthropic, Gemini, LangChain and the Vercel AI SDK, and metrics export to PostHog in a single line for teams keeping their dashboards elsewhere.

The platform gathers the tasks that follow a deployed model.

• Tracing: sessions and traces for agents, chatbots and document processing pipelines, inspectable when something goes wrong.
• Cost and latency: per-request accounting of spend, latency and quality across every provider in use.
• Playground: prompts, sessions and traces can be replayed and iterated on in the interface.
• Prompt management: versioning against production data, deployed through the gateway with no code change.
• Fine-tuning: handoff to partner services for training on the data already collected.

Self-hosting runs from a Docker Compose setup, with a Helm chart offered for enterprise workloads and manual deployment discouraged. The stack splits into a Next.js web frontend, a Cloudflare Workers proxy for logging, an Express server that collects logs, Supabase for the application database and auth, ClickHouse for analytics and MinIO for object storage. The hosted service is SOC 2 and GDPR compliant, with a free tier of ten thousand requests a month.

Latitude watches AI agents in production and closes the gap between noticing a failure and shipping the fix. Instead of leaving a wall of traces for somebody to read, it groups failing runs into tracked signals and hands them, with full context, to a coding agent.

Instrumentation is one line of telemetry. The TypeScript SDK wraps the LLM client an application already imports, Python and any OpenTelemetry-compatible runtime are supported, and traces arrive from OpenAI, Anthropic, Bedrock, the Vercel AI SDK, LangChain and other frameworks. A capture call at request, conversation or agent boundaries attaches user IDs, session IDs, tags and metadata.

The loop it runs has four stages.

• Observe: multi-turn sessions, tool calls and full execution paths captured as traces over OpenTelemetry ingest.
• Understand: failing traces grouped automatically into signals with status, size and trend, searchable semantically and open to annotation.
• Fix: Agent Dispatch sends Claude Code or Cursor the sample traces and a deep link, and the agent writes the smallest correct fix and opens a pull request.
• Verify: fixes are replayed against the real failing traces, with regression datasets, monitors and alerts to stop them coming back.

Everything available in the interface is also reachable from your coding agent through the MCP server and the CLI, and a separate package captures full Claude Code session transcripts as traces from the terminal, desktop app or IDE extensions. Self-hosting runs from published container images as a single-host Docker Compose instance, a Kubernetes cluster through a Helm chart, or a one-click Railway deploy.

LangWatch is a platform for LLM evaluation and AI agent testing, covering the same system before release and once it is live. It exists so a team stops assembling its own regression harness, simulation runner and tracing stack separately and then gluing them together.

Tracing is OpenTelemetry and OTLP native, which keeps it agnostic about frameworks and model providers. The working loop is a single circuit rather than a pile of tools: trace, build a dataset, evaluate, optimize prompts and models, then test again.

The platform breaks into a few parts.

• Agent simulations: run realistic scenarios against the full stack, tools, state, user simulator and judge, and see where an agent breaks down to each decision.
• AI gateway: an OpenAI and Anthropic compatible proxy with virtual keys, hierarchical budgets, inline guardrails and automatic fallback across providers, shipped as a separate Go binary and Helm sub-chart.
• Annotations and queues: domain experts label edge cases and review runs beside the engineers.
• Prompt management: prompts live in Git through the GitHub integration, and prompt versions link back to the traces they produced.
• MCP server: reach LangWatch from Claude Desktop and other MCP clients.

Local setup needs only Node.js. One command installs Postgres, Redis, ClickHouse, the gateway binary and the assistant runtime into a single directory, writes an env file with generated secrets and starts everything. The code is open-core, an Apache 2.0 floor with an enterprise extension. Docker Compose, Kubernetes with Helm and cloud-specific on-premises setups cover self-hosting, and a hybrid mode keeps data in your own environment.

Laminar is an open source observability platform aimed at AI agents rather than at general web services. It exists to answer the question that gets hard once an agent is loose in production, which is what the thing actually did and where it went wrong.

Instrumentation is OpenTelemetry-native. One line of SDK setup traces Vercel AI SDK, Browser Use, Stagehand, LangChain, OpenAI, Anthropic and Gemini calls automatically, and an observe wrapper in TypeScript or a decorator in Python captures the inputs and outputs of your own functions.

Around that tracing layer sits the rest of the platform.

• Signals: describe a behavior to watch for in plain English, such as an agent stuck in a loop, and get pinged in Slack when it occurs.
• Evals: an unopinionated SDK and CLI run evaluations locally or in CI, with a UI for comparing results.
• Dashboards: traces, metrics and events are charted, including through custom SQL queries.
• Datasets: a custom rendering UI speeds up annotating data and assembling evaluation sets.
• Agent access: an MCP server and CLI let your coding agent query traces and spans with SQL while debugging.

The backend is written in Rust, compresses agent traces roughly twentyfold for ingestion and storage, and runs a realtime engine for watching traces as they arrive. Self-hosting is a clone and docker compose up, with a fuller compose file recommended for production and telemetry disabled by one environment variable.

OpenLIT handles the unglamorous parts of building with large language models: seeing what calls actually cost, keeping prompts versioned, holding API keys somewhere safer than a config file, and comparing models before committing to one. A single line of code switches on monitoring across LLMs, vector databases and GPUs.

Telemetry is OpenTelemetry-native throughout. SDKs for Python, TypeScript and Go send traces and metrics to an OpenTelemetry Collector, which stores them in ClickHouse for the OpenLIT interface to read, and the same stream can go to whatever observability tool you already run. The project maintains the Gen AI semantic conventions with the OpenTelemetry community.

Several tools are bundled around that pipeline.

• Analytics dashboard: application and GPU health, metrics, costs and user interactions in one view.
• Evaluations: eleven built-in LLM-as-a-judge checks covering hallucination, bias, toxicity, safety, instruction following, relevance and more, treating supplied context as the source of truth.
• Rule engine: conditional rules with AND and OR logic match trace attributes at runtime and pull in contexts, prompts and evaluation configs.
• Prompt Hub: prompts are managed and versioned for consistent access across applications.
• OpenGround: models are explored, tested and compared side by side.

Self-hosting is a clone and docker compose up, with a Helm chart for Kubernetes installs. Cost tracking accepts custom pricing files, so fine-tuned and in-house models can be budgeted as precisely as commercial ones. An exceptions dashboard surfaces recurring errors.

Provider dashboards report total spend and stop there. Spanlens sits between an application and the model API, logging every call with its model, tokens, cost, latency and full request and response body, streaming replies reassembled, so a bill traces back to the feature behind it.

Turning it on means pointing an OpenAI, Anthropic or Gemini client at the Spanlens proxy, which a CLI wizard can rewrite for you, or attaching a callback to the Vercel AI SDK, LangChain, LangGraph or LlamaIndex. SDKs ship on npm and PyPI, and a bundled MCP server lets you query the data from Cursor or Claude Desktop.

What the dashboard adds on top of the raw log is where the work went.

• Agent tracing: multi-step runs render as span waterfalls with the critical path highlighted, plus a node and edge topology view for callback traces.
• Cost tracking: per-request breakdowns, daily rollups and budget alerts, with discounted cache-read tokens counted separately.
• Anomaly detection: three-sigma moves in latency, cost or error rate against a seven-day baseline, with root-cause hints.
• Content scanning: regex detection of PII and prompt injection on requests and responses, with optional blocking per project.
• Prompt experiments: traffic-split tests compared using Welch's t-test on latency and cost and a z-test on error rate.
• Evaluations: judge models score responses against rubric anchors, with pairwise mode and human agreement measured by Pearson r or Cohen's kappa.

Alerts reach email, Slack or Discord webhooks from a fifteen-minute cron, and the service self-hosts with one Docker command under an MIT license.