Open Source Galileo Alternatives

OpenLLMetry extends OpenTelemetry to cover LLM applications, so model calls, vector database queries and agent frameworks appear in the observability stack a team already runs. Nothing has to move to a separate vendor dashboard built only for AI.

The repository holds standard OpenTelemetry instrumentations for LLM providers and vector databases, plus a Traceloop SDK that shortens the setup. Getting started is a pip install and a single init call in your code. Teams already instrumented with OpenTelemetry can skip the SDK and add the instrumentations directly.

Coverage spans the layers an LLM request passes through.

• Model providers: instrument OpenAI, Anthropic, Bedrock, Vertex AI, Cohere, Mistral, Groq, Ollama, HuggingFace and more.
• Vector databases: cover Chroma, Pinecone, Qdrant, Weaviate, Milvus, LanceDB and Marqo.
• Frameworks: trace LangChain, LlamaIndex, LangGraph, CrewAI, Haystack, LiteLLM and OpenAI Agents.
• Protocol support: instruments the Model Context Protocol alongside the frameworks that use it.
• Destinations: export to Datadog, Honeycomb, Grafana, New Relic, Splunk, Sentry, SigNoz, Dynatrace and around two dozen more.

Everything OpenTelemetry already instruments, a database or an outbound API call, keeps working next to these extensions. Traceloop maintains the project under the Apache 2.0 license and has removed telemetry collection from both the SDK and the instrumentations. The semantic conventions it introduced are now part of OpenTelemetry itself, and a JavaScript and TypeScript version lives in a sibling repository.

LangWatch is a platform for LLM evaluation and AI agent testing, covering the same system before release and once it is live. It exists so a team stops assembling its own regression harness, simulation runner and tracing stack separately and then gluing them together.

Tracing is OpenTelemetry and OTLP native, which keeps it agnostic about frameworks and model providers. The working loop is a single circuit rather than a pile of tools: trace, build a dataset, evaluate, optimize prompts and models, then test again.

The platform breaks into a few parts.

• Agent simulations: run realistic scenarios against the full stack, tools, state, user simulator and judge, and see where an agent breaks down to each decision.
• AI gateway: an OpenAI and Anthropic compatible proxy with virtual keys, hierarchical budgets, inline guardrails and automatic fallback across providers, shipped as a separate Go binary and Helm sub-chart.
• Annotations and queues: domain experts label edge cases and review runs beside the engineers.
• Prompt management: prompts live in Git through the GitHub integration, and prompt versions link back to the traces they produced.
• MCP server: reach LangWatch from Claude Desktop and other MCP clients.

Local setup needs only Node.js. One command installs Postgres, Redis, ClickHouse, the gateway binary and the assistant runtime into a single directory, writes an env file with generated secrets and starts everything. The code is open-core, an Apache 2.0 floor with an enterprise extension. Docker Compose, Kubernetes with Helm and cloud-specific on-premises setups cover self-hosting, and a hybrid mode keeps data in your own environment.

Laminar is an open source observability platform aimed at AI agents rather than at general web services. It exists to answer the question that gets hard once an agent is loose in production, which is what the thing actually did and where it went wrong.

Instrumentation is OpenTelemetry-native. One line of SDK setup traces Vercel AI SDK, Browser Use, Stagehand, LangChain, OpenAI, Anthropic and Gemini calls automatically, and an observe wrapper in TypeScript or a decorator in Python captures the inputs and outputs of your own functions.

Around that tracing layer sits the rest of the platform.

• Signals: describe a behavior to watch for in plain English, such as an agent stuck in a loop, and get pinged in Slack when it occurs.
• Evals: an unopinionated SDK and CLI run evaluations locally or in CI, with a UI for comparing results.
• Dashboards: traces, metrics and events are charted, including through custom SQL queries.
• Datasets: a custom rendering UI speeds up annotating data and assembling evaluation sets.
• Agent access: an MCP server and CLI let your coding agent query traces and spans with SQL while debugging.

The backend is written in Rust, compresses agent traces roughly twentyfold for ingestion and storage, and runs a realtime engine for watching traces as they arrive. Self-hosting is a clone and docker compose up, with a fuller compose file recommended for production and telemetry disabled by one environment variable.

Provider dashboards report total spend and stop there. Spanlens sits between an application and the model API, logging every call with its model, tokens, cost, latency and full request and response body, streaming replies reassembled, so a bill traces back to the feature behind it.

Turning it on means pointing an OpenAI, Anthropic or Gemini client at the Spanlens proxy, which a CLI wizard can rewrite for you, or attaching a callback to the Vercel AI SDK, LangChain, LangGraph or LlamaIndex. SDKs ship on npm and PyPI, and a bundled MCP server lets you query the data from Cursor or Claude Desktop.

What the dashboard adds on top of the raw log is where the work went.

• Agent tracing: multi-step runs render as span waterfalls with the critical path highlighted, plus a node and edge topology view for callback traces.
• Cost tracking: per-request breakdowns, daily rollups and budget alerts, with discounted cache-read tokens counted separately.
• Anomaly detection: three-sigma moves in latency, cost or error rate against a seven-day baseline, with root-cause hints.
• Content scanning: regex detection of PII and prompt injection on requests and responses, with optional blocking per project.
• Prompt experiments: traffic-split tests compared using Welch's t-test on latency and cost and a z-test on error rate.
• Evaluations: judge models score responses against rubric anchors, with pairwise mode and human agreement measured by Pearson r or Cohen's kappa.

Alerts reach email, Slack or Discord webhooks from a fifteen-minute cron, and the service self-hosts with one Docker command under an MIT license.