Open Source Humanloop Alternatives

Langfuse is an open source LLM engineering platform for teams that develop, monitor, evaluate and debug AI applications together. It replaces the improvised mix of log files and spreadsheets most projects fall back on when they need to work out why a particular model call went wrong.

Instrumentation sends traces of LLM calls and the logic around them, retrieval, embedding and agent actions, so a complex run or a whole user session can be inspected step by step. Prompt management is centralized with version control and collaborative iteration, and caching on both server and client means changing a prompt adds no latency to the running application.

Four further pieces close the development loop.

• Evaluations: LLM-as-a-judge, code evaluators, user feedback, manual labeling and custom pipelines through the API.
• Datasets: test sets and benchmarks for pre-deployment testing and structured experiments.
• Playground: prompts and model configurations are iterated directly, reached from a bad result seen in a trace.
• API access: an OpenAPI spec, a Postman collection and typed Python and JavaScript SDKs power bespoke workflows.

Integrations cover the two SDKs, a drop-in replacement for the OpenAI SDK, callback handlers for LangChain, LlamaIndex and Haystack, and LiteLLM for reaching a hundred or more models. Self-hosting takes about five minutes locally with Docker Compose, runs on a single virtual machine, or goes to Kubernetes with Helm as the preferred production route. A managed cloud with a free tier exists for teams who would rather skip all of that.

Arize Phoenix is an open source AI observability platform built for experimentation, evaluation, and troubleshooting. It removes the guesswork from LLM applications, where a bad answer can come from a prompt, a retrieval step, or a model call, and nothing in an ordinary log tells you which.

Instrumentation is OpenTelemetry based through the OpenInference project, which keeps Phoenix vendor and language agnostic. It ships with support for frameworks such as LangGraph, CrewAI, LlamaIndex, DSPy and the OpenAI Agents SDK, and for providers including OpenAI, Anthropic, Google GenAI, AWS Bedrock and LiteLLM.

The platform is organized around a handful of workflows.

• Tracing: capture your application runtime and inspect every span sitting behind an answer.
• Evaluation: use LLMs to benchmark response and retrieval quality against criteria you define.
• Datasets and experiments: version examples, then track how prompt, model and retrieval changes score.
• Playground: compare models, adjust parameters, and replay traced calls while tuning a prompt.
• Prompt management: version, tag and test prompt changes instead of editing them in place.

A remote MCP server exposes traces, datasets and experiments to clients such as Claude Code and Cursor, and a setup command in the CLI wires instrumentation into an existing project by detecting its framework and provider.

Phoenix installs with pip or conda and runs locally, in a container, or on a cluster through published Docker images and a Helm chart. It is self hosted throughout, so traces stay inside your own infrastructure.

Latitude watches AI agents in production and closes the gap between noticing a failure and shipping the fix. Instead of leaving a wall of traces for somebody to read, it groups failing runs into tracked signals and hands them, with full context, to a coding agent.

Instrumentation is one line of telemetry. The TypeScript SDK wraps the LLM client an application already imports, Python and any OpenTelemetry-compatible runtime are supported, and traces arrive from OpenAI, Anthropic, Bedrock, the Vercel AI SDK, LangChain and other frameworks. A capture call at request, conversation or agent boundaries attaches user IDs, session IDs, tags and metadata.

The loop it runs has four stages.

• Observe: multi-turn sessions, tool calls and full execution paths captured as traces over OpenTelemetry ingest.
• Understand: failing traces grouped automatically into signals with status, size and trend, searchable semantically and open to annotation.
• Fix: Agent Dispatch sends Claude Code or Cursor the sample traces and a deep link, and the agent writes the smallest correct fix and opens a pull request.
• Verify: fixes are replayed against the real failing traces, with regression datasets, monitors and alerts to stop them coming back.

Everything available in the interface is also reachable from your coding agent through the MCP server and the CLI, and a separate package captures full Claude Code session transcripts as traces from the terminal, desktop app or IDE extensions. Self-hosting runs from published container images as a single-host Docker Compose instance, a Kubernetes cluster through a Helm chart, or a one-click Railway deploy.

OpenLIT handles the unglamorous parts of building with large language models: seeing what calls actually cost, keeping prompts versioned, holding API keys somewhere safer than a config file, and comparing models before committing to one. A single line of code switches on monitoring across LLMs, vector databases and GPUs.

Telemetry is OpenTelemetry-native throughout. SDKs for Python, TypeScript and Go send traces and metrics to an OpenTelemetry Collector, which stores them in ClickHouse for the OpenLIT interface to read, and the same stream can go to whatever observability tool you already run. The project maintains the Gen AI semantic conventions with the OpenTelemetry community.

Several tools are bundled around that pipeline.

• Analytics dashboard: application and GPU health, metrics, costs and user interactions in one view.
• Evaluations: eleven built-in LLM-as-a-judge checks covering hallucination, bias, toxicity, safety, instruction following, relevance and more, treating supplied context as the source of truth.
• Rule engine: conditional rules with AND and OR logic match trace attributes at runtime and pull in contexts, prompts and evaluation configs.
• Prompt Hub: prompts are managed and versioned for consistent access across applications.
• OpenGround: models are explored, tested and compared side by side.

Self-hosting is a clone and docker compose up, with a Helm chart for Kubernetes installs. Cost tracking accepts custom pricing files, so fine-tuned and in-house models can be budgeted as precisely as commercial ones. An exceptions dashboard surfaces recurring errors.