OpenLIT

OpenLIT handles the unglamorous parts of building with large language models: seeing what calls actually cost, keeping prompts versioned, holding API keys somewhere safer than a config file, and comparing models before committing to one. A single line of code switches on monitoring across LLMs, vector databases and GPUs.

Telemetry is OpenTelemetry-native throughout. SDKs for Python, TypeScript and Go send traces and metrics to an OpenTelemetry Collector, which stores them in ClickHouse for the OpenLIT interface to read, and the same stream can go to whatever observability tool you already run. The project maintains the Gen AI semantic conventions with the OpenTelemetry community.

Several tools are bundled around that pipeline.

• Analytics dashboard: application and GPU health, metrics, costs and user interactions in one view.
• Evaluations: eleven built-in LLM-as-a-judge checks covering hallucination, bias, toxicity, safety, instruction following, relevance and more, treating supplied context as the source of truth.
• Rule engine: conditional rules with AND and OR logic match trace attributes at runtime and pull in contexts, prompts and evaluation configs.
• Prompt Hub: prompts are managed and versioned for consistent access across applications.
• OpenGround: models are explored, tested and compared side by side.

Self-hosting is a clone and docker compose up, with a Helm chart for Kubernetes installs. Cost tracking accepts custom pricing files, so fine-tuned and in-house models can be budgeted as precisely as commercial ones. An exceptions dashboard surfaces recurring errors.