Spanlens

Provider dashboards report total spend and stop there. Spanlens sits between an application and the model API, logging every call with its model, tokens, cost, latency and full request and response body, streaming replies reassembled, so a bill traces back to the feature behind it.

Turning it on means pointing an OpenAI, Anthropic or Gemini client at the Spanlens proxy, which a CLI wizard can rewrite for you, or attaching a callback to the Vercel AI SDK, LangChain, LangGraph or LlamaIndex. SDKs ship on npm and PyPI, and a bundled MCP server lets you query the data from Cursor or Claude Desktop.

What the dashboard adds on top of the raw log is where the work went.

• Agent tracing: multi-step runs render as span waterfalls with the critical path highlighted, plus a node and edge topology view for callback traces.
• Cost tracking: per-request breakdowns, daily rollups and budget alerts, with discounted cache-read tokens counted separately.
• Anomaly detection: three-sigma moves in latency, cost or error rate against a seven-day baseline, with root-cause hints.
• Content scanning: regex detection of PII and prompt injection on requests and responses, with optional blocking per project.
• Prompt experiments: traffic-split tests compared using Welch's t-test on latency and cost and a z-test on error rate.
• Evaluations: judge models score responses against rubric anchors, with pairwise mode and human agreement measured by Pearson r or Cohen's kappa.

Alerts reach email, Slack or Discord webhooks from a fifteen-minute cron, and the service self-hosts with one Docker command under an MIT license.