An LLM application without observability is a closed-loop system you can't debug. Observability is the prerequisite to everything else — eval, optimization, incident response.
What to trace
Every LLM call:
- Inputs: prompt, system, params (model, temperature, max_tokens).
- Outputs: completion text, finish reason, token counts.
- Tool calls: name, arguments, result, duration, error.
- Cost & latency: real $ and ms per call and per session.
- User context: session id, user id (hashed), feature flag state, model version.
- Trajectory: parent/child relationships of nested calls.
If you log only the final answer, you're flying blind.
Platforms
| Tool | Hosting | Strength |
|---|---|---|
| LangSmith | Hosted by LangChain | Best LangChain/LangGraph integration, dataset eval, online evaluation, prompt playground |
| LangFuse | Open-source + cloud | Similar feature set, OSS, easy self-host |
| Phoenix (Arize) | Open-source + cloud | Strong trajectory viz, OpenTelemetry-native |
| Helicone | Hosted | Drop-in proxy, simple to integrate |
| Weights & Biases | Hosted + self-host | Mature ML side; LLM features growing |
| Custom | OpenTelemetry → your stack | Native to your existing observability |
Pick by integration ease. If you're on LangChain, use LangSmith. If you're on a custom stack, use OpenTelemetry + your existing observability (Datadog / Honeycomb / Grafana Tempo).
What good traces enable
- Replay. Re-run a problematic trace with a new prompt or model.
- Compare. Diff two model responses on the same input.
- Aggregate. "Show me all sessions where the agent looped > 5 times."
- Eval. Run nightly LLM-judge against last 24h of traces; alert on regressions.
- Cost attribution. Per-feature, per-tenant cost.
Common observability mistakes
- Logging only on success. Failures and edge cases are exactly what you need to see.
- Logging without context. "GPT-4o said X" without input/params is useless 6 months later.
- No PII redaction. Real user data in traces becomes a compliance nightmare.
- Storing forever. Set retention; LLM traces explode in size.
- No sampling strategy. At scale you can't trace 100%; sample by user / feature / cost tier.
The first three things to instrument
- Every LLM call — inputs, outputs, params, tokens, latency, cost.
- Every tool call — same shape.
- End-to-end task latency — wall-clock from user request to user response.
That alone catches 80% of production issues.