LLM eval has three settings. Each catches what the others miss.
1. Offline eval (development)
- Curated labeled set + automated judges.
- Run on every prompt / model / chunking change.
- Cheap, fast, deterministic — but only as good as the labels.
- Tools: RAGAS, LangSmith Datasets, LangFuse, custom.
Pitfall: offline sets become stale as production drift makes them unrepresentative.
2. Online eval (production)
- Sample live traces; run LLM-judge or rule-based scoring.
- Catches things offline misses: real query distribution, real user phrasing, real edge cases.
- Tools: LangSmith online evals, custom cron over trace store.
Pitfall: sampling bias — sample by feature, tenant, error to avoid blind spots.
3. A/B testing
- Two configurations served to user buckets; measure outcome metrics.
- Outcome metric should be business, not just LLM (conversion, retention, task completion, NPS).
- Statistical care: power analysis before launching, sequential testing for early-stop.
Pitfall: running A/B on too small a population to detect real differences.
The reinforcement loop
- Find a regression in online eval.
- Add the failing example to offline eval.
- Iterate prompts/models against offline.
- Promote the winner via A/B test before global rollout.
- Continue online monitoring.
Eval anti-patterns
- One-shot eval. A 50-example set you wrote 6 months ago and never updated.
- LLM-judge bias. Same model judges its own outputs — favorable bias is real. Use a different model or human-anchored prompts.
- Aggregate-only. Average score hides catastrophic failures on critical sub-segments. Always slice by intent / tenant / language.
- No human anchor. At least 50 examples should be human-labeled to calibrate judges.
A pragmatic eval set
- 100-300 examples, labeled.
- Stratified: easy / medium / hard, intents, languages, edge cases.
- Versioned. Updated quarterly. New failures from prod added immediately.
- Runs in CI on every prompt change.