If you can't measure your RAG, you can't improve it. The standard framework is RAGAS (Es et al. 2023, arxiv 2309.15217), but the metrics matter more than the framework.
The four RAGAS core metrics
| Metric | Question it answers | Computed how |
|---|---|---|
| Faithfulness | Is the answer grounded in retrieved context? | LLM judge: claim extraction → check each claim against context |
| Answer relevancy | Does the answer address the question? | Generate hypothetical questions from the answer; embed-similarity to original question |
| Context precision | Of retrieved chunks, what % are relevant? | LLM judge per chunk × top-K |
| Context recall | Of relevant info needed, what % did we retrieve? | Compare retrieved chunks to ground-truth references |
How to use these in practice
- Context recall is the ceiling. Generation can never beat what retrieval gave it. If recall is bad, fix retrieval first — re-ranking, hybrid search, query rewriting.
- Faithfulness diagnoses generator failures. Low faithfulness with high recall = the LLM is ignoring context. Tighten the system prompt; consider a smaller / more obedient model.
- Answer relevancy catches off-topic generation. A chatty model that drifts hurts here.
- Context precision diagnoses retriever noise. Lots of irrelevant chunks dilute the model's attention.
Beyond RAGAS
- Hit-rate@K / nDCG@K for retrieval-only eval. Cheaper than LLM-judge and deterministic.
- End-to-end task accuracy with a labeled QA set. The actual business metric.
- Trajectory eval for agentic RAG — was the retrieval path sensible?
- Human eval at small scale for high-stakes domains. LLM judges miss subtle factuality issues in legal/medical.
Online evaluation
- Log all (query, retrieved IDs, answer, user signal) tuples — implicit feedback (was the answer copied? was a follow-up needed?) and explicit (thumbs).
- A/B test components independently using LangSmith or LangFuse traces.
- Watch for quality drift: as your corpus grows, recall can silently drop.
The eval anti-patterns
- Evaluating only end-to-end accuracy, never decomposing — you can't tell if it's retrieval or generation failing.
- Relying on a single LLM judge — judge bias is real. Use multiple judges or a stricter cross-encoder for critical metrics.
- Static eval set that never updates — production drift makes a 6-month-old eval set lie.