SYSTEM ONLINE

MODELgemini-3.1-pro

LATENCY142ms

THROUGHPUT4.2 tok/ms

● RAG // 10 lessons● EMBEDDINGS // 7 lessons● MODELS // 8 lessons● AGENTS // 6 lessons● PROD AI // 7 lessons● ENTERPRISE // 5 lessons● RAG // 10 lessons● EMBEDDINGS // 7 lessons● MODELS // 8 lessons● AGENTS // 6 lessons● PROD AI // 7 lessons● ENTERPRISE // 5 lessons

// LOADING

RAG/12 MIN/3 QUIZ

What is RAG & Why It Exists

The two problems RAG solves: the context window ceiling and the hallucination floor. The mental model that makes every RAG variant click.

RAG · What is RAG & Why It Exists

LIVE

RAG is what you build when you accept that an LLM is a fixed-weight reasoning engine, not a database.

Two problems force RAG into existence:

1. The context window problem. Even a 1M-token context window doesn't hold your enterprise corpus, doesn't refresh as your knowledge changes, and turns "answer one question" into a million-token bill. Stuffing everything into context is economically unviable above ~10MB of source.

2. The hallucination floor. Without grounding, models confabulate. They produce plausible, fluent, wrong answers — and they do it confidently. Confidence calibration on closed-book QA is famously broken (see the original OpenAI TruthfulQA paper, arxiv 2109.07958).

The RAG pattern: at inference time, retrieve a small, relevant slice of your corpus and inject it into the prompt as grounding. The model now reasons over evidence you control instead of over its parametric memory.

Three things matter:

Recall of retrieval. If the right chunk isn't in your top-K, the model can't answer correctly. This is your single biggest failure mode.
Faithfulness of generation. Even with the right chunk, the model can ignore it and hallucinate. RAGAS faithfulness measures this.
Latency. You added an embed → retrieve → re-rank hop in front of generation. Production budgets get tight fast.

The mental model: RAG is a prompt-construction strategy. Everything else (vector DBs, chunking, re-ranking) is infrastructure to make the prompt good.

// KEY_TAKEAWAYS

What to Remember

RAG exists because LLMs are reasoners, not databases — and stuffing everything in context doesn't scale.

Retrieval recall, generation faithfulness, and latency are the three axes to optimize.

RAG is fundamentally a prompt-construction strategy — the rest is plumbing.

// DECISION_MATRIX

When to Use

USE WHEN

Knowledge changes frequently (docs, support content, news).
Source-of-truth answers required (legal, medical, finance).
Corpus too large for context windows or fine-tuning is too slow to iterate.

AVOID WHEN

Stable, narrow domain that fits in fine-tuning — e.g., a structured-output classifier on a fixed taxonomy.
Pure reasoning tasks with no external knowledge (math, code transformations).
Latency budget under 200ms end-to-end with no caching.

TRADE-OFFS

Adds 50–500ms latency vs. raw LLM call.
Quality is bottlenecked by retrieval — bad retriever = bad answers no matter the model.
Eval is harder: you now have two systems (retriever + generator) to evaluate independently.

// INTERVIEW_GOTCHAS

What Trips People Up

THE TRAP

Saying 'because LLMs hallucinate' — only half the answer.

GOOD ANSWER

Two reasons: (1) LLMs hallucinate without grounding, (2) you can't fit or afford to fit a real corpus in the context window even with 1M-token models. RAG fixes both at once by injecting a small, relevant evidence slice per query.

// GLOSSARY

ragcontext windowhallucinationretrieval recallfaithfulness