Advanced RAG splits the pipeline into three intervention points:
Pre-retrieval — fix the query before searching
- Query rewriting. Use an LLM to expand or reformulate the user query into something closer to your corpus vocabulary. Cheap, often +5-10% recall.
- HyDE (Hypothetical Document Embeddings). Have the LLM write a fake answer to the question, then embed that and search for it. Works because answers are vocabulary-aligned with documents, queries are not. Paper: Gao et al. 2022 (arxiv 2212.10496).
- Step-back prompting. First ask "what's the more general question this implies?", retrieve for that, then answer the specific. Good for multi-hop. Paper: Zheng et al. 2023.
- Multi-query. Generate N paraphrases, retrieve for each, deduplicate. Trades latency for recall.
Retrieval — better search
- Hybrid search. Combine sparse (BM25, SPLADE) and dense (embedding) retrievers. Sparse catches exact terms (product codes, function names); dense catches semantics. Fuse with Reciprocal Rank Fusion (RRF):
score(d) = Σ 1/(k + rank_i(d))over retrievers. - Metadata filters. Narrow before similarity. Always filter by tenant, time window, source type if applicable. Free recall improvement, free safety improvement.
- Self-querying. Use an LLM to extract structured filters from the query ("docs from 2024 about pricing" →
{filter: {year: 2024, topic: 'pricing'}, q: ...}).
Re-ranking — order top-K correctly
- Cross-encoder re-rankers. Take top-50 from retriever, re-rank with a model that scores (query, doc) pairs. Cohere Rerank-3, BGE Reranker, Jina Reranker. Adds 50–200ms but +10-20% on real eval.
- LLM-as-reranker. Cheaper at low scale: prompt the LLM to score relevance 0-10. Slower at high scale.
Post-retrieval — clean up the context
- Context compression. Use an LLM (or a fine-tuned compressor like LLMLingua) to keep only sentences relevant to the question. Cuts tokens 60-80%, fixes lost-in-the-middle.
- Chunk merging / parent-document. Retrieve small chunks, then expand each to its surrounding paragraph or full document for context.
- De-duplication. Two chunks with the same content waste a slot; cluster on cosine similarity > 0.9 and keep one.
Stack you actually run in production: query rewriting → hybrid (BM25 + dense) → top-50 → re-ranker → top-5 → compression → generate.