Chunking is where most RAG systems silently bleed quality. Pick by document type.
Strategies
| Strategy | Mechanic | Best for | Failure mode |
|---|---|---|---|
| Fixed-size | Split every N tokens | Uniform prose | Splits mid-sentence, mid-table, breaks coherence |
| Recursive (LangChain default) | Try paragraph → sentence → word splits to fit window | General prose | Loses cross-paragraph context |
| Sentence-window | Index a sentence, retrieve, then expand to ±N sentences | FAQ, conversational | Larger storage |
| Semantic | Split when embedding distance between adjacent sentences spikes | Topic-shifting docs | Slow ingest; brittle on technical jargon |
| Parent-document | Index small chunks, but on retrieval return the parent paragraph or full doc | Code, contracts | More tokens at generate-time |
| Hierarchical | Index multiple granularities (sentence + paragraph + section) | Long, structured docs | Storage and complexity |
| Layout-aware | Use unstructured, Docling, LlamaParse to parse with the doc structure | PDFs with tables, slides | Parser-dependent quality |
Decision matrix
- Markdown / clean prose: recursive, 512-token chunks, 64 overlap.
- PDFs with tables/figures: layout-aware (
unstructuredpartition_pdf or LlamaParse). Table → Markdown rows that survive chunking. - Code: AST-aware splitters (LangChain
Language.PYTHONetc.) so functions stay intact. - FAQs / chat logs: sentence-window — single Q-A pair indexed, retrieve with surrounding context.
- Long contracts / books: hierarchical or parent-document.
- Scientific papers: layout-aware + section-aware (cite
paper-qapatterns); abstracts and methods chunked separately.
Overlap
- 10-20% of chunk size is the conventional default.
- Too low: cuts contextual signal at boundaries.
- Too high: duplicates content, retrieval surfaces near-duplicates, wastes context budget.
Practical sizes
- 256-512 tokens for prose Q&A (favor recall).
- 1024-2048 for analytical summaries (favor coherence).
- 128 sentence-window for chat / FAQ.
The metric to optimize
For chunk-strategy selection, don't rely on offline LLM-judge metrics alone. Run hit-rate@K on a labeled set with each strategy. The strategy with highest hit-rate@K wins for retrieval; then optimize chunk-context tradeoffs for generation.