Modular RAG (Gao et al. 2024, arxiv 2312.10997) is the architectural pattern where the pipeline is a DAG of pluggable modules rather than a linear chain.
The five module categories:
- Search modules. Multiple parallel retrievers — vector DB, graph DB, SQL, web, internal API.
- Memory modules. Conversation memory, user profile memory, scratch memory.
- Routing modules. Decide which downstream module(s) to call based on the query.
- Predict modules. The LLM call(s) — could be generation, classification, extraction.
- Task adapter modules. Convert outputs to a target schema or trigger a downstream tool.
The benefit isn't theoretical — it's independent A/B testing. You can swap your re-ranker without touching the generator. You can add a new data source as a parallel retriever and merge results, instead of rebuilding the index.
In code, this is exactly what frameworks like LangGraph, Haystack, and DSPy model: nodes are modules, edges are data flow.
When modular wins: more than one data source, more than one query type, more than one team owning components.
When modular hurts: a 3-engineer team with one corpus and one query type — the abstraction tax exceeds the optimization wins.