Agentic RAG flips the script: instead of one fixed retrieval before generation, the LLM decides when, what, and how to retrieve mid-reasoning.
The mechanic:
- The agent receives a query.
- The agent decides: is retrieval needed? Which retriever? What sub-query?
- The agent retrieves, observes results, and either: re-queries with refined terms, calls a different retriever, or generates the final answer.
- Loop until confident or budget exhausted.
This is the ReAct pattern (Yao et al. 2022) applied with retrieval as a primary tool.
Why this beats fixed-pipeline RAG for hard questions:
- Multi-hop questions. "Which IBM watsonx model is best for Arabic NER, and what's its cost?" → first retrieves model list, then retrieves pricing for the chosen model. A fixed pipeline can't do this in one retrieval.
- Ambiguous queries. The agent can disambiguate via clarifying retrieval before answering.
- Self-correction. If the first retrieval returns nothing, the agent rewrites the query.
Implementation sketch (LangGraph):
def should_retrieve(state): ...
def retrieve(state): ...
def grade(state): ...
def rewrite_query(state): ...
def generate(state): ...
graph = StateGraph(...)
graph.add_node("decide", should_retrieve)
graph.add_node("retrieve", retrieve)
graph.add_node("grade", grade)
graph.add_node("rewrite", rewrite_query)
graph.add_node("generate", generate)
graph.add_conditional_edges("decide", lambda s: "retrieve" if s.needs_docs else "generate")
graph.add_conditional_edges("grade", lambda s: "rewrite" if s.bad_results else "generate")
Costs and risks:
- Latency multiplied by hops. A 3-hop agent is 3× a fixed pipeline.
- Loops. Without strict iteration limits and termination conditions, agents loop on bad queries.
- Eval is hard. Need trajectory eval, not just final-answer eval.
Production rule: cap iterations (3–5), cache per-query, use a cheap model for routing decisions and reserve the expensive model for synthesis.