The eternal interview question: "Should we fine-tune or use RAG?"
The wrong answer is to pick one. The right answer is to know which knob each turns:
| Need | RAG | Fine-tuning |
|---|---|---|
| Inject fresh / changing facts | ✅ Native | ❌ Re-train per update |
| Citations / provenance | ✅ Native | ❌ No source attribution |
| Consistent format / structured output | ⚠️ Possible via prompt | ✅ Native |
| Tone / persona / domain register | ⚠️ Brittle via prompt | ✅ Native |
| New skill or behavior | ❌ Can't teach skills | ✅ Native |
| Token-cost reduction | ❌ Adds tokens | ✅ Smaller model can match larger |
| Compliance redaction | ✅ Per-doc gating | ❌ Knowledge baked in |
Decision flowchart
- Does the task require fresh / changing knowledge? → RAG.
- Does the task require strict format, persona, or new skill? → Fine-tune.
- Both? → Fine-tune for behavior + RAG for facts. This is the production pattern at most serious AI companies.
Concrete patterns
- Fine-tune for tool-calling format (lots of base models can be tuned to output your tool-calls more reliably than a general model).
- Fine-tune for domain register (legal-tone English, medical-tone Arabic, IBM internal terminology).
- RAG for the knowledge (every fact, citation, and policy that changes).
- Fine-tune the embedder (with synthetic query-document pairs from your domain) and fine-tune the generator (for output structure) — the embedder fine-tune often gives 5-10% recall lift cheaper than swapping models.
Cost shape
- Fine-tuning: high upfront ($100s-$1000s), low per-query.
- RAG: low upfront, recurring per-query (vector DB + LLM tokens).
- Combined: high upfront + moderate per-query, lowest total error rate.
Evaluation rule
Always eval the same task on (base model + RAG) vs (fine-tuned model + RAG) before claiming fine-tuning is worth it. The lift over a strong RAG baseline is the only number that matters.