In production, a single model is rarely optimal. Routing picks the right model per query.
Cascade pattern
A waterfall:
- Tiny classifier (small model or rule) decides query type / difficulty.
- Cheap model attempts the answer.
- Confidence check — if low, escalate.
- Frontier model handles only escalated cases.
Real cost: typically 60-90% reduction with <5% quality loss.
Routing patterns
- By query type: Code → Codestral / Claude. Math → o3 / R1. Translation → multilingual specialist.
- By query length: Short factoid → mini model. Long synthesis → frontier.
- By user tier: Free users → cheap. Paid → frontier.
- By confidence: Cheap model emits a confidence; if below threshold, escalate.
Frameworks
- LiteLLM — single SDK across 100+ providers, native routing rules. Default abstraction in any serious system.
- RouteLLM (LMSYS) — learned router, distills GPT-4-quality routing into a small model.
- Martian Router — managed routing service.
- OpenRouter — vendor-neutral API for many providers.
A minimal custom router
def route(query: str) -> str:
# rule-based fast path
if "<code>" in query or query.startswith("def "): return "claude-sonnet-4"
if len(query) < 200 and "?" in query: return "gpt-4o-mini"
# learned classifier fallback
label = small_classifier.predict(query)
return ROUTE_MAP[label]
When NOT to cascade
- Latency-critical: every escalation hop adds 1-2s.
- Low volume: engineering cost > savings below ~10K queries/day.
- High variance in quality is unacceptable (legal, medical) — pick one strong model and ride it.
Eval routing itself
A bad router is worse than no router. Track:
- Misroute rate — frontier-needed queries answered by cheap model. (Quality cost.)
- Wasted-frontier rate — easy queries answered by frontier. (Cost cost.)
- End-to-end task accuracy — routing should preserve this.
The IBM-flavored variant
For regulated environments, route by data-sensitivity not just difficulty:
- Sensitive data → on-prem Granite or self-hosted Llama only.
- Non-sensitive → vendor frontier allowed.
- Some queries can't be routed externally regardless of difficulty.