Pick a model like an engineer, not like a hype-watcher. Walk these seven questions in order — the first 'no' usually narrows the list to 2-3 candidates.
1. Compliance & data residency
- Can data leave your tenant?
- Can it cross country lines?
- Is indemnity required?
→ No to any: Azure OpenAI private endpoint, IBM watsonx Granite, self-hosted Llama / Qwen / Mistral. → Yes: all options on the table.
2. Task type
- Pure reasoning (math, planning, code) → o3, Claude 4.x extended thinking, DeepSeek R1.
- General Q&A / chat → GPT-4o, Claude Sonnet, Gemini Flash.
- Tool use / agents → Claude Sonnet 4.x, GPT-4o.
- RAG-grounded answers → Cohere Command R+, GPT-4o, Claude Sonnet.
- Multimodal (image, audio, video) → Gemini 3.x Pro, GPT-4o, Claude 4.x with vision.
3. Latency budget
- < 500ms TTFT → small models (Haiku, GPT-4o-mini, Gemini Flash, Llama 8B).
- < 2s TTFT acceptable → mid-tier (Sonnet, GPT-4o).
-
5s ok → frontier reasoning models.
Streaming reduces perceived latency dramatically — first tokens often matter more than total latency.
4. Context window needed
- < 8K → anything.
- 32K-128K → most modern frontier models.
- 200K+ → Claude 4.x, Gemini 3.x Pro.
- 1M+ → Gemini 3.x Pro.
But: long context is expensive and quality often degrades past 128K. Don't reach for it as a substitute for retrieval.
5. Cost at scale
Compute per-task cost with realistic prompt + completion sizes. At 1M daily calls:
- Frontier ($5-15/Mtok) → $1-5K/day.
- Mid-tier ($1-3/Mtok) → $200-1K/day.
- Self-hosted Llama 70B (your hardware) → fixed amortized cost, ~10-100× cheaper at 1M+ daily.
6. Fine-tunability
- OpenAI, Google, Mistral, Cohere offer fine-tuning APIs.
- Anthropic does not (as of writing) for general customers.
- Self-hosted: full LoRA / QLoRA / DPO control.
If you need persona, format, or skill specialization → fine-tunable model.
7. Ecosystem fit
- On Vertex / GCP → Gemini.
- On AWS Bedrock → Claude, Llama, Mistral, Cohere all available.
- On Azure → OpenAI primarily, growing Mistral and Llama.
- On IBM Cloud → watsonx Granite, also hosts Llama, Mistral.
- Self-hosted → vLLM, TGI, TensorRT-LLM.
The cascade pattern
In production, don't pick one model. Cascade:
- Cheap classifier model decides if a query is simple → answer with mini model.
- Mid-tier handles 80% of real queries.
- Frontier model only on hard / multi-hop / explicit-escalation queries.
Examples: LiteLLM + custom router, RouteLLM, Martian Router, or a hand-rolled cascade. Cost reductions of 60-90% with quality loss <5% are routine.