LLM costs sneak up. The first time a startup gets a $40K monthly bill from OpenAI is rarely the last. Lever the 80/20.
The cost surface
- Input tokens. Usually 70-90% of cost in RAG-heavy apps because retrieved context dwarfs queries.
- Output tokens. 3-5× input price; often the smaller share by volume but expensive per token.
- Tool calls. Each tool call is a round-trip → input + output tokens.
- Embeddings. Usually small but at very high volume (millions of docs) becomes meaningful.
- Re-rankers. Per-query cost; can dominate at high QPS.
The biggest levers (in order)
1. Prompt prefix caching
Anthropic / OpenAI / Gemini cache long static prefixes. RAG system prompts + tool schemas = the perfect candidate. Up to 90% input cost reduction. This is free money — implement first.
2. Model cascade
Cheap model for easy queries, frontier for hard. 60-90% cost cuts typical at scale.
3. Output budget
Most prompts don't need 4096 tokens of output. Set max_tokens aggressively. A 200-token cap on extraction tasks vs default 4096 is a 20× saving on the output side.
4. Compression / truncation
- Drop retrieved docs you didn't use.
- Compress conversation history (summary > raw).
- Truncate redundant tool schemas.
5. Batching
- OpenAI Batch API: 50% off for non-urgent processing.
- Anthropic Message Batches API: similar.
- Async eval, summarization, classification → batch.
6. Right-size embedders
- Matryoshka truncation: 1536-dim → 256-dim, often <1% quality loss, 6× cheaper similarity.
- Self-host BGE-M3 if vendor embedding bills exceed $1K/mo.
7. Retrieve smaller
- Top-3 with re-rank > top-30 raw.
- Better retrieval = fewer tokens stuffed into context.
What to monitor
- $/active-user/day (or per session, per task).
- Cost split by feature, by model, by tenant.
- Cost-per-task trend (alert on drift).
- Cache hit rate (prompt cache and semantic cache).
Anti-patterns
- No max_tokens. Default of 4096 vs needed 200 — 20× waste.
- Re-embedding the same docs. Implement deduplication and content-hashing.
- Logging full prompts to vendor analytics. You're paying twice.
- No model tier strategy. Frontier-only is the most expensive plausible architecture.
- No caching. "We don't have repeat queries" — yes you do; system prompts repeat every call.
Realistic targets
A well-engineered RAG app:
- $0.001-0.01 per question (depending on context size and model).
- Prompt cache hit rate 60-90%.
- 80% of traffic on mini/small models, 20% on frontier.
If you're 10× higher than these, you have headroom.