Latency Optimization

Perceived latency ≠ total latency. Optimize what users feel.

Streaming

The single biggest UX lever. Time-to-first-token (TTFT) often matters more than total time.

• All major providers stream natively.
• Stream tool calls too where supported.
• Front-end: render tokens as they arrive; never buffer whole response unless you must.

Caching

Exact-match caching

Same prompt → same response. Trivial, fast, only catches exact duplicates.

Semantic caching

Embed the query; if a near-neighbor cached entry exists with high similarity, return it.

• Tools: GPTCache, Redis VectorStore, custom.
• Threshold matters: too low → wrong answers; too high → no cache hits.
• Invalidation: TTL + content-version key.

Prompt prefix caching

Many providers (Anthropic prompt caching, OpenAI prompt caching, Gemini context caching) cache long static prefixes — system prompts, retrieved docs, tool schemas. Up to 90% input cost reduction, big latency win.

• Anthropic: cache_control markers; saves $/tok and ~60% latency.
• OpenAI: automatic prompt caching for prefixes ≥1024 tokens.
• Gemini: explicit cachedContents.

Parallelism

• Parallel tool calls — modern APIs support multiple tool calls per turn.
• Parallel chains — independent steps run concurrently (asyncio.gather, Promise.all).
• Speculative parallel routing — call cheap and frontier model in parallel; cancel cheap if frontier comes back better.

Prompt compression

• LLMLingua / LongLLMLingua — neural compression of context, 60-80% reduction with small quality loss.
• Manual: drop irrelevant context, truncate retrieved docs to relevant spans.
• Summarize-then-prompt — summarize long inputs first; helps when context window is the bottleneck.

Smaller / faster models

• Frontier-tier 4o-mini, Haiku, Gemini Flash are 5-10× faster than top-tier.
• For 80% of queries, the smaller model is indistinguishable.
• Combine with cascading.

Co-locate model and data

For self-hosted, GPU near vector DB near app server. Cross-region adds 50-200ms.

Async patterns

• Long-running tasks → return job id, poll or webhook on completion.
• WebSockets / SSE for streaming UX.
• Background processing for non-critical-path work (logging, eval).

Practical optimization order

1. Stream. Always.
2. Prompt cache (Anthropic / OpenAI / Gemini). Free huge win on RAG.
3. Cascade to smaller models for easy queries.
4. Parallel tool calls. Free if you support them.
5. Semantic cache for FAQ-shaped traffic.
6. Compression if context-bound.
7. Self-host if you've maxed everything else.