Inference Internals — KV Cache, Quantization, Speculative Decoding

An LLM's quality comes from its parameters. Its speed and cost in production come from how it's served. The four levers that matter:

1. KV cache — what you actually pay for at long context

Generation is autoregressive: produce token t, then t+1, ad infinitum. Each step, attention needs Q_t · K_{0..t}^T. Naïvely you'd recompute K and V for every past token at every step — O(N²) work per step. Instead, you cache K and V across steps.

Memory cost for one sequence:

KV_bytes = 2 × n_layers × n_kv_heads × d_head × seq_len × dtype_bytes

Examples (FP16, 2 bytes):

• Llama-3 8B: 32 × 8 × 128 × 8192 × 2 × 2 ≈ 1 GB at 8K context.
• Llama-3 70B (GQA, n_kv_heads=8): 80 × 8 × 128 × 8192 × 2 × 2 ≈ 2.5 GB.
• Llama-3 70B at 32K context: ~10 GB per active sequence.

This is per concurrent user. Serving 100 users at 32K means 1 TB of KV state, which is why long-context production almost always means PagedAttention (vLLM) — paging KV blocks like an OS pages memory.

2. Continuous batching + PagedAttention (vLLM)

Naïve batching: pad all sequences to the longest one. Wasted compute, wasted memory.

Continuous batching: as soon as a request finishes generating, slot a new one in mid-batch. No padding, no wait. Fundamental for high throughput.

PagedAttention: KV cache is split into fixed-size blocks; a request's sequence is a list of block pointers. Frees you from contiguous KV allocations, lets you preempt and resume sequences, enables prefix sharing across concurrent requests with the same system prompt.

Together (the vLLM default), they improve throughput 5–20× over naive HF generate().

3. Quantization — fewer bits, more sequences per GPU

Native LLMs train in BF16 (2 bytes/param). Inference doesn't need that precision. Quantization shrinks weights, KV cache, or both:

Realistic quality cliff:

• 8-bit: ~0% quality loss in practice.
• 4-bit (GPTQ/AWQ): 0–2% on most benchmarks. The default for self-hosting open weights.
• 3-bit: noticeable drop, not always acceptable.
• 2-bit: model-specific; usually painful.

A 4-bit Llama-3 70B fits on a single 48GB GPU (A6000, L40S) comfortably; the BF16 version doesn't.

Quantization-aware training (QAT) vs Post-training quantization (PTQ)

PTQ is what you usually mean by "quantize a model" — it's free, calibration-driven, applied after training. QAT trains with simulated quantization in the loop and yields better low-bit results, but you need the training pipeline; few teams do this.

4. Speculative decoding — get multiple tokens per forward pass

Standard decoding: one forward pass per output token. Speculative decoding (Leviathan et al. 2023, Chen et al. 2023): a small draft model speculatively generates k tokens; the target model verifies them in a single forward pass and accepts the prefix that matches what it would have produced. Acceptance rate is typically 60–85% for matched draft/target pairs, so you get ~2–3× speedup with no quality loss.

Pairings that work:

• Llama 3.1 405B target + Llama 3.1 8B draft.
• Qwen 72B target + Qwen 7B draft.
• Self-distillation drafts (Medusa, EAGLE): the target model itself learns multiple speculation heads.

Medusa is a clever variant: instead of a separate draft model, the target model gets extra prediction heads that predict tokens 1, 2, 3 ahead in parallel. Same forward pass, multiple candidate continuations.

5. Other production levers

• Flash Attention v2/v3: an algorithmic re-formulation of attention that's IO-aware (tile and re-compute instead of materializing the N×N attention matrix). 2–4× faster, same numerics. Standard everywhere now.
• TensorRT-LLM (NVIDIA): kernel-level fusion, FP8 support, the fastest serving stack on NVIDIA hardware. More effort to set up than vLLM.
• Speculative + quantization stack together: a 4-bit AWQ Llama-3 70B with a 4-bit Llama-3 8B draft, served via vLLM, is the typical open-weight production recipe in 2026.

TTFT vs TPOT — the two latencies users actually feel

• TTFT (Time To First Token): dominated by prefill — the cost of running the full prompt through one forward pass to seed the KV cache. Scales linearly with prompt length. Matters for chat UX.
• TPOT (Time Per Output Token): dominated by decode — one KV-augmented forward pass per token. Scales (roughly) with model size and concurrency.

You optimise them differently. Long prompts? Prompt caching (Anthropic, OpenAI, Gemini) reuses cached KV state for a static prefix — TTFT drops near-zero on the cached portion. Slow generation? Speculative decoding cuts TPOT.

Putting it together for self-hosting

Production stack for self-hosted open-weight LLM, late 2026:

        ┌────────────────────────────────┐
        │  Client                         │
        └──────────────┬─────────────────┘
                       │  HTTP / gRPC
        ┌──────────────▼─────────────────┐
        │  vLLM / TensorRT-LLM           │
        │  • continuous batching          │
        │  • PagedAttention (KV pages)    │
        │  • prefix caching               │
        │  • speculative decoding (Medusa)│
        │  • FP8 KV / 4-bit weights (AWQ) │
        │  • Flash Attention v3           │
        └──────────────┬─────────────────┘
                       │
                  GPU (H100 / H200 / B200)

For most workloads under 1B tokens/day, vLLM with 4-bit AWQ + speculative is enough. Beyond that, TensorRT-LLM + dedicated FP8 + tuned scheduler is where the marginal wins live.