Almost every model you'll touch — GPT, Claude, Gemini, Llama, Mistral, Qwen, Granite, BGE, E5 — is built from the same primitive: the transformer block. Knowing it cold separates you from people who just call APIs.
The block
Pre-LayerNorm formulation (the modern default — Llama, Mistral, Qwen all use this):
y1 = x + MultiHeadSelfAttention(Norm(x))
y2 = y1 + FFN(Norm(y1))
Two sub-layers. Each wrapped with a residual connection (x + ...) and a normalization. That's the whole block. Everything else — RoPE, GQA, sliding-window attention, MoE — is a variation on these two pieces.
Self-attention, the actual math
Inputs: a sequence of N token vectors, X ∈ ℝ^{N × d}.
For one attention head (the model has many, in parallel):
Q = X · W_Q (N × d_head)
K = X · W_K (N × d_head)
V = X · W_V (N × d_head)
scores = (Q · K^T) / sqrt(d_head) (N × N)
weights = softmax(scores, dim=-1) (N × N, rows sum to 1)
out_h = weights · V (N × d_head)
The matrix weights[i, j] is the answer to "how much does token i pay attention to token j?". out_h[i] is a weighted sum of the value vectors.
Why /sqrt(d_head)
Without it, dot products grow with d_head, pushing softmax into a near-one-hot regime where gradients vanish. The scaling keeps softmax in a healthy range across model sizes.
Why three projections (Q, K, V)
Q says "what am I looking for?", K says "what do I expose?", V says "what do I share if you choose me?". Decoupling them lets each token selectively read from others — which is the entire information-flow mechanism of the model.
Causal mask (decoder-only models)
For autoregressive models (GPT, Llama), scores[i, j] for j > i is set to -∞ before softmax. Token i can only see tokens 0..i. This is what makes generation valid: token N+1 only sees the past.
Multi-head — parallel attention
A single head learns one kind of relationship. Real models run h heads in parallel, each with d_head = d / h:
out = concat(out_1, out_2, ..., out_h) · W_O (N × d)
Different heads learn different specialisations: copy heads (mostly attend to a single past token), induction heads (n-gram-like patterns), syntactic heads, semantic heads. Mechanistic interpretability is the field that traces individual head behaviors.
Typical: h ∈ {12, 16, 32, 64}. d=4096, h=32, so d_head=128.
Grouped-Query Attention (GQA) — the modern default
Llama-2 70B and beyond, Mistral, Qwen all use GQA: keep h query heads, but share K and V across groups of, say, 4 query heads. Same expressivity in practice; ~4× less KV-cache memory at inference. Multi-Query Attention (MQA) is the extreme — one K/V head shared across all queries.
The FFN — the bigger half of the parameters
FFN(x) = activation(x · W1 + b1) · W2 + b2
W1: d → 4d, W2: 4d → d. The 4× expansion is convention; SwiGLU variants (Llama, Qwen) use d → 8d/3 with a gating mechanism for the same effective parameter budget.
The FFN has no positional or cross-token interaction. It's applied independently to every token. Anthropic's "MLP in transformers as key-value memories" framing is useful here: think of FFN as a giant lookup table that pattern-matches on the contextual token vector and writes back useful information.
In a typical Llama-3 8B: ~70% of parameters live in FFNs.
Residuals — the highway
x + AttnBlock(x) and x + FFN(x) mean the original signal is preserved unless the sub-layer chooses to add. The "residual stream" framing (Anthropic) is now standard: the d-dim vector that flows through the depth of the network is a bus, and each block reads from it (via attention/FFN inputs), computes a delta, and writes it back. Different parts of the residual stream are "sub-spaces" that different heads and FFN neurons read/write.
This is why ablation experiments work — zero out a head's contribution and the rest of the model still functions; the residual just doesn't get that head's update.
Normalization
LayerNorm: (x - mean(x)) / std(x) · γ + β. Stabilizes training. Has trainable scale γ and shift β.
RMSNorm: x / sqrt(mean(x²) + ε) · γ. No mean centering, no β. Cheaper, no quality loss in practice. Llama, Mistral, Qwen use RMSNorm.
Pre-LN vs Post-LN: Pre-LN (norm before sub-layer) is more stable at depth and is now the universal default for >100M-param models.
Positional information
Self-attention is permutation-equivariant. Position must be injected somehow.
- Absolute embeddings (BERT, original Transformer): a learned matrix added to token embeddings at the input. Caps at
max_len. - Sinusoidal (Vaswani et al.): closed-form sin/cos at multiple frequencies. No parameters. Generalizes weakly to longer contexts.
- RoPE — Rotary (Su et al. 2021): instead of adding a vector to
X, RoPE rotates Q and K vectors at each attention step by an angle proportional to position. Two consequences: (1) the dot productQ_i · K_jbecomes a function of(i - j)only — relative position emerges naturally; (2) the model extrapolates to longer contexts (with techniques like YaRN, NTK scaling) better than absolute schemes. Used by every modern open-weight model. - ALiBi: adds a linear bias to attention scores based on distance. Fast, simple, less popular than RoPE in practice.
KV cache — the inference shortcut
At training time, you compute Q, K, V for all N tokens once. At inference time during generation, you generate one new token per step. Naïvely you'd recompute everything. Instead, you cache the K and V tensors of all past tokens, and at step t+1 you only compute Q for the new token. Attention becomes Q_t · K_{0..t}^T.
KV cache size for one sequence:
KV bytes = 2 (K + V) × n_layers × n_kv_heads × d_head × seq_len × dtype_bytes
For Llama-3 70B at FP16, 8K context, that's ~1.6 GB per concurrent user. This is why long-context inference is GPU-memory bound, not compute bound. GQA cuts this by ~4–8×; vLLM's PagedAttention pages it like an OS pages memory.
Encoder-only vs decoder-only vs encoder-decoder
- Encoder-only (BERT, RoBERTa): bidirectional attention. Each token sees all other tokens. Good for classification, embeddings, NER. Cannot generate.
- Decoder-only (GPT, Llama, Claude, Mistral): causal mask. Each token sees only the past. Generates token by token. Can also embed via mean/last-token pooling — which is why E5-mistral (a decoder-only embedder) exists.
- Encoder-decoder (T5, BART): encoder ingests input fully; decoder generates output autoregressively while cross-attending to encoder outputs. Strong for translation, summarization. Less popular for general LLMs because decoder-only with longer context handles the same workload at lower complexity.
The industry consolidated on decoder-only because one architecture handles generation, embeddings (with re-purposing), and classification (via prompting) — operational simplicity wins.
Putting it together
A modern LLM = N transformer blocks (N = 32 for 7B, ~80 for 70B), each block = (RMSNorm + GQA-with-RoPE + residual) → (RMSNorm + SwiGLU FFN + residual). Stack, apply final norm + linear projection back to vocab size, softmax to get next-token probabilities. That's it. The intelligence comes from the parameters, not the architecture.