Now that we have token IDs, here's what actually happens to produce the 1024-dim vector you cosine-compare.
Step 1 — Token embedding lookup
The encoder holds an embedding matrix E ∈ ℝ^{V × d} where V is vocab size (e.g. 250K) and d is hidden dim (e.g. 1024). A token ID t maps to E[t] — a single row, a vector in ℝ^d. So a sequence of N tokens becomes a matrix X ∈ ℝ^{N × d}.
Concretely for a 9-token query and a model with d=384:
X.shape == (9, 384)
Step 2 — Positional information
Self-attention is permutation-equivariant on its own — it has no notion of order. Position information is added (or applied) explicitly:
- Absolute positional embeddings (BERT, original Transformer): another matrix
P ∈ ℝ^{max_len × d}, added toX. Cap atmax_len. - Sinusoidal (Vaswani et al. 2017): closed-form sin/cos, no parameters.
- RoPE — Rotary Position Embeddings (Su et al. 2021): rotates the query/key vectors at each attention step by an angle proportional to position. No additive vector. Composes well with longer contexts. Used in Llama, Mistral, Qwen, BGE-M3, and modern encoders.
After step 2, you still have X ∈ ℝ^{N × d} — but now position-aware.
Step 3 — N transformer encoder blocks
Each block does the same two operations, with residual connections and norm:
y1 = x + MultiHeadSelfAttention(LayerNorm(x))
y2 = y1 + FFN(LayerNorm(y1))
(Pre-LN is the modern default; original BERT used Post-LN.)
Self-attention, the precise formula
For each head:
Q = X · W_Q (shape N × d_head)
K = X · W_K (shape N × d_head)
V = X · W_V (shape N × d_head)
scores = Q · K^T / √d_head (shape N × N)
weights = softmax(scores, axis=-1) (shape N × N)
out = weights · V (shape N × d_head)
weights[i][j] is "how much token i attends to token j". The /√d_head scaling keeps softmax in a healthy gradient regime.
Multi-head runs h heads in parallel with smaller d_head = d / h, concatenates their outputs back to size d, and applies a final projection W_O. h is typically 8, 12, 16, or 32.
The FFN
FFN(x) = activation(x · W1) · W2 (W1: d → 4d, W2: 4d → d)
This is just a 2-layer MLP applied independently to each token position. Activation is GELU (BERT), SiLU/SwiGLU (Llama). The 4× expansion is convention; some modern models use different ratios.
After all N blocks (12, 24, 36, depending on the model), X is still shape (N, d) — but now each row is a contextual representation: every token has been allowed to look at every other token, multiple times.
Step 4 — Pooling
You came in with N token vectors. You need one vector for the sentence. Four common choices:
CLS pooling
Prepend a special [CLS] token to every input. After the final layer, take X[0] as the sentence embedding. Default for BERT-family bi-encoders. Works because the model is trained so that [CLS] aggregates information.
Mean pooling
Average all token vectors (or all non-padding token vectors): mean(X, axis=0). Default for Sentence-Transformers, BGE-M3, E5. More forgiving than CLS — it doesn't require a single token to learn the aggregation role.
Last-token pooling
Take X[-1] (the final token). Used by some decoder-only embedders (E5-mistral, Voyage). Decoder-only models are causal, so only the last token has seen everything.
Attention pooling
A small extra layer learns weights to combine token vectors. Strongest, slightly more expensive.
Step 5 — L2 normalize
v = pool(X) / ||pool(X)||. Now ||v|| = 1, and cosine similarity becomes a plain dot product. Most modern embedders normalize automatically.
Worked dimensions, BGE-M3 (open weights, multilingual)
- vocab V = 250 002 (XLM-RoBERTa tokenizer)
- d = 1024
- 24 transformer blocks
- 16 attention heads, d_head = 64
- FFN inner dim = 4096
For a 32-token query:
- Token embedding:
(32, 1024) - After 24 blocks: still
(32, 1024)— but contextual - After mean pooling:
(1024,) - After L2 normalize:
(1024,)with norm 1.0
That's the vector you put in your index.
Instruction prefixes — how they really work
E5, BGE, GTE accept queries prefixed like "Represent this query for retrieving relevant passages: <query>". There is nothing magical — those tokens just go through the same forward pass. They condition the contextual representations of the real query tokens via attention, and the pooled vector ends up in a region of space the model was trained to put queries in. The instruction is doing the same job as a system prompt does for an LLM: shifting the activation distribution.
Documents get a different prefix ("Represent this passage for retrieval: ..."). Query and document end up in compatible-but-asymmetric regions, mirroring how cross-encoders distinguish (query, doc) pairs.
Why bi-encoders are fast at retrieval
Bi-encoder = independent forward pass for the query and for each document. Documents are pre-computed; only the query is encoded online. One forward pass per query, then ANN search over pre-computed vectors. Cross-encoders, by contrast, need a fresh forward pass per (query, doc) pair — fine for re-ranking 50 candidates, dead at retrieving from 10M.