Every LLM and every text encoder you'll ever use starts with the same step: split a string into a sequence of integer IDs. That step is tokenization, and the algorithm matters more than most engineers realize.
Why subword tokenization exists
Two extremes are bad:
- Character-level. Every character is a token. Sequences explode in length, so attention cost (
O(n²)) goes through the roof. The model wastes capacity learning that "t-h-e" tends to mean "the". - Word-level. Every whitespace-separated word is a token. Vocabulary explodes (English alone has hundreds of thousands of inflections, names, typos), and any unseen word becomes
[UNK]— catastrophic for any specialised domain.
Subword tokenization is the compromise: common chunks are single tokens, rare or novel sequences fall back to multiple tokens, nothing is [UNK].
Byte-Pair Encoding (BPE) — the workhorse
BPE was a 1994 compression algorithm. Sennrich et al. 2016 brought it to NLP. GPT, Llama, Mistral, Qwen, DeepSeek all use BPE variants. Algorithm:
- Start with vocabulary = all individual characters (or bytes — see byte-level below).
- Tokenize the training corpus into characters, with word-boundary markers.
- Count every adjacent pair of tokens.
- Merge the most frequent pair into a single new token. Add it to the vocabulary.
- Repeat steps 3–4 until you hit your target vocab size (typically 32K – 256K).
Worked example. Corpus = "low low low low low lower lower newest newest newest newest newest newest widest widest widest".
After enough iterations, common pairs like l+o, lo+w, new+est, wid+est, +er, +est get merged. "newest" may end up as a single token; "widest" likely two tokens (wid, est); a novel word like "oldest" would be old, est — falling back to subwords gracefully.
Byte-level BPE — what GPT actually does
Plain BPE has a bootstrap problem: you need an initial alphabet. For unicode that's 100K+ characters. Byte-level BPE (Radford et al. 2019, GPT-2) uses the 256 raw bytes as the alphabet. Now any unicode codepoint, including emoji, Arabic, Chinese, code, base64, raw bytes — anything — can be represented losslessly. UTF-8 multi-byte characters decompose into multiple byte tokens until the merge process stitches them.
Consequence: a single Arabic character is typically 2–3 bytes, so it costs 2–3 BPE tokens before merges have a chance to compress it. This is why Arabic, Chinese, Japanese, Korean, and Thai prompts cost 2-4× more tokens than English of the same semantic length — even though "translate this" sentence is the same idea.
WordPiece — BERT's variant
WordPiece (Schuster & Nakajima 2012, Devlin et al. 2018) is BPE with a different merge criterion: instead of picking the most frequent pair, pick the pair that maximizes the likelihood of the training data given the current vocabulary. Marginal quality gain in practice; same shape of algorithm.
Subword pieces inside a word are prefixed with ##: "unhappy" → ["un", "##happy"]. BERT, DistilBERT, MobileBERT use WordPiece.
SentencePiece — language-agnostic
BPE and WordPiece both presuppose whitespace tokenization as a pre-step. Useless for Japanese (no spaces), Chinese (no spaces), Thai (no spaces). SentencePiece (Kudo & Richardson 2018) treats the raw string including spaces as input, with the space encoded as ▁. "hello world" becomes one token stream where the leading space of "world" is part of the ▁world token.
T5, Llama, Gemma, Mistral all use SentencePiece (specifically the unigram-LM variant, which is a probabilistic alternative to BPE merges). Llama's tokenizer is SentencePiece BPE.
What this means in practice
Token count ≠ word count
A 1000-word English document is roughly 1300–1400 GPT tokens. A 1000-word Arabic document is closer to 3000–4000 tokens. Your cost model breaks if you assume otherwise.
Tokenizer choice matters for code
Python is well-represented in modern tokenizers. def, return, self are usually single tokens. Less popular languages get more tokens per character. SQL keywords are often single tokens; obscure DSLs are not.
Whitespace tokens are sneaky
" function" and "function" are different tokens in most BPE tokenizers (the leading space is part of the token). Models attend to this; chat templates depend on it; if you build prompts by string concatenation, you can silently change the tokenization and the model behavior.
You CANNOT trust string length for budgeting
Always tokenize. Tools:
tiktoken(Python) — fast Rust-backed tokenizer for OpenAI models.tiktoken.encoding_for_model("gpt-4o").@dqbd/tiktoken(JS/TS) — Tiktoken bindings.tokenizers(Hugging Face, Rust) — for any HF model.sentencepiecefor Llama / Gemma / T5.
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
n = len(enc.encode("How does this tokenize?")) # → 6
Tokenizer pitfalls in retrieval
- Two embedders trained with different tokenizers cannot share vector spaces — even if both have 1024 dims, the spaces are uncomparable.
- Re-tokenize your corpus when you switch embedders. Don't reuse old chunk boundaries; the token counts will be wrong.
- For multilingual corpora, pick an embedder whose tokenizer was trained with that language meaningfully represented (BGE-M3, Cohere multilingual). Otherwise short Arabic chunks become long, expensive token sequences.