An embedding is a learned function f: text → ℝ^d such that distance(f(a), f(b)) correlates with semantic similarity of a and b.
That's it. The whole field hangs on that one property.
What "distance" actually means
- Cosine similarity (
a·b / (||a|| ||b||)) — angle between vectors. Default for normalized vectors. Range [-1, 1]. - Dot product (
a·b) — same as cosine if vectors are L2-normalized; otherwise magnitude leaks into the score. Most modern embedding models output normalized vectors, making cosine and dot product equivalent. - Euclidean distance (
||a-b||) — sensitive to magnitude. Rarely the right metric for text.
Practical rule: if your vectors are normalized (most modern models), use cosine = dot product and pick whichever your vector DB optimizes for (Pinecone uses cosine, FAISS often uses dot product on normalized vectors).
Dimensionality
- Smaller (384, 768) → faster, cheaper, may underfit niche domains.
- Larger (1024, 1536, 3072) → more capacity, more cost, may overfit small corpora.
- Matryoshka embeddings (Kusupati et al. 2022) let you truncate a 1536-dim embedding to 256 dims and still get most of the quality. Used by OpenAI
text-embedding-3-*. This is non-negotiable for cost-conscious deployments.
What lives in vector space
Modern text embeddings encode roughly: topic, named entities, sentiment polarity, syntactic patterns, language identity (in multilingual models). They do not reliably encode: numbers, dates, negation. ("Cat is alive" and "Cat is not alive" are often >0.9 cosine similar.)
Test your embeddings on edge cases your application cares about. Don't trust MTEB rank for negation, numerical reasoning, or your specific domain.