Sparse vs Dense vs Hybrid

Dense embeddings encode meaning. Sparse retrieval encodes exact terms. Both miss things the other catches.

Sparse retrieval

• BM25 — the classical baseline. Term frequency × inverse document frequency, length-normalized. 50-year-old algorithm, still in production at every search company. Great for rare terms, exact codes, named entities.
• SPLADE — sparse but learned. Predicts term importance per query/doc; output is a sparse vector that lives in the BM25 ecosystem but is contextual. Beats BM25 on most benchmarks.

When pure dense fails

• Product codes: "ZX-4471" → embedding has no signal; BM25 finds it.
• Rare entities: company names, codes, file names.
• Acronyms in unusual contexts.
• Negation (sometimes).

When pure sparse fails

• Synonyms ("cancel" vs "terminate").
• Cross-lingual ("how do I cancel?" → an Arabic doc).
• Conceptual queries with no shared vocabulary.

Hybrid search

Run both retrievers in parallel; fuse with Reciprocal Rank Fusion (RRF):

def rrf(rankings, k=60):
    scores = defaultdict(float)
    for r in rankings:
        for rank, doc in enumerate(r):
            scores[doc] += 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

RRF works because it ignores raw scores (which aren't comparable across retrievers) and uses rank only.

Production stack: BM25 (or SPLADE) + dense + RRF + re-ranker. This is what serious search teams ship.

Vector DBs that do hybrid natively

• Weaviate (BM25 + dense in one query)
• Qdrant (sparse + dense via separate indices, fused on query)
• Vespa (battle-tested at large scale)
• Pinecone serverless (sparse-dense hybrid via dotproduct on sparse + dense)

Tuning hybrid

• Don't 50/50 weight by default. Start with RRF (parameter-free).
• If you must weight: tune on your eval set; sparse weight often 0.3-0.5 for prose corpora, higher for codes/entities.