Generic embedding models are trained on web-scale data with a generic similarity objective. Your domain has its own vocabulary, query style, and relevance definition. Fine-tuning the embedder on your domain often beats swapping to a bigger generic model — at a fraction of the inference cost.
When to fine-tune
- Specialized terminology (legal, medical, finance, internal product names).
- Long-tail entity names (your products, your engineers, your codebase symbols).
- Query–document style mismatch (queries are colloquial, docs are formal).
- A tight latency budget that forbids large generic embedders.
How it actually works
Most modern embedders train with contrastive loss on (query, positive doc, negative doc) triples. To fine-tune:
- Synthesize pairs. Use an LLM to generate questions for each chunk in your corpus. Each (chunk, question) is a positive pair. Hard negatives are nearest neighbors of the chunk that don't answer the question.
- Train. A few thousand triples, 1-3 epochs, AdamW. Sentence-Transformers
MultipleNegativesRankingLossorInfoNCEare the defaults.SetFitcan do this in minutes. - Evaluate. Hit-rate@K and nDCG@K on a held-out labeled set. Don't trust train loss.
Matryoshka fine-tuning
Modern recipe: fine-tune with a Matryoshka loss so the resulting embedder still supports truncation to 256 / 512 / 1024 dims. Best of both worlds — domain quality and dimension flexibility.
Realistic numbers
- Synthetic pair generation: $20-100 in LLM calls for a few thousand pairs.
- Training: ~1 GPU-hour on A100 for 7B encoder; minutes for smaller.
- Lift: 5-15% nDCG@10 on domain eval is typical. Sometimes much more on jargon-heavy corpora.
When NOT to fine-tune
- Corpus is tiny (<10K chunks): you'll overfit; query rewriting + hybrid search wins.
- Generic eval already strong: no headroom.
- No labeled eval set: you can't measure lift; don't optimize what you can't measure.
A pragmatic alternative: instruction prefixes
Many modern embedders (E5, BGE) accept instruction prefixes like "Represent this query for retrieving relevant passages:". Just adding the right prefix can match 50% of the lift of a fine-tune for a fraction of the work. Try this first.