Multimodal RAG retrieves over mixed-content corpora: PDFs with figures, product catalogs, slide decks, technical manuals, screenshots.
Two architectural patterns:
Pattern A — Joint embedding space (CLIP-style)
Use a model that embeds both images and text into the same vector space (CLIP, OpenCLIP, SigLIP, Cohere multimodal embed). At index time, embed every image and every text chunk into one index. Query — text or image — retrieves both modalities by similarity.
When this works: product search, figure retrieval, "find the slide with this chart" use cases.
Limitation: CLIP-style models are weak at fine-grained text-in-image (charts, schematics). They embed gist, not detail.
Pattern B — Multimodal LLM as the chunker
Parse the document with a vision LLM (GPT-4o, Gemini, Claude with vision) — describe every figure, table, and chart in text — then index those descriptions plus the original images. Retrieve text descriptions, return the original image to the user.
When this works: technical documentation, scientific papers, financial reports with charts. Captures detail CLIP misses.
Limitation: Indexing cost scales with image count × vision LLM cost. ~$0.01-0.03 per image at index time.
Practical pipeline
- Use a doc parser (
unstructured,Docling,LlamaParse) that emits{type: text|image|table, content, bounding_box}. - For text: standard RAG.
- For images: vision LLM caption + CLIP embedding (both!). Two indices, two retrieval paths.
- For tables: convert to Markdown rows so they survive chunking; index both row text and a one-line description.
- At generation time, pass the original images to a vision LLM, not just captions — captions are lossy.
Common failures:
- Tables silently break under naive chunking — split mid-row, lose headers. Always parse layout-aware.
- Captions lose scale and units. "Chart shows revenue rising" tells you nothing actionable.