About
CV
Projects
Sports
Blog
← Back to blog
RAG and Evaluation

Multimodal RAG Production Playbook: Documents, Images, and Tables

A production-oriented design guide for parsing, indexing, retrieving, citing, and evaluating text, images, and tables in a multimodal RAG system.
April 11, 20264 min readMultimodal RAGDocument AI
A PDF is not a sequence of clean paragraphs. It can contain multi-column text, screenshots, diagrams, footnotes, repeated headers, and tables whose meaning depends on row and column structure. Flattening all of that into plain text destroys evidence before retrieval even begins. This article is an architecture playbook, not a claim that I have shipped the exact multimodal stack described below. It extends the evidence and citation discipline used in my public RAG Equity Research Agent case study to documents where meaning is distributed across text and visual elements.

Preserve document structure during ingestion

The first objective is not chunking. It is producing a reliable document model. For every extracted element, preserve:
  • document, page, and section identity;
  • element type, such as paragraph, title, image, caption, or table;
  • bounding box and reading order when available;
  • relationships between figures, captions, and surrounding text;
  • table structure, including headers and merged cells;
  • source version and ingestion timestamp;
  • access-control metadata.
Use separate extraction paths for native PDFs, scanned pages, images, and office documents. OCR should be a fallback for image-based content, not an automatic replacement for embedded text. Extraction confidence and parser warnings belong in the metadata so weak pages can be reviewed or handled conservatively. Repeated headers, footers, and navigation elements should be removed without losing section titles. A layout-aware parser can preserve reading order, but it still needs tests for the document families that matter to the product.

Create representations for each evidence type

One representation rarely serves every query. I use element-specific views:
  • clean text for semantic retrieval;
  • structured Markdown or JSON for tables;
  • captions or generated descriptions for images and diagrams;
  • optional visual embeddings for queries that depend on appearance;
  • compact parent summaries to recover surrounding context.
The original element remains the source of truth. Generated descriptions are retrieval aids, not authoritative evidence. Store their model and prompt version so they can be regenerated and evaluated. Tables deserve special care. Row serialization may work for lookup questions, while comparisons often need headers, units, and multiple rows together. Keep a structured representation and link every serialized chunk back to the original table and page.

Retrieve candidates in stages

Multimodal retrieval benefits from a staged pipeline:
  • classify the query's likely evidence needs;
  • apply tenant, document, date, and permission filters;
  • retrieve candidates from the relevant text, table, and image indexes;
  • merge and deduplicate results;
  • rerank across modalities with a shared relevance policy;
  • expand selected elements with their parent section or neighboring context.
Routing should not prevent recovery. A query that appears textual may still require a diagram. Keep a fallback path that broadens retrieval when confidence is low or the first pass lacks sufficient evidence. Hybrid retrieval remains useful. Exact identifiers, product codes, and table labels often favor lexical search, while conceptual questions benefit from embeddings. The ranking policy should preserve source diversity when several independent pieces of evidence are required.
Multimodal RAG pipeline from layout-aware ingestion to grounded answer generation
The pipeline preserves element identity from parsing through retrieval so the final answer can point back to the original page, figure, or table.

Generate answers with precise citations

The generation layer should receive a compact evidence package rather than a dump of every candidate. Each element needs a stable citation identifier and enough metadata to render a useful source reference. The answer policy should require the model to:
  • distinguish text evidence from inferred interpretation;
  • cite the page and element supporting each important claim;
  • preserve units, time periods, and table headers;
  • state when a visual is unreadable or ambiguous;
  • avoid answering when the evidence package is insufficient.
For sensitive workflows, a deterministic post-processing step can verify that citations exist, refer to retrieved elements, and respect access controls. It cannot prove that the answer is correct, but it prevents several avoidable failure classes.

Evaluate by modality and failure layer

A single answer-quality score is not enough. Build test cases for:
  • native and scanned documents;
  • multi-column pages and footnotes;
  • charts with legends and units;
  • tables with merged headers or missing cells;
  • questions requiring text plus a visual;
  • conflicting evidence across pages;
  • documents with extraction or OCR errors.
Measure extraction fidelity, candidate recall, reranking quality, citation correctness, grounded answer quality, and safe fallback separately. When a response fails, the team should know whether the parser lost the content, the retriever missed it, the reranker demoted it, or the model misread good evidence. The production RAG blueprint explains the same layered failure taxonomy for text-first systems. Multimodal RAG adds more evidence types, but it does not remove the need to isolate each failure.

Plan for reprocessing and operations

Parsers, OCR models, descriptions, and embeddings will change. Store lineage so a team can reprocess only the affected layer rather than rebuilding the corpus blindly. Use versioned indexes or aliases to validate a new pipeline before switching production traffic. Monitor ingestion failures, parser warnings, index freshness, retrieval by modality, citation usage, latency, and fallback reasons. Protect raw documents and extracted content with the same access policy, including temporary artifacts and observability traces. A pragmatic rollout begins with one or two representative document families and a real question set. Validate extraction manually, establish retrieval baselines, add answer generation only after evidence quality is understood, then expand modalities and document types. In multimodal RAG, reliable ingestion is not preprocessing detail. It is the foundation of answer quality.

Sources and references

  1. LlamaIndex documentationMultimodal indexing and retrieval concepts
  2. Unstructured documentationDocument parsing, partitioning, and element extraction
  3. FAISS documentationVector search foundations and indexing tradeoffs

From principles to shipped systems

These articles document the methods behind my work. The project case studies show how I apply them across enterprise agents, RAG, and MLOps.
Explore project case studiesView my CV

Continue exploring

Related field notes on the architecture, evaluation, and operating decisions behind production AI systems.
April 11, 20264 min readSynthetic DataFine-Tuning

Synthetic Data Pipeline for Domain Fine-Tuning

A controlled pipeline for generating, filtering, versioning, and evaluating synthetic domain data without hiding contamination, policy, or distribution risks.