MM-RAG: Multimodal Retrieval-Augmented Generation

• MM-RAG is a framework that combines multimodal retrieval and generation by aligning text, images, and structured data into a cohesive design space.
• It employs innovative techniques such as adaptive retrieval controllers, cross-modal alignment, and graph-based reranking to optimize evidence selection.
• Applications include image captioning, knowledge-based VQA, long-document QA, and video understanding, while addressing issues like hallucination and modality mismatch.

Multimodal Retrieval-Augmented Generation (MM-RAG) denotes retrieval-augmented systems in which the query, the evidence store, the retrieved context, the generator, or the output is multimodal rather than purely textual. In the recent literature, this includes multimodal retrieval for text generation, multimodal retrieval for multimodal generation, adaptive retrieval controllers that decide whether retrieval should be used at all, and graph- or structure-aware retrievers that model relations among text, images, pages, regions, and layout rather than treating them as isolated nearest neighbors (Hu et al., 29 May 2025, Yu et al., 6 Feb 2025, Li et al., 12 Jan 2026). Across image captioning, knowledge-based VQA, long-document QA, enterprise QA, and video understanding, the central technical problem is consistent: align heterogeneous modalities, retrieve evidence units that are jointly relevant, and condition generation on them without amplifying hallucination, distractor sensitivity, or modality mismatch (Jaiswal et al., 6 Aug 2025, Bhathena et al., 2 Jun 2026).

1. Scope and formal problem definition

A common formalization treats MM-RAG as retrieval over a multimodal document collection $\mathcal{D}$, followed by generation conditioned on the retrieved set. In the M$^2$RAG benchmark, the retriever encodes a query $q$ and a multimodal document $d$ as $h_q = \mathrm{Enc}(q)$ and $h_d = \mathrm{Enc}(d)$, scores them with $S(q,d)=\mathrm{Sim}(h_q,h_d)$, retrieves a top-$k$ set $\tilde{\mathcal{D}}$, and then generates with $y = \mathcal{M}(\tilde{\mathcal{D}}, q)$ (Liu et al., 24 Feb 2025). MRAMG-Bench extends this formulation to multimodal outputs: given a text query $^2$0, retrieved multimodal documents $^2$1, and a foundation model $^2$2, the objective is a multimodal answer $^2$3 that interleaves text and images rather than verbalizing all visual evidence into text (Yu et al., 6 Feb 2025).

This literature therefore uses “MM-RAG” in more than one sense. Some papers reserve the term for multimodal retrieval with text generation; others treat multimodal-output generation as the natural extension of the same paradigm. A plausible implication is that MM-RAG is better understood as a design space than as a single pipeline.

Regime	Representative formulation	Example papers
Multimodal retrieval, text output	Retrieve text/image evidence, generate text	(Hu et al., 29 May 2025, Liu et al., 24 Feb 2025)
Multimodal retrieval, multimodal output	Retrieve multimodal evidence, generate text plus images	(Yu et al., 6 Feb 2025)
Adaptive or structured MM-RAG	Retrieval is gated, fused, or graph/structure-aware	(Du et al., 28 Feb 2026, Li et al., 12 Jan 2026, Bhathena et al., 2 Jun 2026)

At the task level, the field spans image captioning, KB-VQA, document QA, video understanding, multimodal fact verification, image reranking, and enterprise QA. Query modality is similarly heterogeneous: text-only queries over multimodal corpora, image-plus-question queries over Wikipedia-like knowledge bases, visual reverse-image queries over the web, and video/audio streams converted into retrievable memory (Mortaheb et al., 8 Jan 2025, Yang et al., 10 May 2025, Mao et al., 29 May 2025). This breadth is not incidental. It reflects the fact that in MM-RAG, retrieval relevance is often modality-dependent: some questions are text-answerable, some are image-answerable, and some require cross-modal composition that cannot be reduced to a single modality without loss (Liu et al., 24 Feb 2025).

2. Retrieval architectures and cross-modal alignment

A core axis of MM-RAG design is how a query in one modality is made retrievable against evidence in another. One clean example is mRAG-gim, which addresses image-to-text generation by fitting a linear map $^2$4 from CLIP image embeddings to CLIP text embeddings via ordinary least squares: $^2$5 At inference, an unseen image embedding $^2$6 is projected to $^2$7, and top-$^2$8 training captions are retrieved by cosine similarity,

$^2$9

after which the retrieved captions are used as prompt context for FLAN-T5 to generate a new description (Jaiswal et al., 6 Aug 2025). The method is notable because it treats the “modality gap” not as a problem requiring end-to-end multimodal finetuning, but as a lightweight alignment problem over frozen encoders.

A second design pattern is coarse-to-fine retrieval over multiple knowledge granularities. OMGM begins with image-to-summary retrieval over entity summaries generated offline, then performs multimodal reranking over image-section pairs within the top-$q$0 entities, and finally applies text reranking over sections of the selected entity (Yang et al., 10 May 2025). In its second stage, a multimodal encoder computes late-interaction similarity

$q$1

so that image and question jointly refine entity selection before final section extraction. This suggests that “granularity alignment” is as important as modality alignment: coarse image signals may best identify entities, while fine text signals may best identify answer-bearing passages.

A third pattern converts all modalities into a single retrievable modality. Multi-RAG for video understanding adopts what it calls the “single primary modality paradigm”: video frames are described by a VLM, audio is transcribed by Whisper, auxiliary summaries are generated, and all outputs are stored as Markdown text chunks in a Chroma vector store. Retrieval is then purely textual: $q$2 This is a genuine MM-RAG system because the source evidence is multimodal, but the retrievable memory is textualized (Mao et al., 29 May 2025). The broader implication is that MM-RAG need not preserve native modalities at every stage; some systems preserve them, while others normalize them into a common retrieval substrate for simplicity and tooling compatibility.

3. Retrieval control, reranking, and evidence fusion

A recurring result in the literature is that retrieval is not uniformly helpful. MMA-RAG makes this explicit by reframing visual RAG as a retrieval utility estimation problem. Given no-retrieval and retrieval-augmented internal features $q$3 and $q$4, it constructs

$q$5

and predicts one of four outcome classes with

$q$6

The four classes distinguish cases where retrieval is essential, harmful, irrelevant because both paths fail, or redundant because both paths succeed. At inference, the system can adopt a pessimistic or optimistic reverse-image-retrieval policy based on $q$7 (Du et al., 28 Feb 2026). This replaces the standard “always retrieve” assumption with a learned multimodal gate built from the VLM’s own internal states.

A second response to noisy retrieval is evidence fusion rather than evidence gating. BayesRAG models retrieval over tuples $q$8 of text, visual element, and screenshot/page structure, and ranks them with a posterior

$q$9

Here $d$0 is estimated by Dempster-Shafer fusion of modality-specific evidence, while $d$1 is a graph-topology or layout prior capturing whether the tuple is intrinsically coherent (Li et al., 12 Jan 2026). In this formulation, a text result and an image result are not treated as independently good merely because each has high similarity to the query; they must also mutually corroborate each other.

A third approach focuses on reranking and adaptive context size. “Re-ranking the Context for Multimodal Retrieval Augmented Generation” introduces a learned relevancy score $d$2 that re-ranks CLIP-retrieved candidates and applies an adaptive up-to-$d$3 policy with $d$4 and $d$5 (Mortaheb et al., 8 Jan 2025). Rather than always returning a fixed number of context items, the system keeps only candidates whose RS survives thresholding and a median-based filter. This is significant because irrelevant multimodal context is not neutral: it can actively mislead downstream generation.

These mechanisms collectively show that MM-RAG has moved beyond raw nearest-neighbor retrieval. Retrieval utility estimation, tuple-level corroboration, and adaptive reranking all treat evidence selection as a structured decision problem rather than a one-shot similarity lookup.

4. Structured knowledge, graph retrieval, and document-centric MM-RAG

One branch of MM-RAG replaces flat retrieval with graph-structured evidence. MMGraphRAG refines images into scene graphs, constructs a multimodal knowledge graph (MMKG) by combining image-derived and text-derived KGs, and uses spectral clustering for cross-modal entity linking. Its weighted adjacency is defined as

$d$6

after which spectral clustering and LLM-based alignment yield cross-modal entity links (Wan et al., 28 Jul 2025). The system reports $d$7 overall accuracy on DocBench and $d$8 accuracy with $d$9 F1 on MMLongBench, and on unanswerable questions it achieves $h_q = \mathrm{Enc}(q)$0 versus M3DocRAG’s $h_q = \mathrm{Enc}(q)$1 (Wan et al., 28 Jul 2025). The point is not merely better retrieval. It is retrieval over explicit entities, relations, and reasoning paths rather than over isolated page embeddings.

MG$h_q = \mathrm{Enc}(q)$2-RAG pursues a related but more lightweight route. It builds a hierarchical graph over sentences, chunks, images, and objects, fuses textual entities and grounded visual objects into unified multimodal nodes, and performs multi-granularity graph retrieval by aggregating dense similarities and propagating relevance through graph edges. The paper reports an average $h_q = \mathrm{Enc}(q)$3 speedup and $h_q = \mathrm{Enc}(q)$4 cost reduction compared with advanced graph-based frameworks while achieving state-of-the-art performance across E-VQA, InfoSeek, ScienceQA, and CrisisMMD (Dai et al., 4 Apr 2026). This suggests that graph-based MM-RAG need not be synonymous with expensive MLLM-driven triplet extraction.

Document-centric MM-RAG introduces a second kind of structure: document layout and reading order. MM-BizRAG splits enterprise documents into vertically structured and horizontally structured regimes, routes them through different ingestion pipelines, and defines structure-specific representations such as

$h_q = \mathrm{Enc}(q)$5

Its central insight is that retrieval representations and generation context should be decoupled: lean chunks are indexed, but richer multimodal context is reconstructed at inference time through placeholder-based alignment (Bhathena et al., 2 Jun 2026). On a large enterprise dataset and public benchmarks, MM-BizRAG reports gains of up to $h_q = \mathrm{Enc}(q)$6 percentage points over vision-centric baselines, with especially strong improvements on report-style layouts (Bhathena et al., 2 Jun 2026).

A related retrieval-only perspective appears in the multi-granularity framework for visually rich documents, which combines page-level retrieval, region-level retrieval, OCR-text retrieval, layout-aware search, and Qwen2.5-VL reranking, reaching a score of $h_q = \mathrm{Enc}(q)$7 (Xu et al., 1 May 2025). Taken together, these document-centric systems indicate that MM-RAG over PDFs, reports, and slides is increasingly a problem of explicit structural modeling rather than only cross-modal embedding alignment.

5. Generation strategies and multimodal outputs

Even when retrieval quality is high, evidence integration remains a bottleneck. A systematic study of the MM-RAG design space shows that for knowledge-based VQA, retrieval recall and answer quality are not monotonically aligned: with Qwen on E-VQA, retrieval accuracy rises from $h_q = \mathrm{Enc}(q)$8 at top-1 to $h_q = \mathrm{Enc}(q)$9 at top-5 after reranking, yet ROUGE-L drops from $h_d = \mathrm{Enc}(d)$0 to $h_d = \mathrm{Enc}(d)$1, and semantic response accuracy also declines (Hu et al., 29 May 2025). The paper’s conclusion is blunt: providing only the most relevant document is optimal, and listwise reranking followed by top-1 generation is the best zero-shot default (Hu et al., 29 May 2025). This is one of the clearest empirical statements in the literature that “more retrieved evidence” can degrade MM-RAG generation.

Generation design also matters on the training side. The M$h_d = \mathrm{Enc}(d)$2RAG benchmark shows that vanilla multimodal in-context RAG often improves only modestly with top-1 or top-3 retrieval and can degrade at top-5, while MM-RAIT—multi-modal retrieval-augmented instruction tuning—substantially improves image captioning, multimodal QA, fact verification, and image reranking by fine-tuning the generator to operate over retrieved multimodal context (Liu et al., 24 Feb 2025). The implication is that MM-RAG quality is constrained not only by retrieval but also by the model’s ability to exploit retrieved multimodal evidence rather than falling back to internal priors.

At the output side, MRAMG-Bench argues that retrieval-augmented systems should sometimes return multimodal answers rather than text-only answers. It evaluates interleaved text-and-image outputs and decomposes answer quality into image precision, image recall, image ordering, image position, textual similarity, and comprehensive answer quality (Yu et al., 6 Feb 2025). Across 11 generators, the broad pattern is $h_d = \mathrm{Enc}(d)$3 on average, while image ordering remains poor for all systems, particularly on recipe and manual-like documents (Yu et al., 6 Feb 2025). This suggests that multimodal answer planning—how many images to use, which ones to place, and where—remains underdeveloped relative to retrieval.

mRAG-gim adds a captioning-specific variant of retrieval-conditioned generation. After retrieving similar captions from a text datastore with a mapped visual query, it prompts FLAN-T5 with a fixed instruction of the form "Show similar images: [retrieved captions] The image describes: " and further refines the datastore through synthetic-caption filtering, with CIDEr-D identified as the best criterion for synthetic-data acceptance (Jaiswal et al., 6 Aug 2025). This makes MM-RAG generation explicitly retrieval-conditioned even when the target output is still caption-like text.

6. Evaluation, uncertainty, and security

As MM-RAG has diversified, evaluation has become a research problem in its own right. MiRAGE reframes multimodal RAG evaluation as subclaim-level verification and attribution, introducing InfoF1 for information quality and CiteF1 for citation quality in multimodal settings such as article generation from multiple videos (Martin et al., 28 Oct 2025). Its central methodological claim is that reference overlap is insufficient: one must measure whether generated subclaims are supported by multimodal sources and whether cited sources actually ground those claims (Martin et al., 28 Oct 2025).

MRAG-Suite extends this diagnostic view to Visual RAG benchmarks by introducing difficulty-based and ambiguity-aware filtering and a claim-level checker, MM-RAGChecker. On its curated ambiguity subset, ambiguous queries reduce EM/Accuracy by $h_d = \mathrm{Enc}(d)$4, increase hallucination by $h_d = \mathrm{Enc}(d)$5, reduce Claim Recall by $h_d = \mathrm{Enc}(d)$6, and reduce Context Precision by $h_d = \mathrm{Enc}(d)$7 relative to matched clear queries (Ji, 29 Sep 2025). The paper’s broader finding is that hard, ambiguous, and distractor-rich queries expose failures that pooled benchmark scores can hide.

A parallel line of work studies answer reliability. LeMUQ addresses uncertainty quantification for MM-RAG by rescoring the same generated answer under four input configurations—full input, no image, no context, and question only—and feeding the resulting token-probability traces into a learned scorer: $h_d = \mathrm{Enc}(d)$8 Across datasets, retrievers, and VLMs, it improves AUROC by $h_d = \mathrm{Enc}(d)$9 on average over a finetuned LARS baseline (Binz et al., 28 May 2026). This is a strong indication that uncertainty in MM-RAG is intrinsically multimodal and retrieval-aware rather than reducible to answer text alone.

Security work shows that grounding does not guarantee robustness. MM-MEPA presents a metadata-only poisoning attack against MM-RAG knowledge bases. Given a retrieval score

$S(q,d)=\mathrm{Sim}(h_q,h_d)$0

it optimizes malicious metadata to maximize query alignment while maintaining image-metadata cohesion (Edemacu et al., 26 Feb 2026). Across four retrievers and two multimodal generators, the attack achieves up to $S(q,d)=\mathrm{Sim}(h_q,h_d)$1 attack success rate, and representative defenses such as query paraphrasing and simple image-metadata consistency checks are largely ineffective (Edemacu et al., 26 Feb 2026). This establishes metadata as a first-class MM-RAG attack surface.

The cumulative lesson of this evaluation literature is that MM-RAG cannot be assessed only by end-task accuracy. Factual support, citation quality, ambiguity sensitivity, confidence calibration, distractor robustness, and poisoning resistance are all now measurable failure modes. This suggests that future MM-RAG systems will increasingly be judged not only by whether they retrieve and generate, but by whether they know when retrieval is useful, whether they can prove what they used, and whether their external knowledge interface is robust under adversarial conditions.