Multimodal Retrieval-Augmented Generation

• MRAG is a computational paradigm that integrates diverse modalities—text, images, tables, charts—into generative models, enabling enriched and factually-grounded outputs.
• It employs modular architectures with specialized retrieval, iterative planning, and cross-modal synthesis to reduce hallucination and improve interpretability.
• MRAG enables advanced applications in scientific document analysis, technical assistance, and dynamic content delivery across a range of real-world domains.

Multimodal Retrieval-Augmented Generation (MRAG) is a computational paradigm that integrates multimodal retrieval—encompassing text, images, tables, charts, video, and additional formats—into the generative process of large (multimodal) LLMs (MLLMs). Unlike standard RAG, which restricts retrieval and output to the textual modality, MRAG leverages diverse evidence sources for grounding, reasoning, and content synthesis, supporting rich multimodal outputs essential for contemporary real-world applications such as technical assistance, educational content delivery, scientific document understanding, and advanced question answering.

1. Foundations: Paradigm and Motivation

MRAG generalizes text-only Retrieval-Augmented Generation (RAG) by broadening both the evidence retrieval and answer composition spaces to multiple modalities. This evolution is motivated by limitations observed in monomodal RAG—specifically, its inability to access, reason with, or generate outputs that faithfully reference visual, tabular, or otherwise non-textual artifacts. Empirical studies indicate that MRAG frameworks reduce hallucination, increase factuality, and outperform text-only systems in tasks demanding cross-modal understanding or when original information is strongly encoded in non-textual sources (Mei et al., 26 Mar 2025).

The MRAG paradigm is operationalized via an external knowledge base (KB) of multimodal data (e.g., image–text pairs, charts), a cross-modal retriever, and a generative model capable of ingesting the retrieved context and producing answers that interleave or fuse modalities as required.

2. System Architecture and Core Methodologies

Modern MRAG systems adopt deeply modular architectures with explicit separation of critical functions:

1. Multimodal Indexing & Representation: Raw web documents, scientific papers, or enterprise data are parsed into structured, interleaved text, images, tables, and charts, often preserving layout and employing OCR for embedded text. Advanced pipelines segment documents hierarchically (page/region-level), enabling fine-grained retrieval (Xu et al., 1 May 2025).
2. Query and Planning Module: Query planning is central to MRAG. The planner determines the necessity and modality of retrieval (text, image, both, or none), refines or decomposes complex queries, and selects retrieval strategies—potentially iteratively—inspired by human cognitive strategies for information seeking (Yu et al., 26 Jan 2025). Planning frameworks such as CogPlanner and agentic modules (e.g., Windsock) perform dynamic modality selection and retrieval routing using lightweight classifiers or reinforcement learning (RL) based agents (Zhao et al., 26 Oct 2025).
3. Multimodal Retriever:
   • Employs cross-modal embedding models (e.g., CLIP, SigLIP, EVA-CLIP) and sometimes modality-specific retrievers (e.g., separate image and text vector stores), augmented with layout-aware architectures for document/image–text alignment.
   • Multi-granularity retrieval with hierarchical fusion (e.g., Reciprocal Rank Fusion, VLM-based reranking).
   • Specialized attention to structured content (tables, charts) (Xu et al., 1 May 2025, Yang et al., 20 Feb 2025).
4. Multimodal Generation:
   • Outputs may consist of both text and embedded visual elements, with image insertion points and selections determined either via supervised heuristics or via RL-trained inserters (e.g., RL-based inserter in M2IO-R1), which are optimized to select and position relevant images for maximal semantic alignment and interpretability (Xiao et al., 8 Aug 2025).
   • Generative frameworks support various output formats: text-only, interleaved text-image, or generalized to video/audio outputs in advanced setups.
5. End-to-End Pipeline:
   • Modular approach allows for independently replacing, optimizing, or fine-tuning components (retriever, planner, generator). Recent research emphasizes fully end-to-end learnable or tightly integrated, agentic architectures (Hu et al., 29 May 2025).

3. Advancements in Reasoning, Planning, and Control

Classical MRAG systems employed static, single-step or heuristic retrievals. Recent progress addresses their critical limitations (non-adaptivity, context overload, rigidity):

• Iterative and Adaptive Planning: Systems such as CogPlanner (Yu et al., 26 Jan 2025) and OmniSearch (Li et al., 5 Nov 2024) formalize adaptive, step-wise information acquisition. These agents iteratively decide what sub-query to pursue, in which modality, and when to halt retrieval, resembling a Markov decision process:

$$\mathcal{F}: \mathcal{S} \times \mathcal{P} \to \mathcal{S}$$

where $\mathcal{S}$ is the current information state and $\mathcal{P}$ the planner's decision space.

• RL-Enhanced Inserters and Agents: Frameworks such as M2IO-R1 (Xiao et al., 8 Aug 2025) implement a dedicated RL agent (Inserter-R1-3B) trained with Group Relative Policy Optimization (GRPO), optimizing for reward functions that jointly balance image recall, position accuracy, and output format. Such agents provide fine-grained control, interpretable action traces, and outperform baseline and rule-based methods by substantial margins:
  • On MRAMG-Bench ArXiv: Recall 84.2, Relevance 97.4, Overall 76.3—lower latency (4.34s/instance) compared to single-shot and rule-based alternatives.
• Step-wise Routing and Multi-Source Reasoning: R1-Router (Peng et al., 28 May 2025) integrates dynamic, intermediate query generation and routing across heterogeneous KBs (text, image, table), guided by step-specific RL (Step-GRPO) rewards to minimize unnecessary retrievals and enforce outcome-driven reasoning trajectories. R1-Router achieves average F1-Recall improvements of >7% over strong iterative and KB-routing baselines.
• Dynamic Noise Resistance: DANCE (Zhao et al., 26 Oct 2025) introduces targeted instruction tuning on “hard” queries—those where misleading or noisy retrieval is most detrimental—substantially improving robustness (+17% generation quality, -9% retrieval cost) and enabling effective modality selection.

4. Dataset Landscape and Evaluation Methodologies

MRAG progress necessitated new benchmarks capturing open-domain complexity, cross-modal reasoning, and faithful multimodal output:

• MRAMG-Bench (Yu et al., 6 Feb 2025): 4,800 QA pairs, 4,346 docs, 14,190 images across Web, Academia, and Lifestyle, with ground-truth annotated, interleaved multimodal answers supporting complex multi-image, multi-step reasoning.
• M^2RAG (Liu et al., 24 Feb 2025): Structured around four core tasks—image captioning, multimodal QA, fact verification, and image reranking—and introduces instruction tuning (MM-RAIT) that yields up to 34% absolute performance gains.
• mmRAG (Xu et al., 16 May 2025): Modular evaluation for retrieval, query routing, and generation over text, tables, and KGs, with component-level diagnostics.
• CogBench (Yu et al., 26 Jan 2025): Over 7,000 samples, real-world multimodal queries, annotated planning trajectories for iterative retrieval/reformulation research.
• Chart-MRAG Bench (Yang et al., 20 Feb 2025): Focuses on chart-based reasoning, exposing substantial failures of unified embedding retrieval (e.g., Recall@5 = 0% on charts) and systematic text-over-visual bias even in SOTA MLLMs.

Evaluation blends statistical (F1, Recall, BERTScore, etc.), knowledge-unit (claim-level), and LLM-based protocols (faithfulness, image relevance/effectiveness), often complemented with layout-sensitive or ordering metrics (weighted edit distance for image sequencing).

5. Empirical Performance and Technical Insights

Model	Recall	F1	Relevance	Order	Overall	Latency (s)	Model Size
Single-Shot	80.1	69.1	90.8	25.6	74.8	5.98	Large
Rule-Based	65.7	57.5	82.5	32.4	69.8	22.60	-
M2IO-R1-3B	84.2	68.4	97.4	39.4	76.3	4.34	3B
M2IO-Base-72B	83.7	69.3	97.6	38.6	76.6	-	72B

Parameter-efficient RL agents (e.g., Inserter-R1-3B, R1-Router) outperform much larger models, provide substantially reduced latency, and generalize better under data scarcity or domain shift. Modular and agentic architectures enable precise manipulation of output modality and format while maintaining strong semantic grounding.

Instruction-tuned MLLMs (MM-RAIT) or dynamic planning agents close performance gaps on complex, multi-hop tasks; careless concatenation or static retrieval leads to information overload and degraded performance.

Benchmarks reveal persistent domain challenges, notably in visually rich formats (charts, tables, dense manuals): unified embedding approaches collapse in chart retrieval (Recall@5 = 0.0% (Yang et al., 20 Feb 2025)) and even with ground truth, outputs attain only ~58% correctness with ~74% coverage.

6. Challenges and Open Directions

MRAG remains challenged by:

• Modality Alignment: Cross-modal retrieval, especially for dense or structured information (charts, tables), is not yet solved; unified embeddings frequently underrepresent visual semantics, demanding dedicated or hybrid strategies.
• Multi-hop and Dynamic Planning: Iterative, context-sensitive retrieval strategies outperform static pipelines but increase complexity in planning, state maintenance, and error propagation.
• Bias and Robustness: Systematic text-over-visual bias persists; models default to text-based answers even when visual evidence is superior, especially in small-scale MLLMs.
• Security and Privacy: MRAGs are vulnerable to data poisoning—knowledge injection attacks (Poisoned-MRAG) achieve up to 98% attack success rate (Liu et al., 8 Mar 2025); privacy leakage risks are acute in multimodal contexts, requiring systematic defense mechanisms (Zhang et al., 20 May 2025).
• Evaluation and Benchmarking: Cross-domain, cross-format benchmarking and fine-grained metrics tailored to multi-modal, multi-step, real-world queries are essential for progress and diagnosis (Ji, 29 Sep 2025, Yu et al., 6 Feb 2025).

A plausible implication is that future MRAG research will focus on (i) agentic and RL-based planning modules for adaptive retrieval/action selection, (ii) end-to-end multimodal training for deep cross-modal alignment, (iii) security/privacy auditing, and (iv) multi-level, claim/fact-based evaluation protocols for modeling faithfulness and hallucination under multimodal regimes.

7. Impact and Prospective Role

MRAG moves computational reasoning and answer synthesis from monomodal, static paradigms toward agentic, multimodally grounded frameworks. This impacts a spectrum of applications including automated scientific reporting, legal/financial document analysis, technical instruction generation, large-scale enterprise knowledge access, and dynamic robotic assistance (including adaptive video understanding (Mao et al., 29 May 2025)). MRAG systems, by aggregating, verifying, and communicating information across modalities, represent a principal vector for future AI systems operating in the inherently diverse and information-dense real world.