MRAG-Suite: Multimodal RAG Evaluation

• MRAG-Suite is a diagnostic evaluation platform for visual Retrieval-Augmented Generation, integrating heterogeneous multimodal benchmarks and two-step filtering techniques.
• It employs MM-RAGChecker for claim extraction and verification, enabling precise measurement of hallucination, retrieval accuracy, and ambiguity handling.
• The suite also serves as a modular design blueprint, combining retrievers, rerankers, planning agents, and domain-specific toolkits for robust system evaluation.

Searching arXiv for papers on “MRAG-Suite” and closely related multimodal RAG benchmarks/frameworks. MRAG-Suite denotes, in the most specific sense, a diagnostic evaluation platform for Visual Retrieval-Augmented Generation that integrates heterogeneous multimodal benchmarks, applies difficulty-based and ambiguity-aware filtering, and performs claim-level diagnosis with MM-RAGChecker (Ji, 29 Sep 2025). In a broader and increasingly common usage, the same label functions as a suite-level design target for multimodal Retrieval-Augmented Generation: a modular assemblage of benchmarks, retrievers, rerankers, planning agents, privacy and attack harnesses, and domain-specific toolkits that can be combined to evaluate or harden visual and multimodal RAG systems (Hu et al., 2024).

1. Terminology and scope

The literature uses “MRAG-Suite” in two related but non-identical ways. One usage refers to the named platform introduced as “MRAG-Suite: A Diagnostic Evaluation Platform for Visual Retrieval-Augmented Generation,” whose core emphasis is evaluation under hard, ambiguous, and distractor-rich conditions (Ji, 29 Sep 2025). The other usage appears in suite-design blueprints, where authors describe how a benchmark, attack protocol, reranking framework, or toolkit can serve as a component of an MRAG suite, even when no standalone package with that exact name is released (Hu et al., 2024).

Usage	Representative source	Role
Diagnostic platform	(Ji, 29 Sep 2025)	Cross-domain Visual RAG evaluation with filtering and MM-RAGChecker
Vision-centric evaluation backbone	(Hu et al., 2024)	Standardized dataset and scoring harness
Privacy assessment starting point	(Al-Lawati et al., 25 Jan 2026)	Reusable evaluation framework for multimodal RAG privacy
Attack and robustness module	(Luo et al., 19 Nov 2025)	Hierarchical visual attack plug-in
Evidence selection module	(Wang et al., 27 Apr 2026)	Semantic-aware reranking via MEG

This dual usage suggests that MRAG-Suite is best understood as both a concrete benchmark platform and an architectural pattern. In the architectural reading, the suite comprises at least five recurrent layers: benchmark curation, retrieval and reranking, agentic control or planning, robustness and security stress testing, and reporting with reproducibility controls. That pattern is explicit in papers that position MRAG-Bench as “the evaluation backbone of a vision-centric MRAG-Suite,” or that describe released code and evaluation protocols as a practical basis for building such a suite (Hu et al., 2024).

2. Diagnostic evaluation platform and MM-RAGChecker

The named MRAG-Suite platform consolidates eight heterogeneous sources into a unified schema consisting of question, short_answer, long_answer, evidence_imgs, and evidence_txts (Ji, 29 Sep 2025). Its benchmark inventory spans WebQA, Chart-RAG, Visual-RAG, MRAG-Bench, and four VisRAG subsets—Arxiv, Plot, Slide, and Document—thereby covering open-web images, charts, technical plots, scanned documents, slide decks, and scholarly PDFs. The platform is explicitly designed to measure multimodal retrieval accuracy, evidence selection under distractors, multi-hop reasoning, numeric and visual grounding, and robustness to ambiguity.

A central design choice is two-step filtering. The “Retrieval-Independent Filter” drops questions solved by a strong closed-book model with confidence greater than 0.9 and removes questions whose answer appears verbatim in the question text. The “Difficulty-Based Filter” then ranks the remaining items by “multi-hop requirement, modality dependency, and baseline success rate,” and drops the easiest approximately 10% per domain. In parallel, the platform constructs an ambiguity-aware subset by using an LLM pre-filter, followed by human adjudication with Cohen’s kappa $\kappa = 0.74$, and then rewriting both answers and queries into evidence-grounded, disambiguated forms (Ji, 29 Sep 2025).

The diagnostic core is MM-RAGChecker, which extends claim-level checking from text-only RAG to multimodal RAG. Its pipeline has three stages: claim extraction from the long-form answer, per-evidence judging against top-$K_{\text{img}}=3$ images and top-$K_{\text{txt}}=3$ text chunks, and final aggregation with the precedence rule “Entailment, else Contradiction, else Neutral.” This supports hallucination analysis, modality attribution, and evidence-usage accounting. The platform defines task output as

$$(a^{\text{short}}, a^{\text{long}}) = f(q, E^{\text{txt}}, E^{\text{img}})$$

and evaluates long-form outputs with claim-level metrics such as

$$\mathrm{Hallucination\ Rate} = \frac{|\{\, i : L_i = \text{Neutral} \,\}|}{|\mathcal{C}|}$$

and

$$\mathrm{Faithfulness} = \frac{|\{\, i : L_i = \text{Entailment} \,\}|}{|\mathcal{C}|}.$$

The reported results show that the filtering is not cosmetic. Aggregate performance for LLaVA-2 drops from 62.5% on the Full split to 47.8% on the Filtered split, indicating that trivial or guessable items materially inflate headline accuracy (Ji, 29 Sep 2025). On the curated ambiguity subset, short-answer EM/Accuracy drops from 23.5 on clear items to 17.6 on ambiguous items, hallucination rate rises from 49.2 to 58.7, claim recall falls from 12.4 to 9.1, and context precision falls from 14.1 to 12.5. Claim-level results are likewise severe: on Visual-RAG (Filtered), hallucination rates are 74.0 for Qwen2.5-VL-72B, 76.8 for LLaVa-v1.6-34B, and 85.1 for Claude 3.5. The platform’s diagnostic value lies precisely in surfacing these unsupported clauses, distractor failures, and modality skews rather than only reporting end-task EM or Accuracy.

3. Benchmark backbones and task coverage

Outside the named platform, MRAG-Suite is assembled from several benchmark backbones that each isolate a different failure regime. MRAG-Bench is the best-developed vision-centric backbone: it contains 16,130 images and 1,353 human-annotated multiple-choice questions across 9 scenarios, including Angle, Partial, Scope, Occlusion, Temporal, Deformation, Incomplete, Biological, and Others (Hu et al., 2024). It is explicitly evaluation-only, uses a query-first multi-image prompt format, measures QA accuracy and Recall@5, and demonstrates that “all tested LVLMs improve more with visual augmentation than with textual augmentation.” The same paper explicitly positions MRAG-Bench as “the evaluation backbone of a vision-centric MRAG-Suite.”

Chart-based MRAG expands this backbone toward information-dense visual documents. Chart-MRAG Bench contains 4,738 question-answering pairs across 8 domains, built from 267 documents, 1,283 text passages, and 627 charts, with 12 expert annotators and Fleiss’s $\kappa = 0.82$ (Yang et al., 20 Feb 2025). Its CHARGE framework operationalizes structured keypoint extraction, crossmodal verification, and keypoint-based QA generation. The benchmark exposes a characteristic suite-level asymmetry: unified multimodal embedding retrieval performs strongly on text-only retrieval but can collapse on chart-only and text-chart retrieval, while separate text and chart stores substantially improve Recall@5 and Recall@10.

V-QPP-Bench extends suite coverage from retrieval and generation to pre-retrieval image repair. It contains 46,700 imperfect queries derived from InfoSeek and ViQuAE, covers 10 imperfection types, and formulates Visual Query Pre-processing as an agentic decision-making task with tools such as rotation, flip, brightness adjustment, deblur, denoise, locate, crop, and fill (Zhang et al., 13 Feb 2026). Dyn-VQA and CogBench then stress adaptive retrieval rather than static retrieval. Dyn-VQA contains 1,452 questions across 9 domains, with 715 English and 737 Chinese items, and was constructed specifically to surface “rapidly changing answers,” “questions requiring multi-modal knowledge,” and “multi-hop questions” (Li et al., 2024). CogBench contains 5,718 queries, 1,381 images, and 9 cognitive domains, with 46.62% of examples requiring more than 2-hop reasoning (Yu et al., 26 Jan 2025).

Domain-specific variants extend the suite beyond general-purpose vision. In biomedicine, MRAG-Bench spans 13 test datasets and 14,816 test samples across multiple-choice QA, long-form QA, information extraction, and biomedical link prediction, while MRAG-Toolkit supplies a modular retrieval and evaluation harness over Wikipedia, PubMed, StatPearls, medical textbooks, and web search (Zhu, 23 Jan 2026). A plausible implication is that “MRAG-Suite” is no longer confined to one benchmark package; it has become a template for organizing modality-aware evaluation across general, chart-centric, dynamic, biomedical, and retrieval-preprocessing settings.

4. Privacy, attacks, and robustness stress testing

A substantial part of the MRAG-Suite literature concerns adversarial and privacy evaluation. “A Systemic Evaluation of Multimodal RAG Privacy” does not introduce a named privacy suite, but explicitly provides “a complete, reusable evaluation framework, attack protocol, prompts, metrics, datasets, pipeline components, and ablations” and releases code in mrag-attack-eval (Al-Lawati et al., 25 Jan 2026). Its threat model is black-box API access only; the attacker can submit a query image and textual prompt but cannot access internal parameters, retriever index contents, or reranking scores. Two tasks are defined: Membership Inference Attack (MIA), which determines whether an image or transformation $\mathcal{T}(i)$ is present in the indexed images, and Image Caption Retrieval (ICR), which attempts to retrieve the associated caption once membership is established.

The privacy findings are severe. In the exact-image setting, MIA leakage is “very high” across datasets and models, with example F1 values of approximately 0.989 for Cosmos-Reason1 on Conceptual Captions and 0.993 for Qwen2.5-VL on Pokemon BLIP (Al-Lawati et al., 25 Jan 2026). Caption leakage is likewise high when retrieval is accurate; for Conceptual Captions, exact-match averages are approximately 0.835 for Qwen2.5-VL and 0.747 for InternVL3.5. The paper also isolates operational mitigations: placing the query image first in the prompt (“RAG-Last”) substantially reduces MIA leakage, cross-modal reranking can reduce ICR leakage where image-text alignment is weak, smaller $k$ reduces leakage, and rotation is the most protective transformation. By contrast, a simple refusal-style system prompt did not induce refusal in any evaluated VLMs.

HV-Attack studies a different threat: imperceptible perturbations applied only to the user image in an MRAG query (Luo et al., 19 Nov 2025). Its hierarchical two-stage procedure first breaks cross-modal alignment and then disrupts semantic alignment in the retriever’s embedding space, with a generator-side perturbation $\delta_g$ further degrading image understanding. Under an $K_{\text{img}}=3$0 budget of $K_{\text{img}}=3$1, the attack produces large retrieval and generation failures. On InfoSeek with Siglip-so400m, Recall@5 drops from 44.47 to 4.37 and Precision@5 from 20.12 to 1.56; BLIP-2 EM drops from 21.63 to 5.47. The same paper explicitly discusses “MRAG-Suite Design Implications,” including configurable attack modules, stage toggles, transfer tests, and robustness metrics.

Poisoned-MRAG adds a database-side attack model in which a small number of malicious image-text pairs are injected into the multimodal knowledge base (Liu et al., 8 Mar 2025). The paper reports “up to 98\% attack success rate with just five malicious image-text pairs injected into the InfoSeek database (481,782 pairs),” and distinguishes dirty-label and clean-label variants. Dirty-label attacks achieve ASR-R approximately 1.00 and ASR-G up to 0.98 on InfoSeek, while clean-label attacks remain strong though more sensitive to retriever choice and $K_{\text{img}}=3$2. Defenses such as paraphrasing, duplicate removal, structure-driven mitigation, and purification all show limited effectiveness or high utility cost.

Robustness is not only adversarial. V-QPP-Bench shows that ordinary visual imperfections can catastrophically degrade retrieval before any downstream reasoning begins (Zhang et al., 13 Feb 2026). On the primary I2T setting, Original Query Recall@5 is 0.496, while RealWorld corruption yields a relative drop of −98.6%, Expand yields −80.3%, Rotation yields −49.0%, and Crop yields −19.1%. Oracle preprocessing recovers near-perfect performance for severe defects, and supervised fine-tuning on 1,000 oracle traces enables a compact Qwen3-VL-4B-Instruct model to outperform larger off-the-shelf agents across most imperfections. This suggests that a mature MRAG-Suite must treat security, privacy, and perceptual robustness as first-class evaluation axes rather than peripheral ablations.

5. Retrieval control, reranking, and agentic planning

A second major branch of MRAG-Suite concerns how retrieved evidence is selected, ordered, and iteratively refined. The design-space study “mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation” decomposes the pipeline into retrieval, reranking, generation, and a unified self-reflective agentic framework (Hu et al., 29 May 2025). Its principal recipe is explicit: EVA-CLIP as retriever, LVLM listwise re-ranking, and only the most relevant document passed to the generator. The study reports that the unified framework improves average response accuracy by approximately 5% on E-VQA and approximately 2% on InfoSeek without fine-tuning, and emphasizes that increasing retrieval depth can improve retrieval accuracy while harming response accuracy because of positional bias and irrelevant context.

MEG-RAG formalizes evidence selection with a semantic-aware metric, Multi-modal Evidence Grounding (MEG), defined as the change in semantic confidence on high-IDF answer anchors when a retrieved document is present (Wang et al., 27 Apr 2026). The resulting reranker is trained on 31K positives and 31K negatives with a hybrid loss

$K_{\text{img}}=3$3

using a Jina-m0 backbone. The gains transfer across tasks and models: for example, on MMQA with MM-RAIT-Qwen2.5-VL and top-1 selection, CIDEr rises from 243.65 under Jina reranking to 260.78 under MEG-RAG, while on Fact Verification with Qwen2.5-VL and top-3 selection, F1 rises from 65.96 to 73.75.

MMAgent-R$K_{\text{img}}=3$4 pushes this further into agentic internal verification. It combines external retrieval, visual reranking, and active rejection, allowing the system to discard an unreliable candidate batch and retrieve additional candidates rather than remaining trapped inside a fixed top-$K_{\text{img}}=3$5 pool (Zhang et al., 8 Jul 2026). The default configuration uses $K_{\text{img}}=3$6, $K_{\text{img}}=3$7, $K_{\text{img}}=3$8, EVA-CLIP-8B for image retrieval, E5 for paragraph retrieval, and GRPO training with step-level verification rewards. The empirical pattern is consistent across datasets: on InfoSeek, the retriever’s Recall@1 is 53.7%, but the agent raises identification accuracy to 63.6%; on E-VQA, Recall@1 is only 15.4%, while identification rises to 29.5%. The paper presents these results as evidence that visual reranking and active rejection mitigate distractor-heavy candidate pools.

Planning-oriented frameworks treat retrieval itself as a sequential decision process. OmniSearch addresses “Non-adaptive Retrieval Queries” and “Overloaded Retrieval Queries” by decomposing complex multimodal questions into sub-question chains and dynamically selecting among web search, image search with the input image, and image search with text queries (Li et al., 2024). On Dyn-VQA, OmniSearch with GPT-4V achieves 50.03 overall F1-Recall, versus 37.15 for a heuristic two-step caption-first baseline and 30.25 for GPT-4V without retrieval. CogPlanner formulates “MRAG Planning” as a budgeted optimization over query reformulation and retrieval-action selection, with parallel and sequential variants (Yu et al., 26 Jan 2025). On CogBench, CogPlanner improves GPT-4o end-to-end F1 from 0.2121 to 0.2349 in the sequential setting and improves Qwen2-VL-72B-Instruct F1 from 0.1819 to 0.2174 in the parallel setting, while a lightweight Qwen2-7b-VL-Cog planner keeps token overhead near 10%. Across these papers, the suite-level conclusion is stable: effective MRAG evaluation now requires not only judging what was retrieved, but how retrieval decisions were staged, reformulated, reranked, and, when necessary, rejected.

6. Domain extensions, limitations, and research directions

MRAG-Suite is increasingly domain-specialized. In biomedical QA, MRAG-Bench and MRAG-Toolkit provide a modular testbed over English and Chinese tasks, multiple corpora, BM25 and dense retrievers, RRF hybrid retrieval, and long-form evaluation with GPT-4-as-judge Elo ratings (Zhu, 23 Jan 2026). The paper reports that RAG improves average performance across cohorts, but also that long-form responses may become “slightly less readable for long-form questions.” In time-sensitive QA, the trainless modular retrieval framework for TempRAGEval separates “main content” from “temporal constraint,” summarizes top passages while preserving dates, and applies a semantic-temporal hybrid ranking scheme (Siyue et al., 2024). On perturbed TimeQA questions, AR@1 rises from 46.7% for Gemma to 57.6% for MRAG, and ER@1 rises from 26.0% to 32.4%. These domain-specific systems are not always named MRAG-Suite, but they instantiate the same suite logic: modular decomposition, targeted retrieval, domain-aware scoring, and explicit benchmarking.

The main limitations recur across the literature. The diagnostic Visual RAG platform does not cover videos, temporal evidence, embodied settings, or low-resource languages, and its scores are sensitive to OCR quality, prompt choices, distractor sampling, and LLM or VLM judging behavior (Ji, 29 Sep 2025). Privacy and attack papers are explicitly dual-use, and the privacy study notes that its methods “could be repurposed to attack live systems” (Al-Lawati et al., 25 Jan 2026). Many components rely on proprietary APIs, external search engines, or evolving backbones, which complicates reproducibility. Public artifacts are uneven: some papers release code or datasets, some announce future release, and some do not specify license terms in the paper text.

A plausible implication is that MRAG-Suite should now be viewed less as a single benchmark package than as a layered research program. Its stable core consists of: curated multimodal benchmarks; claim-level and retrieval-level diagnostics; modular retrieval, reranking, and planning interfaces; attack, privacy, and robustness harnesses; and domain-specific extensions for biomedical, chart-centric, dynamic, and temporal reasoning. What remains unsettled is the coupling among these layers: whether reranking should be semantic-anchor-aware, whether retrieval should be agentic or trainless, how to calibrate rejection and stopping, and how to evaluate multimodal grounding when the evidence itself is ambiguous or partially corrupted. The literature converges on one point, however: end-task accuracy alone is no longer an adequate description of multimodal RAG quality.