MM-RAGChecker: Multimodal Claim Diagnosis

• MM-RAGChecker is a claim-level diagnostic tool integrated within MRAG-Suite, providing fine-grained evaluation of multimodal support in generated answers.
• It employs a three-stage pipeline—claim extraction, evidence judging, and aggregation—to assess factual support via entailment, neutral, or contradiction judgments.
• The tool reports detailed metrics on hallucination, faithfulness, context precision, and cross-modality agreement to highlight evidence bias and failure modes.

Searching arXiv for the named topic and closely related work to ground the article in current papers. {"query":"MM-RAGChecker OR \"MRAG-Suite\" visual retrieval augmented generation claim-level diagnostic tool", "max_results": 10} MM-RAGChecker is a claim-level diagnostic tool integrated into the MRAG-Suite evaluation platform for multimodal, or Visual, Retrieval-Augmented Generation. Its stated purpose is to diagnose factuality, grounding, and failure reasons in answers supported by retrieved visual and textual evidence. Rather than relying on end-task accuracy alone, it checks each factual statement in a generated long-form answer against retrieved multimodal evidence such as charts, images, and text snippets, thereby supporting fine-grained analysis of hallucinations, faithfulness degradation under query difficulty and ambiguity, evidence usage, and modality bias (Ji, 29 Sep 2025).

1. Position in the multimodal RAG evaluation landscape

MM-RAGChecker appears within MRAG-Suite, a diagnostic evaluation platform that integrates diverse multimodal benchmarks, including WebQA, Chart-RAG, Visual-RAG, and MRAG-Bench, and augments them with difficulty-based and ambiguity-aware filtering (Ji, 29 Sep 2025). In that setting, MM-RAGChecker is not a standalone benchmark; it is the component responsible for claim-level diagnosis after a Visual RAG system has produced an answer.

Its role is best understood against the broader shift from coarse end-task scoring toward modular diagnosis in RAG evaluation. RAGChecker introduced a fine-grained, claim-level framework for diagnosing retrieval and generation in text-centric RAG systems, with dedicated metrics for retrieval and generation modules (Ouyang et al., 2024). In multimodal retrieval, contemporaneous benchmarks have emphasized different failure axes. M$^2$RAG evaluates open-domain multi-modal retrieval-augmented generation across image captioning, multi-modal question answering, multi-modal fact verification, and image reranking (Liu et al., 24 Feb 2025), whereas REAL-MM-RAG focuses on real-world multi-modal document retrieval, enhanced difficulty, realistic-RAG queries, and accurate labeling (Wasserman et al., 17 Feb 2025). MM-RAGChecker occupies the diagnostic layer within this ecosystem: it is intended to determine, at the level of individual claims, whether a multimodal answer is actually supported by retrieved evidence.

A central implication is that MM-RAGChecker shifts evaluation from question-level correctness to evidence-grounded decomposition. This suggests a different target of measurement: not only whether an answer is acceptable, but also which atomic propositions are grounded, unsupported, or contradicted, and whether images and text are being used jointly or only superficially.

2. Three-stage verification pipeline

MM-RAGChecker is described as a three-stage, automated claim verification engine for long-form, evidence-grounded answers (Ji, 29 Sep 2025).

The first stage is claim extraction. The input is a generated long-form answer $a^{\text{long}}$, which is split into minimal verifiable factual claims $\mathcal{C} = \{c_i\}$. The paper reports a hybrid procedure: basic heuristics break text on full stops and conjunctions such as “and,” and an LLM-based splitter further decomposes compound or complex clauses. The given example is “The dog is brown and catching a frisbee,” which is decomposed into “The dog is brown.” and “The dog is catching a frisbee.”

The second stage is evidence judging. For each claim $c_i$ and each retrieved evidence item, whether an image $I_k$ or a text passage $t_k$, MM-RAGChecker performs a three-way entailment judgment with labels $$\{\text{Entailment}, \text{Neutral}, \text{Contradiction}\}$$. The description explicitly states that no attempt is made to synthesize image captions, so claims may be matched directly to visual features. By default, the system checks the top 3 images and top 3 text passages per question, denoted by $K_{\text{img}} = 3$ and $K_{\text{txt}} = 3$. For each claim, this yields multisets

$$\mathcal{J}^{\text{img}}_i = \{j(c_i, I_k)\}, \qquad \mathcal{J}^{\text{txt}}_i = \{j(c_i, t_k)\}.$$

The third stage is aggregation and labeling. Each claim receives a final label $a^{\text{long}}$0 under precedence rules. A claim is assigned Entailment if any judgment across all evidence is Entailment. If no evidence entails the claim, but at least one judgment is Contradiction, the final label is Contradiction. Otherwise, the label is Neutral. For cross-modality analysis, the system sets binary support flags: $a^{\text{long}}$1

$a^{\text{long}}$2

Each retrieved item $a^{\text{long}}$3 is marked as “used” if it supports an entailed claim.

This pipeline operationalizes multimodal grounding without collapsing the image and text channels into a single synthetic text representation. That design choice is significant because the diagnostic target is not only factual support, but also whether support came from visual evidence, textual evidence, or both.

3. Diagnostic metrics

MM-RAGChecker reports a suite of metrics for hallucinations, faithfulness, context usage, and modality interaction (Ji, 29 Sep 2025). Let $a^{\text{long}}$4 be the set of extracted claims, $a^{\text{long}}$5 the final label for claim $a^{\text{long}}$6, and $a^{\text{long}}$7 the set of retrieved evidence items.

The core answer-grounding metrics are defined as follows: $a^{\text{long}}$8

$a^{\text{long}}$9

Claim recovery with respect to a reference claim set $\mathcal{C} = \{c_i\}$0 is measured by Claim Recall: $\mathcal{C} = \{c_i\}$1

Evidence efficiency is measured by Context Precision: $\mathcal{C} = \{c_i\}$2 where

$\mathcal{C} = \{c_i\}$3

The framework also reports Self-Knowledge: $\mathcal{C} = \{c_i\}$4

To expose modality asymmetries, MM-RAGChecker adds cross-modality metrics. The image-text gaps in claim recovery and context usage are

$\mathcal{C} = \{c_i\}$5

Modality hit and miss rates are defined as

$\mathcal{C} = \{c_i\}$6

$\mathcal{C} = \{c_i\}$7

Cross-Modality Agreement measures the fraction of supported claims jointly grounded by both modalities: $\mathcal{C} = \{c_i\}$8

Finally, the framework includes hallucination attribution metrics: $\mathcal{C} = \{c_i\}$9

$c_i$0

Taken together, these metrics move beyond a single groundedness score. They separate unsupported-claim frequency, supported-claim frequency, gold-claim recovery, evidence utilization, and the relative contribution of image and text evidence. This makes MM-RAGChecker a diagnostic instrument rather than a single-number evaluator.

4. Filtering strategy and retrieval stress tests

A distinctive feature of MM-RAGChecker within MRAG-Suite is its use alongside explicit filtering schemes intended to remove trivial cases and expose robustness failures (Ji, 29 Sep 2025).

The retrieval-independent filter removes questions that can be answered with high confidence by models without retrieval, or questions whose answer strings appear verbatim in the query. The goal is to eliminate cases where retrieval plays little role.

The difficulty-based filter ranks items and drops the easy cases, approximately the lowest 10% per domain, to emphasize multi-hop, multi-modal, or ambiguous queries. The paper also defines ambiguity subsets, consisting of 200 ambiguous items and 200 clear items, annotated with LLM-based rationales and human adjudication, with $c_i$1.

MM-RAGChecker is further evaluated under controlled retrieval modes:

• gt_only (GO): only gold, or relevant, evidence;
• gt_plus_distractors (GPD): gold evidence mixed with similar CLIP-based distractors;
• distractors_only (DO): only irrelevant distractors.

These settings isolate the effect of evidence composition on faithfulness and hallucination. The paper reports that adding distractors sharply decreases supported claims, increases hallucinations, and decreases precision. Conversely, using only gold evidence can increase the supported claim fraction to approximately 90%, whereas GPD and DO settings substantially increase hallucinations.

This evaluation design is important because it treats query ambiguity and distractor sensitivity as first-class stressors. A plausible implication is that MM-RAGChecker is intended not only for model comparison, but also for controlled fault injection: the retrieval stack can be perturbed, and the downstream claim-level damage can then be quantified.

5. Empirical findings

MM-RAGChecker is used across multiple multimodal benchmarks and models, including WebQA, Chart-RAG, Visual-RAG, MRAG-Bench, and additional VisRAG subsets such as VisRAG-Arxiv, VisRAG-Plot, VisRAG-Doc, and VisRAG-Slide (Ji, 29 Sep 2025).

The reported findings emphasize the gap between answer plausibility and claim-level grounding. On filtered hard queries, 30–40% of claims in long answers are unsupported. An example reported in the paper is LLaVa-2 with a hallucination rate of 32% and Faithfulness of 68% on filtered sets. On filtered WebQA, Faithfulness is approximately 72%, whereas on Visual-RAG it is approximately 60%.

Query ambiguity is a major failure source. On ambiguous queries, short-answer accuracy drops by approximately 6 percentage points, and hallucination rate increases by 9.5 points compared to clear queries. The abstract of MRAG-Suite likewise states that the platform finds substantial accuracy reductions under difficult and ambiguous queries, with prevalent hallucinations (Ji, 29 Sep 2025).

The framework also reveals systematic modality imbalance. Large positive $c_i$2 or $c_i$3 indicates that a model leans toward image or text evidence; the description notes that models are often overly reliant on text. Low Cross-Modality Agreement suggests that models frequently use only one modality per claim rather than performing integrative reasoning across modalities. Evidence efficiency is also limited: the paper reports that models use only about 21% of retrieved items for actual grounding.

Several common assumptions are directly challenged by these findings. One misconception is that correct retrieval suffices for faithful generation. The reported GO, GPD, and DO experiments indicate otherwise: even when relevant evidence is available, current systems can still produce plausible-sounding but ungrounded claims. Another misconception is that multimodal RAG systems naturally fuse image and text evidence. Low CMA and positive modality-gap metrics indicate that many systems instead default to single-modality support patterns.

6. Relation to adjacent diagnostic frameworks and significance

MM-RAGChecker belongs to a broader family of diagnostic frameworks that decompose RAG behavior into retrieval-side and generation-side phenomena. The original RAGChecker defines claim-level metrics such as Claim Recall, Context Precision, Faithfulness, Hallucination, Self-knowledge, and Context Utilization for text-centric RAG (Ouyang et al., 2024). MM-RAGChecker preserves the claim-level philosophy but extends diagnosis to multimodal evidence by introducing image-text entailment judgments, per-modality support flags, Cross-Modality Agreement, and modality-specific hit, miss, and hallucination-attribution metrics (Ji, 29 Sep 2025).

This development parallels other domain-specific claim verifiers. MedRAGChecker, for example, performs claim-level verification for biomedical RAG by combining evidence-grounded NLI with biomedical knowledge-graph consistency signals and aggregates claim decisions into diagnostics such as faithfulness, under-evidence, contradiction, and safety-critical error rates (Ji et al., 10 Jan 2026). The comparison indicates a broader trend: claim-level verification is becoming specialized by modality and domain rather than remaining a single generic evaluation recipe.

MM-RAGChecker is also complementary to system-analysis platforms that operate at a different granularity. RAGExplorer supports macro-to-micro comparative diagnosis of RAG configurations through performance overviews, failure attribution, side-by-side instance analysis, and interactive context perturbation (Tian et al., 19 Jan 2026). InspectorRAGet similarly emphasizes aggregate and instance-level introspection with both human and algorithmic metrics, annotator analysis, and significance testing (Fadnis et al., 2024). In contrast, MM-RAGChecker’s distinctive contribution lies in direct multimodal evidence grounding at the claim level.

Its significance is therefore methodological rather than purely benchmark-oriented. It provides a way to measure where hallucinations occur, how much retrieved evidence is actually used, whether unsupported claims arise under distractors or ambiguity, and whether a system is grounding claims in images, text, or both. For Visual RAG, this diagnostic decomposition supplies a basis for targeted interventions in retrieval, reranking, prompting, and multimodal fusion, rather than treating answer accuracy as the only observable outcome (Ji, 29 Sep 2025).