GroundMM: Multimodal Misinformation Grounding

• Grounding Multimodal Misinformation (GroundMM) is a research task that links multimodal content with localized, evidence-based verification to counter deceptive narratives.
• It integrates methods such as fusion, mapping, and compositional representations within adversarial frameworks to ensure truthful cross-modal alignment.
• GroundMM challenges binary detection by requiring explicit localization and interpretability through benchmarks, retrieval-augmented techniques, and reasoning-centric architectures.

Grounding Multimodal Misinformation (GroundMM) denotes a line of research—and, in one formalization, a task—in which multimodal content is not only classified as authentic or misleading, but also localized, explained, or otherwise anchored to evidence across modalities. In this literature, “grounding” refers to linking claims, captions, speech, or generated rationales to concrete visual regions, text spans, temporal segments, or retrieved external evidence. GroundMM therefore departs from binary fake/real classification and from single-modality forensics by requiring multimodal verification plus interpretable attribution of where the misinformation resides or what evidence supports the verdict (Yang et al., 8 Sep 2025, Beinborn et al., 2018).

1. Conceptual foundations

The broader grounding literature characterizes multimodal processing through multimodal fusion, mapping (projection), joint learning, and compositional representations (Beinborn et al., 2018). In that framework, fusion combines representations directly, mapping learns functions from one modality to another, joint learning optimizes unimodal and cross-modal objectives together, and compositionality builds phrase- or sequence-level representations from multimodal concept representations. The same survey organizes architectures by cross-modal transfer, cross-modal interpretation, and joint multimodal processing, aligning technical pipelines with different information-flow assumptions (Beinborn et al., 2018).

GroundMM inherits these representational issues but places them in an adversarial setting. The central question is not merely whether image and text are correlated, but whether they are truthfully correlated, whether one modality has been manipulated, whether the pair is out-of-context, and whether the model can indicate the manipulated content or retrieve evidence that resolves ambiguity. This is why the literature repeatedly distinguishes multimodal misinformation from classical deepfake detection and from text-only fake news detection. A recurring theme is that multimodal misinformation often requires deeper reasoning about semantic consistency, event context, and evidence provenance rather than only artifact detection (Shao et al., 2023, Yang et al., 12 Jul 2025).

A formal GroundMM instance in the video setting is written as $x = (\mathcal{V}, \mathcal{T}, \mathcal{R})$, where $\mathcal{V}$ is video, $\mathcal{T}$ is title text, and $\mathcal{R}$ is transcript. The prediction targets are a veracity label $\hat{y} \in \{0,1\}$, a fake-type vector $\hat{\mathbf{c}} \in \{0,1\}^6$, and grounding outputs $\hat{\mathbf{G}} = \{\hat{G}_i \mid c_i=1\}$ (Yang et al., 8 Sep 2025). This formulation makes explicit that GroundMM is simultaneously a verification problem, a structured prediction problem, and a localization problem.

2. Task formulations and benchmark datasets

The benchmark landscape shows a progression from synthetic image–text manipulation datasets to more realistic, semantically coordinated, and multimodal video settings. The earliest large-scale formulations emphasize image bounding boxes and manipulated text tokens; later benchmarks add temporal spans, speech grounding, and external fact-check evidence. This evolution reflects a shift from detecting artificial mismatches to grounding realistic misinformation patterns (Shao et al., 2023, Yang et al., 8 Sep 2025).

Benchmark	Modalities	Key properties
DGM$^4$	image-text	230,000 pairs; binary label, 4D manipulation type label, image bounding boxes, text token annotations
GroundLie360	video, title text, ASR transcript	2,044 target videos across 1,466 events, 588 auxiliary videos, 6 fake types, fine-grained grounding
MDSM	image-text	441,423 semantically coherent manipulated pairs with fake detection, manipulation type, and manipulated image region bounding box
SAMM	image-text with external celebrity knowledge	~261,000 pairs: 111k original, 80k swap manipulations, 69k attribute manipulations, image box and token-level text labels

DGM$^4$ formalized “Detecting and Grounding Multi-Modal Media Manipulation” as a task with four manipulation types: Face Swap (FS), Face Attribute (FA), Text Swap (TS), and Text Attribute (TA). The dataset contains 230,000 image-text pairs, including 77,426 pristine and 152,574 manipulated samples, with binary labels, fine-grained manipulation types, manipulated image bounding boxes, and manipulated text tokens (Shao et al., 2023). The later HAMMER++ description reports the dataset at approximately 230k news samples, with 77,426 pristine and 152,574 manipulated pairs, and notes that image perturbations such as JPEG compression and blur are randomly applied to 50% of samples (Shao et al., 2023).

GroundLie360 introduced GroundMM explicitly for web video misinformation. It is described as the first real-world dataset for this task, with 2,044 target videos (under 5 mins) across 1,466 events, plus 588 auxiliary original/debunking/evidence videos. Its annotation taxonomy spans False Title, False Speech, Temporal Edit, CGI, Contradictory Content, and Unsupported Content, with token-level text grounding, temporal video spans, and frame-level bounding boxes. Reported inter-annotator IoU scores are 0.74 for text, 0.84 for temporal spans, and 0.98 for bounding boxes (Yang et al., 8 Sep 2025).

Two later datasets address a methodological criticism of earlier benchmarks: that many multimodal manipulations are too obviously misaligned. MDSM constructs semantically coherent manipulations by first editing images and then generating matching deceptive narratives with MLLMs; it contains 441,423 annotated image-text pairs, with labels for fake detection, manipulation type (FS, FA, TF), and manipulated image regions (Zhang et al., 23 May 2025). SAMM similarly targets semantically-coordinated manipulations using a two-stage pipeline and a Celeb Attributes Portfolio (CAP) as an external knowledge base; it reports ~261,000 pairs, comprising 111k original, 80k swap manipulations, and 69k attribute manipulations, with ground-truth bounding boxes and token-level text labels (Shen et al., 16 Sep 2025).

3. Localized and alignment-based architectures

A major methodological family treats GroundMM as a fine-grained alignment problem between image regions and text tokens. HAMMER introduced a hierarchical design with manipulation-aware contrastive learning as “shallow manipulation reasoning” and modality-aware cross-attention as “deep manipulation reasoning.” It combines binary and multi-label detection heads with bounding-box and token-grounding heads. HAMMER++ adds local-view manipulation-aware contrastive learning to strengthen patch/token alignment. On DGM$^4$, the reported metrics for HAMMER++ are 93.33 AUC for binary classification, 86.41 mAP for multi-label classification, 76.46 IoUmean for image grounding, and 72.59 F1 for text grounding (Shao et al., 2023).

FMS extends this line by explicitly supervising modality reliability, unimodal forgery cues, and cross-modal interaction. Its three modules are Multimodal Decision Supervised Correction (MDSC), Unimodal Forgery Mining Reinforcement (UFMR), and Multimodal Forgery Alignment Reasoning (MFAR). The stated purpose is to correct the multi-modal decision-making process when one modality is unreliable, amplify disparities between real and fake content within each modality, and reason over consistency and inconsistency across modalities. On DGM$\mathcal{V}$0, FMS reports 96.46% AUC, 93.43% mAP, 84.82% IoUmean, and 75.34% text-grounding F1 (Yu et al., 4 Aug 2025).

MaLSF shifts the design from passive holistic fusion to “active, bidirectional verification.” Its core idea is to use mask-label pairs as local semantic anchors that explicitly connect pixel regions to labels. It then applies a Bidirectional Cross-modal Verification (BCV) module with Text-as-Query and Image-as-Query streams, followed by Hierarchical Semantic Aggregation (HSA). On DGM4, MaLSF reports 89.37 binary accuracy, 83.93 multi-label CF1, 82.37/82.47 IoUmean for image grounding, and 76.88/77.19 text-grounding F1, depending on the parser variant; on Weibo21 and Weibo17, it reports 95.5% and 93.5% accuracy, respectively (Chen et al., 27 Mar 2026).

A contrasting design is LADLE-MM, which targets limited-annotation settings. It uses two unimodal branches plus a frozen BLIP multimodal branch as a fixed reference space, and it does not use explicit grounding annotations during training. The model is reported to use 60.3% fewer trainable parameters than previous state-of-the-art models, to achieve 87.90 mAP on DGM4 multi-label classification, and to reach 79.6% accuracy and 83.5% F1 on VERITE, where it is described as robust to unimodal bias (Cardullo et al., 23 Dec 2025). This suggests that some grounding effects can emerge from alignment to a stable multimodal reference space even when bounding-box and token-level supervision are absent.

4. Retrieval-augmented and evidence-grounded verification

A second family grounds misinformation decisions in external evidence rather than only in internal cross-modal alignment. In these systems, grounding is operationalized through retrieval, re-ranking, evidence aggregation, or graph reasoning over retrieved documents.

A zero-shot pipeline for multimodal misinformation detection with large vision-LLMs combines initial retrieval, re-ranking with LLMs and LVLMs, and fact verification using the top evidence. On a more completely annotated subset of MOCHEG, the proposed evidence retrieval method reports 90.00 top-1 mAP for both text and image evidence, and on fact verification with gold multimodal evidence, LLaVA reports micro F1 of 0.534 on MOCHEG and 0.595 on Factify (Tahmasebi et al., 2024). The same work argues that earlier evaluations understated retrieval quality because of incomplete ground-truth evidence labels.

RAMA organizes evidence-grounded verification into WebRetriever, VLJudge, and DecisionFuser. The retrieval module uses a tool-augmented iterative loop for query planning, searching, browsing, and synthesis; the judgment module deploys three agents, two based on Qwen-VL and one on InternVL; and the final verdict uses weighted majority voting. On the 1,000 image-caption pairs of the Multimedia Verification Challenge public test set, RAMA reports 0.9100 accuracy and 0.9100 F1 using only the first caption, while the paper emphasizes that it performs without any training or fine-tuning and is less sensitive to dataset shifts (Yang et al., 12 Jul 2025).

EGMMG represents evidence and claims as graphs. It performs reverse image search to collect web pages containing the image, builds an evidence graph from retrieved text and a claim graph from the caption, and then compares them using attention-based GNNs. The reported final encoder is TransformerConv, and the method reaches 93.05% detection accuracy on the evaluation set, outperforming the second-best performing method by 2.82% (Duwal et al., 23 May 2025). Here the grounding signal is not a localized image box or token span, but a structured correspondence between retrieved event/entity graphs and claim graphs.

Two later systems push evidence-grounded reasoning toward more modular inference-time agents. MIRAGE decomposes detection into visual veracity assessment, cross-modal consistency analysis, retrieval-augmented factual checking, and a calibrated judgment module. On MMFakeBench validation, it reports 81.65% F1 and 75.1% accuracy, outperforming GPT-4V with MMD-Agent at 74.0% F1 by 7.65 points; on the test set it reports 81.44% F1 and 75.08% accuracy. Its ablations attribute 5.18 F1 points to visual verification and 2.97 to retrieval-augmented reasoning (Shopnil et al., 20 Oct 2025). RamDG, by contrast, integrates retrieved celebrity images and metadata from CAP into a multimodal detector for semantically coordinated manipulation. On SAMM, it reports 94.66% detection accuracy, 2.06% higher than the strongest state-of-the-art baseline, as well as 80.90% IoUmean for image grounding and 84.83% F1 for text grounding (Shen et al., 16 Sep 2025).

5. Generative, reasoning-centric, and concept-based grounding

A third family treats GroundMM as a reasoning problem in which missing or weakly informative modality content must be supplemented, explained, or routed through explicit reasoning structures.

RETSIMD begins from the claim that, in multimodal misinformation detection, text often comprehensively describes the whole event, while images only present partial scenes. It therefore splits text into segments, uses a post-trained Stable Diffusion model to generate a sequence of images, imposes text-image mutual information and image-label mutual information objectives, and fuses the original and generated images with a graph whose edges encode central, temporal, and dependency relations. The information-gain measure for the image modality is defined as

$\mathcal{V}$1

The paper reports that $\mathcal{V}$2 is best for segmentation, that the generator is post-trained with high-quality image-text pairs such as LAION-2B, and that RETSIMD improves GossipCop accuracy from 87.32 to 88.42 and Twitter accuracy from 66.71 to 69.57 relative to BMR, while consistently increasing $\mathcal{V}$3 (Wang et al., 9 Nov 2025).

MMGrounded-PostAlign addresses GroundMM as multimodal hallucination. It produces structured outputs

$\mathcal{V}$4

where $\mathcal{V}$5 is a visual grounding token, $\mathcal{V}$6 a textual rationale, and $\mathcal{V}$7 a final answer. The framework adds a negative rejection mechanism via a <REJ> token and a selective reasoning strategy that distinguishes <SIMPLE> from <COMPLEX> queries. On hallucination benchmarks, MMGrounded-PostAlign-13B reports 88.9 / 87.3 / 85.6 on POPE (Random/Popular/Adversarial), 33.9 on HaloQuest (False Premise), and 89.2 / 79.7 on RefCOCO (REC/RES), while also reporting no drop in performance on VQAv2, MME, and MMBench (Wu et al., 22 Jun 2025).

MMD-Thinker inserts task-specific reasoning trajectories into MLLMs. It defines three thinking modes—Quick Response, Semantic Analysis, and Prospective Simulation—and trains them using task-specific instruction tuning plus reinforcement learning with a mixed advantage function. The accompanying MMR dataset contains more than 8K image-text pairs with both reasoning processes and classification labels. Reported results for Qwen2.5-VL-7B, SFT + MMPO are 90.74 F1 and 92.90 accuracy in-domain on MMR, plus 50.86 F1 on PHEME and 62.53 F1 on Twitter, with 68.04 average F1 across tasks (Wu et al., 17 Nov 2025).

PCGR makes the reasoning structure explicit by building a probabilistic concept graph whose nodes are human-readable predicates and whose edges encode semantic, statistical, and logical relations. Hierarchical attention weights are defined over parent concepts,

$\mathcal{V}$8

and node aggregation uses

$\mathcal{V}$9

The framework uses MLLMs such as Qwen3-Omni and GPT-5 to propose and validate new concepts from hard examples. It reports coarse-level accuracies of 80.2% on MiRAGeNews, 80.6% on MMFakeBench, and 84.3% on AMG, as well as fine-grained Mic-F1/Mac-F1 of 68.6/56.9 on MMFakeBench and 75.6/59.9 on AMG (Yang et al., 26 Mar 2026). This suggests a transition from opaque end-to-end detectors toward explicitly auditable reasoning chains.

6. Challenges, misconceptions, and research directions

A persistent misconception is that multimodal misinformation is primarily a matter of obvious image–text mismatch. Multiple papers reject this assumption. “The Coherence Trap” argues that prevailing techniques underestimate MLLM-driven deception risk and rely on unrealistic misalignment artifacts, whereas modern attacks can generate semantically coherent and contextually plausible yet deceptive narratives conditioned on manipulated images (Zhang et al., 23 May 2025). SAMM is motivated by the same criticism, namely that practical attacks often maintain semantic consistency across modalities and therefore evade detectors trained on artificially disrupted alignment (Shen et al., 16 Sep 2025).

A second misconception is that multimodal grounding is equivalent to binary detection. The GroundLie360 formulation explicitly requires verify, localize, and explain misleading segments across modalities, and the DGM$\mathcal{T}$0 line similarly treats grounding as bounding-box and token localization rather than post hoc explanation (Yang et al., 8 Sep 2025, Shao et al., 2023). The difficulty of this requirement is visible in baseline results: on GroundLie360, FakeMark reports subtype F1 of 10.49 for Temporal Edit, 16.10 for Contradictory Content, and textual grounding F1 in the low teens or below for several categories; frame-level grounding F1 for Temporal Edit is 2.37, while CGI grounding reports 13.05 mean temporal IoU and 7.74 mean visual IoU (Yang et al., 8 Sep 2025).

A third recurring issue is modality imbalance. RETSIMD argues that the image modality often contributes less because text describes the full event while images present only partial scenes (Wang et al., 9 Nov 2025). The synthetic-misinformer study similarly reports that NEI-based Synthetic Misinformers tend to suffer from a unimodal bias, where text-only MMDs can outperform multimodal ones, and that after fixing information leakage in COSMOS, the best multimodal accuracy is only 58.1%, with Sensitivity remaining below 50% (Papadopoulos et al., 2023). Domain shift adds another layer of difficulty: RDCM is explicitly designed for settings where target-domain data may be unavailable or only unlabeled, and reports gains on Pheme and Twitter by combining inter-domain alignment with cross-modality alignment (Liu et al., 2023).

The literature also emphasizes evaluation and scope problems familiar from general multimodal grounding. The 2018 survey notes that multimodal grounding for open-domain language remains constrained by ambiguity in multimodal mappings, compositional complexity, limited datasets, and the difficulty of evaluating subjective cross-modal interpretation (Beinborn et al., 2018). In GroundMM, those generic issues become sharper because misinformation often mixes weak visual evidence, context-dependent textual claims, reused media, and rapidly changing world facts. This suggests that current progress is less a convergence on a single architecture than a diversification of grounding strategies: fine-grained localization, retrieval over open-domain evidence, reasoning-trajectory supervision, concept graphs, and mask-anchored bidirectional verification all address different failure modes identified by the field.