---
title: 'GroundMM: Multimodal Misinformation Grounding'
url: https://www.emergentmind.com/topics/grounding-multimodal-misinformation-groundmm
type: topic
---

# GroundMM: Multimodal Misinformation Grounding

Grounding Multimodal Misinformation (GroundMM) denotes a line of research—and, in one formalization, a task—in which multimodal content is not only classified as authentic or misleading, but also localized, explained, or otherwise anchored to evidence across modalities. In this literature, “grounding” refers to linking claims, captions, speech, or generated rationales to concrete visual regions, text spans, temporal segments, or retrieved external evidence. GroundMM therefore departs from binary fake/real classification and from single-modality forensics by requiring multimodal verification plus interpretable attribution of where the misinformation resides or what evidence supports the verdict [2509.08008][1806.06371].

## 1. Conceptual foundations

The broader grounding literature characterizes multimodal processing through **multimodal fusion**, **mapping (projection)**, **joint learning**, and **compositional representations** [1806.06371]. In that framework, fusion combines representations directly, mapping learns functions from one modality to another, joint learning optimizes unimodal and cross-modal objectives together, and compositionality builds phrase- or sequence-level representations from multimodal concept representations. The same survey organizes architectures by **cross-modal transfer**, **cross-modal interpretation**, and **joint multimodal processing**, aligning technical pipelines with different information-flow assumptions [1806.06371].

GroundMM inherits these representational issues but places them in an adversarial setting. The central question is not merely whether image and text are correlated, but whether they are **truthfully** correlated, whether one modality has been manipulated, whether the pair is out-of-context, and whether the model can indicate the manipulated content or retrieve evidence that resolves ambiguity. This is why the literature repeatedly distinguishes multimodal misinformation from classical deepfake detection and from text-only fake news detection. A recurring theme is that multimodal misinformation often requires deeper reasoning about semantic consistency, event context, and evidence provenance rather than only artifact detection [2304.02556][2507.09174].

A formal GroundMM instance in the video setting is written as $x = (\mathcal{V}, \mathcal{T}, \mathcal{R})$, where $\mathcal{V}$ is video, $\mathcal{T}$ is title text, and $\mathcal{R}$ is transcript. The prediction targets are a veracity label $\hat{y} \in \{0,1\}$, a fake-type vector $\hat{\mathbf{c}} \in \{0,1\}^6$, and grounding outputs $\hat{\mathbf{G}} = \{\hat{G}_i \mid c_i=1\}$ [2509.08008]. This formulation makes explicit that GroundMM is simultaneously a verification problem, a structured prediction problem, and a localization problem.

## 2. Task formulations and benchmark datasets

The benchmark landscape shows a progression from synthetic image–text manipulation datasets to more realistic, semantically coordinated, and multimodal video settings. The earliest large-scale formulations emphasize image bounding boxes and manipulated text tokens; later benchmarks add temporal spans, speech grounding, and external fact-check evidence. This evolution reflects a shift from detecting artificial mismatches to grounding realistic misinformation patterns [2304.02556][2509.08008].

| Benchmark | Modalities | Key properties |
|---|---|---|
| DGM$^4$ | image-text | 230,000 pairs; binary label, 4D manipulation type label, image bounding boxes, text token annotations |
| GroundLie360 | video, title text, ASR transcript | 2,044 target videos across 1,466 events, 588 auxiliary videos, 6 fake types, fine-grained grounding |
| MDSM | image-text | 441,423 semantically coherent manipulated pairs with fake detection, manipulation type, and manipulated image region bounding box |
| SAMM | image-text with external celebrity knowledge | ~261,000 pairs: 111k original, 80k swap manipulations, 69k attribute manipulations, image box and token-level text labels |

**DGM$^4$** formalized “Detecting and Grounding Multi-Modal Media Manipulation” as a task with four manipulation types: **Face Swap (FS)**, **Face Attribute (FA)**, **Text Swap (TS)**, and **Text Attribute (TA)**. The dataset contains **230,000 image-text pairs**, including **77,426 pristine** and **152,574 manipulated** samples, with **binary labels**, **fine-grained manipulation types**, **manipulated image bounding boxes**, and **manipulated text tokens** [2304.02556]. The later HAMMER++ description reports the dataset at approximately **230k news samples**, with **77,426 pristine** and **152,574 manipulated pairs**, and notes that **image perturbations** such as JPEG compression and blur are randomly applied to **50% of samples** [2309.14203].

**GroundLie360** introduced GroundMM explicitly for web video misinformation. It is described as the **first real-world dataset for this task**, with **2,044 target videos (under 5 mins) across 1,466 events**, plus **588 auxiliary original/debunking/evidence videos**. Its annotation taxonomy spans **False Title**, **False Speech**, **Temporal Edit**, **CGI**, **Contradictory Content**, and **Unsupported Content**, with token-level text grounding, temporal video spans, and frame-level bounding boxes. Reported inter-annotator IoU scores are **0.74** for text, **0.84** for temporal spans, and **0.98** for bounding boxes [2509.08008].

Two later datasets address a methodological criticism of earlier benchmarks: that many multimodal manipulations are too obviously misaligned. **MDSM** constructs semantically coherent manipulations by first editing images and then generating matching deceptive narratives with MLLMs; it contains **441,423** annotated image-text pairs, with labels for fake detection, manipulation type (**FS**, **FA**, **TF**), and manipulated image regions [2505.17476]. **SAMM** similarly targets **semantically-coordinated manipulations** using a two-stage pipeline and a **Celeb Attributes Portfolio (CAP)** as an external knowledge base; it reports **~261,000 pairs**, comprising **111k original**, **80k swap manipulations**, and **69k attribute manipulations**, with ground-truth bounding boxes and token-level text labels [2509.12653].

## 3. Localized and alignment-based architectures

A major methodological family treats GroundMM as a fine-grained alignment problem between image regions and text tokens. **HAMMER** introduced a hierarchical design with **manipulation-aware contrastive learning** as “shallow manipulation reasoning” and **modality-aware cross-attention** as “deep manipulation reasoning.” It combines binary and multi-label detection heads with bounding-box and token-grounding heads. **HAMMER++** adds **local-view manipulation-aware contrastive learning** to strengthen patch/token alignment. On DGM$^4$, the reported metrics for **HAMMER++** are **93.33** AUC for binary classification, **86.41** mAP for multi-label classification, **76.46** IoUmean for image grounding, and **72.59** F1 for text grounding [2309.14203].

**FMS** extends this line by explicitly supervising modality reliability, unimodal forgery cues, and cross-modal interaction. Its three modules are **Multimodal Decision Supervised Correction (MDSC)**, **Unimodal Forgery Mining Reinforcement (UFMR)**, and **Multimodal Forgery Alignment Reasoning (MFAR)**. The stated purpose is to correct the multi-modal decision-making process when one modality is unreliable, amplify disparities between real and fake content within each modality, and reason over consistency and inconsistency across modalities. On DGM$^4$, **FMS** reports **96.46% AUC**, **93.43% mAP**, **84.82% IoUmean**, and **75.34%** text-grounding F1 [2508.02479].

**MaLSF** shifts the design from passive holistic fusion to “active, bidirectional verification.” Its core idea is to use **mask-label pairs** as local semantic anchors that explicitly connect pixel regions to labels. It then applies a **Bidirectional Cross-modal Verification (BCV)** module with **Text-as-Query** and **Image-as-Query** streams, followed by **Hierarchical Semantic Aggregation (HSA)**. On DGM4, **MaLSF** reports **89.37** binary accuracy, **83.93** multi-label CF1, **82.37/82.47** IoUmean for image grounding, and **76.88/77.19** text-grounding F1, depending on the parser variant; on **Weibo21** and **Weibo17**, it reports **95.5%** and **93.5%** accuracy, respectively [2603.26052].

A contrasting design is **LADLE-MM**, which targets limited-annotation settings. It uses two unimodal branches plus a frozen **BLIP** multimodal branch as a fixed reference space, and it does **not** use explicit grounding annotations during training. The model is reported to use **60.3% fewer trainable parameters** than previous state-of-the-art models, to achieve **87.90** mAP on DGM4 multi-label classification, and to reach **79.6%** accuracy and **83.5%** F1 on **VERITE**, where it is described as robust to unimodal bias [2512.20257]. This suggests that some grounding effects can emerge from alignment to a stable multimodal reference space even when bounding-box and token-level supervision are absent.

## 4. Retrieval-augmented and evidence-grounded verification

A second family grounds misinformation decisions in **external evidence** rather than only in internal cross-modal alignment. In these systems, grounding is operationalized through retrieval, re-ranking, evidence aggregation, or graph reasoning over retrieved documents.

A zero-shot pipeline for multimodal misinformation detection with large vision-language models combines **initial retrieval**, **re-ranking** with LLMs and LVLMs, and **fact verification** using the top evidence. On a more completely annotated subset of **MOCHEG**, the proposed evidence retrieval method reports **90.00** top-1 mAP for both text and image evidence, and on fact verification with **gold multimodal evidence**, **LLaVA** reports micro F1 of **0.534** on **MOCHEG** and **0.595** on **Factify** [2407.14321]. The same work argues that earlier evaluations understated retrieval quality because of incomplete ground-truth evidence labels.

**RAMA** organizes evidence-grounded verification into **WebRetriever**, **VLJudge**, and **DecisionFuser**. The retrieval module uses a tool-augmented iterative loop for query planning, searching, browsing, and synthesis; the judgment module deploys three agents, two based on **Qwen-VL** and one on **InternVL**; and the final verdict uses weighted majority voting. On the **1,000 image-caption pairs** of the Multimedia Verification Challenge public test set, **RAMA** reports **0.9100** accuracy and **0.9100** F1 using only the first caption, while the paper emphasizes that it performs **without any training or fine-tuning** and is less sensitive to dataset shifts [2507.09174].

**EGMMG** represents evidence and claims as graphs. It performs **reverse image search** to collect web pages containing the image, builds an **evidence graph** from retrieved text and a **claim graph** from the caption, and then compares them using attention-based GNNs. The reported final encoder is **TransformerConv**, and the method reaches **93.05%** detection accuracy on the evaluation set, outperforming the second-best performing method by **2.82%** [2505.18221]. Here the grounding signal is not a localized image box or token span, but a structured correspondence between retrieved event/entity graphs and claim graphs.

Two later systems push evidence-grounded reasoning toward more modular inference-time agents. **MIRAGE** decomposes detection into **visual veracity assessment**, **cross-modal consistency analysis**, **retrieval-augmented factual checking**, and a **calibrated judgment module**. On **MMFakeBench** validation, it reports **81.65% F1** and **75.1% accuracy**, outperforming **GPT-4V with MMD-Agent at 74.0% F1** by **7.65** points; on the test set it reports **81.44% F1** and **75.08% accuracy**. Its ablations attribute **5.18** F1 points to visual verification and **2.97** to retrieval-augmented reasoning [2510.17590]. **RamDG**, by contrast, integrates retrieved celebrity images and metadata from **CAP** into a multimodal detector for semantically coordinated manipulation. On **SAMM**, it reports **94.66%** detection accuracy, **2.06% higher** than the strongest state-of-the-art baseline, as well as **80.90%** IoUmean for image grounding and **84.83%** F1 for text grounding [2509.12653].

## 5. Generative, reasoning-centric, and concept-based grounding

A third family treats GroundMM as a reasoning problem in which missing or weakly informative modality content must be supplemented, explained, or routed through explicit reasoning structures.

**RETSIMD** begins from the claim that, in multimodal misinformation detection, **text often comprehensively describes the whole event, while images only present partial scenes**. It therefore splits text into segments, uses a post-trained **Stable Diffusion** model to generate a sequence of images, imposes **text-image mutual information** and **image-label mutual information** objectives, and fuses the original and generated images with a graph whose edges encode **central**, **temporal**, and **dependency** relations. The information-gain measure for the image modality is defined as
\[
\mathscr{G}(y, \mathbf{x}^v) = \mathcal{H}(y \mid \varnothing, \mathbf{x}^t) - \mathcal{H}(y \mid \mathbf{x}^v, \mathbf{x}^t).
\]
The paper reports that **$K=5$ is best** for segmentation, that the generator is post-trained with high-quality image-text pairs such as **LAION-2B**, and that RETSIMD improves **GossipCop** accuracy from **87.32** to **88.42** and **Twitter** accuracy from **66.71** to **69.57** relative to **BMR**, while consistently increasing $\mathscr{G}(y,\mathbf{x}^v)$ [2511.06284].

**MMGrounded-PostAlign** addresses GroundMM as multimodal hallucination. It produces structured outputs
\[
\mathcal{A} = \{\mathcal{V}, \mathcal{T}, \mathcal{F}\},
\]
where $\mathcal{V}$ is a visual grounding token, $\mathcal{T}$ a textual rationale, and $\mathcal{F}$ a final answer. The framework adds a **negative rejection mechanism** via a **`<REJ>`** token and a **selective reasoning** strategy that distinguishes **`<SIMPLE>`** from **`<COMPLEX>`** queries. On hallucination benchmarks, **MMGrounded-PostAlign-13B** reports **88.9 / 87.3 / 85.6** on **POPE** (Random/Popular/Adversarial), **33.9** on **HaloQuest (False Premise)**, and **89.2 / 79.7** on **RefCOCO (REC/RES)**, while also reporting **no drop in performance** on **VQAv2**, **MME**, and **MMBench** [2506.17901].

**MMD-Thinker** inserts task-specific reasoning trajectories into MLLMs. It defines three thinking modes—**Quick Response**, **Semantic Analysis**, and **Prospective Simulation**—and trains them using task-specific instruction tuning plus reinforcement learning with a **mixed advantage function**. The accompanying **MMR** dataset contains **more than 8K image-text pairs with both reasoning processes and classification labels**. Reported results for **Qwen2.5-VL-7B, SFT + MMPO** are **90.74** F1 and **92.90** accuracy in-domain on **MMR**, plus **50.86** F1 on **PHEME** and **62.53** F1 on **Twitter**, with **68.04** average F1 across tasks [2511.13242].

**PCGR** makes the reasoning structure explicit by building a **probabilistic concept graph** whose nodes are human-readable predicates and whose edges encode semantic, statistical, and logical relations. Hierarchical attention weights are defined over parent concepts,
\[
\alpha_{ij} = \frac{\exp(t_i \cdot t_j)}{\sum_{j' \in \text{Pa}(i)} \exp(t_i \cdot t_{j'})},
\]
and node aggregation uses
\[
\hat{p}_i = \lambda p_i \cdot (1-\lambda) \prod_{j \in \text{Pa}(i)} (\alpha_{ij} p_j).
\]
The framework uses MLLMs such as **Qwen3-Omni** and **GPT-5** to propose and validate new concepts from hard examples. It reports coarse-level accuracies of **80.2%** on **MiRAGeNews**, **80.6%** on **MMFakeBench**, and **84.3%** on **AMG**, as well as fine-grained **Mic-F1/Mac-F1** of **68.6/56.9** on **MMFakeBench** and **75.6/59.9** on **AMG** [2603.25203]. This suggests a transition from opaque end-to-end detectors toward explicitly auditable reasoning chains.

## 6. Challenges, misconceptions, and research directions

A persistent misconception is that multimodal misinformation is primarily a matter of obvious image–text mismatch. Multiple papers reject this assumption. **“The Coherence Trap”** argues that prevailing techniques underestimate **MLLM-driven deception risk** and rely on **unrealistic misalignment artifacts**, whereas modern attacks can generate **semantically coherent and contextually plausible yet deceptive narratives conditioned on manipulated images** [2505.17476]. **SAMM** is motivated by the same criticism, namely that practical attacks often maintain semantic consistency across modalities and therefore evade detectors trained on artificially disrupted alignment [2509.12653].

A second misconception is that multimodal grounding is equivalent to binary detection. The GroundLie360 formulation explicitly requires **verify**, **localize**, and **explain** misleading segments across modalities, and the DGM$^4$ line similarly treats grounding as bounding-box and token localization rather than post hoc explanation [2509.08008][2304.02556]. The difficulty of this requirement is visible in baseline results: on GroundLie360, **FakeMark** reports subtype F1 of **10.49** for **Temporal Edit**, **16.10** for **Contradictory Content**, and textual grounding F1 in the low teens or below for several categories; frame-level grounding F1 for **Temporal Edit** is **2.37**, while **CGI** grounding reports **13.05** mean temporal IoU and **7.74** mean visual IoU [2509.08008].

A third recurring issue is **modality imbalance**. RETSIMD argues that the image modality often contributes less because text describes the full event while images present only partial scenes [2511.06284]. The synthetic-misinformer study similarly reports that **NEI-based Synthetic Misinformers tend to suffer from a unimodal bias, where text-only MMDs can outperform multimodal ones**, and that after fixing information leakage in **COSMOS**, the best multimodal accuracy is only **58.1%**, with **Sensitivity** remaining below **50%** [2303.01217]. Domain shift adds another layer of difficulty: **RDCM** is explicitly designed for settings where target-domain data may be unavailable or only unlabeled, and reports gains on **Pheme** and **Twitter** by combining **inter-domain alignment** with **cross-modality alignment** [2311.14315].

The literature also emphasizes evaluation and scope problems familiar from general multimodal grounding. The 2018 survey notes that multimodal grounding for open-domain language remains constrained by **ambiguity in multimodal mappings**, **compositional complexity**, **limited datasets**, and the difficulty of evaluating subjective cross-modal interpretation [1806.06371]. In GroundMM, those generic issues become sharper because misinformation often mixes weak visual evidence, context-dependent textual claims, reused media, and rapidly changing world facts. This suggests that current progress is less a convergence on a single architecture than a diversification of grounding strategies: fine-grained localization, retrieval over open-domain evidence, reasoning-trajectory supervision, concept graphs, and mask-anchored bidirectional verification all address different failure modes identified by the field.

Source: https://www.emergentmind.com/topics/grounding-multimodal-misinformation-groundmm