DGM⁴: Multimodal Media Manipulation Benchmark

Updated 9 February 2026

DGM⁴ is a benchmark dataset and evaluation framework that supports fine-grained detection, classification, and grounding of subtle multimodal manipulations.
It comprises approximately 230,000 news-style image–caption pairs annotated with manipulation types and precise grounding labels for both images and text.
The framework drives advances in automated multimodal misinformation detection through unified tasks and robust evaluation metrics like mAP, IoU, and token-level F1.

Detecting and Grounding Multi-Modal Media Manipulation (DGM⁴) is a benchmark dataset, task taxonomy, and evaluation framework that underpins a line of research on fine-grained detection, classification, and localization of coordinated visual and textual forgeries in human-centric news-style media. DGM⁴ addresses the technical challenge of identifying and grounding subtle semantic, contextual, and pixel-level manipulations in image–text pairs, with the aim of spurring progress in automated multimodal misinformation detection across social media and journalistic contexts (Zhang et al., 2024, Wang et al., 2023, Li et al., 6 Jun 2025, Singh et al., 30 Sep 2025, Singh et al., 30 Sep 2025, Cardullo et al., 23 Dec 2025, Sagar et al., 2 Feb 2026).

1. Dataset Composition and Annotation

DGM⁴ contains approximately 230,000 samples, partitioned into news-style image–caption pairs spanning domains such as politics, health, crisis, and entertainment. Each sample consists of an RGB image and a short caption, either curated or crawled from real outlets. The dataset is balanced (33.7% pristine, 66.3% manipulated in the main split (Zhang et al., 2024)) and supports four canonical manipulation types:

Face Swap (FS): A subject's face is replaced, yielding a new identity.
Face Attribute (FA): Facial details such as age, gender, or expression are edited.
Text Swap (TS): The caption’s subject–action–object tuple is altered for semantic inconsistency.
Text Attribute (TA): Caption wording is modified to shift sentiment, tense, or fine detail.

Each pair is annotated with:

A binary flag $y_{\mathrm{bin}}\in\{0,1\}$ (real/fake).
Fine-grained multi-label vector $y_{\mathrm{mul}}\in\{0,1\}^4$ for the four manipulation types.
Grounding annotations: bounding boxes for manipulated facial regions and token-level binary masks for modified words.

The expanded DGM⁴⁺ extension (Singh et al., 30 Sep 2025) introduces 5,000 samples with global scene inconsistencies (Foreground-Background, FG–BG mismatch) and combinations thereof with TA and TS manipulations, thus probing both local and global multimodal reasoning.

2. Task Taxonomy and Evaluation Metrics

DGM⁴ supports a unified benchmark for four core tasks:

Binary Detection: Classify each pair as pristine or manipulated, based on all cues.
Multi-Label Type Classification: For manipulated examples, label all applicable manipulation types (FS, FA, TS, TA).
Grounding (Image): Localize forged regions via bounding box or pixel mask, particularly for face-centric manipulations.
Grounding (Text): Identify manipulated word tokens via a binary mask corresponding to each caption token.

Standard metrics used in evaluation include

Precision: $TP/(TP+FP)$
Recall: $TP/(TP+FN)$
F1: $2 \cdot (\mathrm{Precision} \cdot \mathrm{Recall})/(\mathrm{Precision}+\mathrm{Recall})$
Mean Average Precision (mAP): Task-level average across manipulation types.
Intersection over Union (IoU): For box/mask region localization.
Token-level F1: For text grounding. No special reweighting is applied; all splits are balanced and standard thresholds are used (Sagar et al., 2 Feb 2026, Zhang et al., 2024, Wang et al., 2023, Li et al., 6 Jun 2025, Cardullo et al., 23 Dec 2025).

Task	Annotation	Example Metric
Binary Detection	$y_{\mathrm{bin}}$	Accuracy, F1
Type Classification	$y_{\mathrm{mul}}$	mAP, CF1, OF1
Image Grounding	$y_{\mathrm{box}}$	IoU, [email protected]
Text Grounding	$y_{\mathrm{tok}}$	token-F1

3. Manipulation Scenarios and Dataset Generation

Beyond GAN/diffusion-based image fabrications, DGM⁴ encompasses semantically challenging manipulations:

Out-of-context reuse: re-captioned authentic images attached to spurious narratives.
Semantic distortions: captions assigning events or agents not present in the image.
Temporal/provenance confabulation: dating or localizing authentic images incorrectly.

All pairs are manually verified using external sources (news archives, reverse image search, public records) to enforce ground-truth label validity at the claim level (Sagar et al., 2 Feb 2026). DGM⁴⁺ global manipulations are generated with gpt-image-1 (OpenAI 2025) and systematically quality-checked using face-gate, OCR scrubbing, and perceptual hash deduplication (Singh et al., 30 Sep 2025).

4. Methodological Advances and Model Families

DGM⁴ catalyzed a sequence of modeling innovations:

Dual-Modality Encoders and Fusion: Transformer-based branches for images and text with cross-modality attention (Wang et al., 2023).
Fine-Grained Consistency Learning: CSCL introduces contextual consistency decoders (CCD) and semantic consistency decoders (SCD), aligning token/patch pairs both within and across modalities for robust grounding (Li et al., 6 Jun 2025).
Semantic Alignment and Attention: ASAP implements large-model-assisted auxiliary supervision and manipulation-guided cross-attention (MGCA), improving alignment and grounding accuracy (Zhang et al., 2024).
Parameter-Efficient Model Ensembles: LADLE-MM leverages a BLIP-based multimodal anchor and model-soup initialized ensembling to match SOTA with minimal annotation requirements (Cardullo et al., 23 Dec 2025).
Region-Aware Reasoning for Global Scene: SGS post-hoc scoring uses segmentation-based region partition and cross-region caption embedding alignment to detect out-of-distribution FG–BG mismatches, addressing a key blind spot of earlier models (Singh et al., 30 Sep 2025).

5. Benchmark Results, Limitations, and Failure Modes

Recent studies have systematized performance benchmarking. CSCL achieves SOTA metrics across DGM⁴ tasks (AUC 96.34%, OF1 86.92, image mIoU 84.07%), with ASAP/other top models showing similar strengths, especially in grounding manipulated content (Li et al., 6 Jun 2025, Zhang et al., 2024):

Deepfake detectors are marginal at claim verification (F1 = 0.33–0.49), and incorporating their outputs into fact-checking pipelines reduces performance by ~0.04 F1 due to over-reliance on non-causal authenticity priors (Sagar et al., 2 Feb 2026).
Evidence-driven multimodal systems relying on external verification systematically outperform artifact-only or shallow fusion approaches.
Performance on global scene manipulations (FG–BG) reveals stark model gaps: HAMMER achieves only 19.1% binary classification accuracy on DGM⁴⁺, while post-hoc SGS achieves 94.3% (Singh et al., 30 Sep 2025, Singh et al., 30 Sep 2025).
Models trained only on local manipulations often cannot generalize to compositional or scene-level inconsistencies. Type-heads are typically label-space limited and lack “out-of-vocabulary” support for FG–BG (Singh et al., 30 Sep 2025, Singh et al., 30 Sep 2025).

6. Model Analysis, Ablations, and Insights

Ablation studies consistently demonstrate:

Cross-modal alignment and consistency decoders (CSCL, ASAP) yield significant gains in both precision and grounding accuracy. For example, adding CCD/SCD in CSCL increases image mean IoU by >2.8 points and text-F1 by >2.1 (Li et al., 6 Jun 2025).
Tri-branch frameworks with fixed multimodal anchors (LADLE-MM) maintain robustness with up to 60% fewer parameters (Cardullo et al., 23 Dec 2025).
Region-based post-hoc methods (SGS) can radically enhance model performance for global manipulations with virtually no computational overhead—by integrating region-level coherence scores, a legacy model's accuracy is boosted by >75 percentage points on FG–BG splits (Singh et al., 30 Sep 2025).
Generalization gaps persist: models achieving mAP ≈88% on DGM⁴ may drop to 64% on external benchmarks (Fakeddit, VERITE), indicating sensitivity to dataset bias and manipulation diversity (Wang et al., 2023, Cardullo et al., 23 Dec 2025).

7. Open Challenges and Future Directions

Critical directions identified include:

Expanding label and model architectures to explicitly handle global inconsistencies (i.e., FG–BG), through vocabulary extension, scene descriptor integration, and region-aware contrastive losses (Singh et al., 30 Sep 2025, Singh et al., 30 Sep 2025).
Reducing dependency on external model supervision for auxiliary captions and explanations (e.g., end-to-end aligners) (Zhang et al., 2024).
Bridging the domain gap to out-of-distribution or open-world manipulations, and handling multimodal and temporal extensions—especially video or audio-visual deepfakes (Li et al., 6 Jun 2025).
Exploiting segmentation-based reasoning, multi-modal SCL, and patch/region-level modeling for detection and grounding at both fine and global scales.
Advancing evaluation protocols to emphasize global scene plausibility and compositionality, not merely local artifact detection (Singh et al., 30 Sep 2025).

DGM⁴ remains foundational to empirical study of multimodal disinformation, motivating the field toward unified, explainable, evidence-driven, and region/plausibility-aware verification systems.