Retrieval-Augmented Manipulation Detection (RamDG)

• The paper introduces RamDG which jointly detects and grounds fine-grained semantic manipulations using an external celebrity knowledge base from CAP.
• RamDG employs retrieval augmentation with contrastive and visual refinement modules to outperform prior methods in both detection accuracy and localization.
• It addresses semantically coordinated image-text manipulations in fake news, demonstrating robust performance even in low-resource settings.

Retrieval-Augmented Manipulation Detection and Grounding (RamDG) is a multimodal media-forensics framework introduced in "Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations" (Shen et al., 16 Sep 2025). It targets manipulated image-text news pairs in which visual edits are systematically paired with semantically consistent textual descriptions, rather than obvious image-text mismatches. In this setting, the image may be altered by face swapping or facial attribute editing, the text is also forged so it stays semantically plausible with the changed image, and the manipulations are small, localized, and coordinated across modalities. RamDG retrieves external celebrity knowledge from the Celeb Attributes Portfolio (CAP), encodes that evidence together with the input pair, and jointly performs fake news binary recognition, manipulation type recognition, image forgery grounding, and text manipulation localization (Shen et al., 16 Sep 2025).

1. Problem formulation and conceptual motivation

RamDG is motivated by a critique of prevailing multimodal manipulation benchmarks. The central argument is that many existing benchmarks create fake samples by artificially breaking semantic alignment between image and text: a relevant image is paired with an unrelated caption, a name is randomly replaced, or words are swapped in a way that makes the text obviously inconsistent with the image. Such setups enable shallow cross-modal mismatch detection. RamDG instead addresses semantically coordinated manipulations, where both modalities remain mutually plausible while the combined news item is false (Shen et al., 16 Sep 2025).

This problem setting is closely related to the earlier formalization of Detecting and Grounding Multi-Modal Media Manipulation (DGM$^4$), which defined the input as an image-text pair $(I,T)\sim P$ and annotated each pair with a binary authenticity label $y_{\rm bin}$, a fine-grained manipulation-type vector $y_{\rm mul}=\{y_j\}_{j=1}^4$, manipulated image bounding boxes $y_{\rm box}$, and manipulated text-token labels $y_{\rm tok}$. In DGM$^4$, the fine-grained manipulation classes are Face Swap (FS), Face Attribute (FA), Text Swap (TS), and Text Attribute (TA), and the task decomposes into binary classification, multi-label classification, image grounding, and text grounding (Shao et al., 2023).

RamDG preserves that multi-task perspective but changes the evidential setting. DGM$^4$ and HAMMER reason directly over the input pair without retrieval over external corpora, whereas RamDG explicitly harnesses external knowledge repositories to retrieve contextual evidence and uses that evidence in both detection and grounding (Shao et al., 2023). This shift matters because semantically coordinated attacks suppress the very inconsistency signals on which many earlier systems rely.

2. Data resources: from DGM$^4$ to SAMM and CAP

The data lineage around RamDG is anchored by two complementary resources. DGM$^4$ introduced the first large-scale dataset for detecting and grounding multi-modal media manipulation, built from VisualNews and focused on human-centric news. It totals 230k samples: 77,426 pristine and 152,574 manipulated. The manipulated set includes 66,722 FS, 56,411 FA, 43,546 TS, and 18,588 TA pairs, plus 32,693 mixed-manipulation pairs where image and text manipulations are combined. The construction process also added random perturbations such as JPEG compression and blur to 50% of the pool to simulate noise (Shao et al., 2023).

RamDG is introduced together with a new benchmark, SAMM, short for Semantic-Aligned Multimodal Manipulation. SAMM contains 260,970 samples and is designed to reflect semantically coordinated fake news. It is built from VisualNews and GoodNews, with filtering to ensure that each news item contains at least one person and that highly redundant images are removed using CLIP-based similarity filtering. Its base set is denoted

$(I,T)\sim P$0

where $(I,T)\sim P$1 is the source image and $(I,T)\sim P$2 is the source text. The final composition is 111K original news, 80K swap-manipulated, and 69K attribute-manipulated samples (Shen et al., 16 Sep 2025).

A second resource, CAP, is external rather than benchmark-internal. CAP, the Celeb Attributes Portfolio, contains multimodal celebrity knowledge gathered from the web via Google Search API, including celebrity images, gender, birth year, occupation, and main achievements. CAP serves two roles simultaneously: it helps drive SAMM construction and it functions as the retrieval source for RamDG (Shen et al., 16 Sep 2025).

SAMM’s manipulation pipeline is explicitly two-stage: first manipulate the image, then generate a semantically plausible text. In the swap case, a celebrity face $(I,T)\sim P$3 and corresponding name $(I,T)\sim P$4 are randomly selected from CAP; SimSwap or InfoSwap generates the manipulated image $(I,T)\sim P$5, and the swapped face bounding box $(I,T)\sim P$6 is recorded. Qwen2.5 extracts all names from the original text, Qwen2-VL matches the face region to the correct original identity $(I,T)\sim P$7, and all occurrences of $(I,T)\sim P$8 are replaced with $(I,T)\sim P$9, producing $y_{\rm bin}$0 with a word-level tampering label. In the attribute case, DSFD detects a face and records its bounding box, Qwen2-VL predicts the emotional state, and HFGI or StyleCLIP produces a manipulated face $y_{\rm bin}$1; emotion-related language is then inserted into the text using an emotion expression set

$y_{\rm bin}$2

again with word-level tampering labels (Shen et al., 16 Sep 2025).

The conceptual significance of SAMM is that it replaces purposeless tampering and artificial cross-modal inconsistency with purposeful tampering and aligned manipulation across image and text. This suggests a stricter evaluation regime for multimodal forensics, because success can no longer be attributed primarily to shallow alignment cues.

3. Retrieval augmentation and CAP-aided context-aware encoding

RamDG’s retrieval stage begins from an input image-text pair

$y_{\rm bin}$3

Person names are extracted from the text, and string matching is used to retrieve associated celebrity entries from CAP: $y_{\rm bin}$4 These retrieved entries act as auxiliary knowledge (Shen et al., 16 Sep 2025).

The image branch uses a ViT-based image encoder $y_{\rm bin}$5. The news image and the retrieved celebrity images are patchified and encoded as

$y_{\rm bin}$6

$y_{\rm bin}$7

All retrieved celebrity patch features are concatenated into $y_{\rm bin}$8, and cross-attention injects celebrity knowledge into the image representation: $y_{\rm bin}$9 Here, the query comes from the news image, while keys and values come from retrieved celebrity patches (Shen et al., 16 Sep 2025).

The text branch appends retrieved celebrity metadata directly to the news text because the metadata is compact and informative. The enhanced text is then encoded by a text encoder $y_{\rm mul}=\{y_j\}_{j=1}^4$0: $y_{\rm mul}=\{y_j\}_{j=1}^4$1 The model also preserves access to the original text representation $y_{\rm mul}=\{y_j\}_{j=1}^4$2 alongside the celebrity-augmented representation $y_{\rm mul}=\{y_j\}_{j=1}^4$3 (Shen et al., 16 Sep 2025).

This CAP-aided encoding differs materially from the earlier HAMMER architecture. HAMMER uses two unimodal encoders—an image encoder $y_{\rm mul}=\{y_j\}_{j=1}^4$4 and a text encoder $y_{\rm mul}=\{y_j\}_{j=1}^4$5—plus a multi-modal aggregator $y_{\rm mul}=\{y_j\}_{j=1}^4$6, and it performs shallow manipulation reasoning through manipulation-aware contrastive learning followed by deep manipulation reasoning through modality-aware cross-attention. However, HAMMER’s evidence remains internal to the image-text pair and its momentum queues; it does not include external retrieval, evidence selection over a knowledge base, or retrieval-grounding supervision (Shao et al., 2023). RamDG’s augmentation therefore lies not in replacing multimodal fusion, but in supplying an external evidential branch before and during that fusion.

4. Joint reasoning modules and optimization objectives

RamDG contains two named modules: Celebrity-News Contrastive Learning (CNCL) and the Fine-grained Visual Refinement Mechanism (FVRM). CNCL is the retrieval-alignment component. It defines four contrastive objectives: $y_{\rm mul}=\{y_j\}_{j=1}^4$7, $y_{\rm mul}=\{y_j\}_{j=1}^4$8, $y_{\rm mul}=\{y_j\}_{j=1}^4$9, and $y_{\rm box}$0, which are summed into

$y_{\rm box}$1

A representative term is

$y_{\rm box}$2

where $y_{\rm box}$3 is cosine similarity and $y_{\rm box}$4 is a learnable temperature. The implementation uses a momentum encoder $y_{\rm box}$5, projection layers $y_{\rm box}$6, $y_{\rm box}$7, and a queue length of 65,536 (Shen et al., 16 Sep 2025).

FVRM addresses the fact that forgery regions are usually small and localized. Global multimodal fusion is first formed by attention over knowledge-enriched image and text features: $y_{\rm box}$8 A patch-level classification head over $y_{\rm box}$9 predicts whether each patch is manipulated, with loss

$y_{\rm tok}$0

The refined patch representation $y_{\rm tok}$1 is fused with global multimodal information through successive attention operations and residual connections, and the final representation is used for bounding-box regression (Shen et al., 16 Sep 2025).

The image grounding loss is written as

$y_{\rm tok}$2

For text manipulation localization, RamDG attends from text tokens to visual knowledge and then refines token-level representations with the global multimodal [CLS] token. The text localization loss is

$y_{\rm tok}$3

Global fake news recognition uses

$y_{\rm tok}$4

and manipulation type recognition uses

$y_{\rm tok}$5

The total objective is

$y_{\rm tok}$6

As presented, the jointly optimized components are contrastive alignment, binary fake detection, manipulation type recognition, patch-level visual localization, bounding-box regression, and token-level text localization (Shen et al., 16 Sep 2025).

A useful architectural comparison is with HAMMER’s hierarchy. HAMMER performs shallow manipulation reasoning via a manipulation-aware contrastive loss,

$y_{\rm tok}$7

and deep manipulation reasoning via modality-aware cross-attention, with dedicated heads for binary classification, multi-label classification, bounding-box detection, and token detection (Shao et al., 2023). RamDG retains the broad logic of joint detection and grounding, but substitutes celebrity-news retrieval and CAP-conditioned reasoning for HAMMER’s purely pair-internal contrastive setting.

5. Evaluation protocol, quantitative performance, and ablations

RamDG is evaluated on four tasks with the same metric families used in DGM$y_{\rm tok}$8: binary classification with AUC, ACC, and EER; manipulation type recognition with mAP, OF1, and CF1; image grounding with IoUmean, IoU50, and IoU75; and text grounding with Precision, Recall, and F1. The paper reports full-data results, low-data settings with 20,000 and 50,000 training samples, and generalization to unseen entities absent from CAP (Shen et al., 16 Sep 2025).

On the full SAMM test set, RamDG achieves 94.66 ACC, 98.79 AUC, 5.42 EER, 98.86 mAP, 80.90 IoUmean, and 84.83 F1. It is reported to outperform existing methods, including the stated 2.06\% higher detection accuracy on SAMM compared to state-of-the-art approaches. The comparison figures listed in the paper include HAMMER at ACC 92.43, IoUmean 77.68, and F1 84.31; HAMMER++ at ACC 92.26, IoUmean 77.66, and F1 84.35; FKA-Owl at ACC 92.60 but very poor manipulation localization and multi-label performance; and Qwen2.5VL-72b at binary accuracy 55.06 in zero-shot mode (Shen et al., 16 Sep 2025).

The low-resource results are emphasized as evidence for the retrieval component. With only 20,000 training samples, RamDG reaches 88.40 ACC, compared with 85.74 for HAMMER, 84.03 for HAMMER++, and 81.97 for ViLT. With 50,000 samples, RamDG reaches 91.07 ACC (Shen et al., 16 Sep 2025). This suggests that external knowledge is particularly useful when direct supervision is limited.

Generalization is also tested on entities absent from CAP. In that setting, RamDG reports ACC 94.1, mAP 97.3, IoUmean 78.7, and F1 83.9 (Shen et al., 16 Sep 2025). The stated interpretation is that the model does not merely memorize CAP entries; it learns a reasoning and localization scheme.

The ablations sharpen the functional roles of RamDG’s components. Removing CAP knowledge drops performance by about 3% on average. Among CAP fields, textual metadata is much more useful than visual celebrity images, and occupation contributes the most among metadata fields; visual knowledge helps less because images are often redundant for identity reasoning. Removing CNCL reduces performance across tasks, while removing FVRM hurts image grounding most strongly, with FVRM improving visual localization by about +2.89%. Efficiency figures reported for RamDG are 203.24M parameters, 53.0 GFLOPs, 12.3 ms latency, and 7357 MB GPU memory (Shen et al., 16 Sep 2025).

6. Relation to adjacent retrieval-grounding research and open distinctions

RamDG occupies a specific position at the intersection of multimodal forensics, grounding, and retrieval augmentation. It is narrower than generic retrieval-augmented generation verification and broader than pair-only deepfake detection. The most immediate predecessor is DGM$y_{\rm tok}$9, which defined the multi-task detection-and-grounding problem and introduced HAMMER, but did not address retrieval from external knowledge sources, evidence ranking, or grounding against retrieved documents (Shao et al., 2023).

A different neighboring line is grounding verification in retrieval-augmented generation, exemplified by "Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generation" (Leemann et al., 2024). That work studies an evidence input consisting of the user prompt plus retrieved documents and a generated claim or response produced by an LLM, and frames verification as binary NLI: positive or entailed or grounded when all information in the claim can be inferred directly from the evidence, and negative or non-entailed or ungrounded when the claim contains contradictory or unsupported information. Its Auto-GDA framework performs unsupervised domain adaptation through synthetic data generation, iterative augmentation, weak teacher scoring, and discrete sample selection, and the best DeBERTaV2 (Auto-GDA) model reaches average ROC-AUC 0.878 compared with 0.883 for GPT-4o, at about 10% of the computational cost (Leemann et al., 2024).

The relation between these directions is conceptual rather than task-identical. Auto-GDA verifies whether a generated textual response is supported by retrieved evidence; RamDG determines whether a multimodal news pair is manipulated and localizes the manipulated image region and text tokens. A plausible implication is that both lines treat retrieval as a mechanism for evidential grounding, but they operate at different granularities: claim-level entailment in Auto-GDA, versus joint fake recognition, manipulation-type recognition, image forgery grounding, and text manipulation localization in RamDG.

Several distinctions follow. First, RamDG should not be conflated with ordinary image-text consistency checking; its stated target is the harder case in which the caption is rewritten to remain plausible after the visual edit. Second, retrieval in RamDG is explicit and external through CAP, whereas in HAMMER the nearest analogue to retrieval is internal negative-pair selection and momentum queues rather than evidence retrieval over a knowledge repository (Shao et al., 2023). Third, the reported weakness of Qwen2.5VL-72b on zero-shot binary accuracy and the statement that VLLMs often fail at fine-grained localization indicate that external knowledge alone is insufficient without localization-specific design (Shen et al., 16 Sep 2025).

In the literature summarized here, RamDG’s importance lies in shifting multimodal manipulation detection from artificial misalignment detection to semantic-coordinated forgery detection. SAMM provides the benchmark for that setting, CAP supplies the external evidence source, CNCL enforces celebrity-news consistency reasoning, and FVRM preserves the fine-grained localization capability needed when manipulated regions and manipulated text spans are small and subtle (Shen et al., 16 Sep 2025).