---
title: Retrieval-Augmented Manipulation Detection (RamDG)
url: https://www.emergentmind.com/topics/retrieval-augmented-manipulation-detection-and-grounding-ramdg
type: topic
---

# Retrieval-Augmented Manipulation Detection (RamDG)

Retrieval-Augmented Manipulation Detection and Grounding (RamDG) is a multimodal media-forensics framework introduced in "Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations" [2509.12653]. It targets manipulated image-text news pairs in which visual edits are systematically paired with semantically consistent textual descriptions, rather than obvious image-text mismatches. In this setting, the image may be altered by face swapping or facial attribute editing, the text is also forged so it stays semantically plausible with the changed image, and the manipulations are small, localized, and coordinated across modalities. RamDG retrieves external celebrity knowledge from the Celeb Attributes Portfolio (CAP), encodes that evidence together with the input pair, and jointly performs fake news binary recognition, manipulation type recognition, image forgery grounding, and text manipulation localization [2509.12653].

## 1. Problem formulation and conceptual motivation

RamDG is motivated by a critique of prevailing multimodal manipulation benchmarks. The central argument is that many existing benchmarks create fake samples by artificially breaking semantic alignment between image and text: a relevant image is paired with an unrelated caption, a name is randomly replaced, or words are swapped in a way that makes the text obviously inconsistent with the image. Such setups enable shallow cross-modal mismatch detection. RamDG instead addresses semantically coordinated manipulations, where both modalities remain mutually plausible while the combined news item is false [2509.12653].

This problem setting is closely related to the earlier formalization of Detecting and Grounding Multi-Modal Media Manipulation (DGM$^4$), which defined the input as an image-text pair \((I,T)\sim P\) and annotated each pair with a binary authenticity label \(y_{\rm bin}\), a fine-grained manipulation-type vector \(y_{\rm mul}=\{y_j\}_{j=1}^4\), manipulated image bounding boxes \(y_{\rm box}\), and manipulated text-token labels \(y_{\rm tok}\). In DGM$^4$, the fine-grained manipulation classes are Face Swap (FS), Face Attribute (FA), Text Swap (TS), and Text Attribute (TA), and the task decomposes into binary classification, multi-label classification, image grounding, and text grounding [2304.02556].

RamDG preserves that multi-task perspective but changes the evidential setting. DGM$^4$ and HAMMER reason directly over the input pair without retrieval over external corpora, whereas RamDG explicitly harnesses external knowledge repositories to retrieve contextual evidence and uses that evidence in both detection and grounding [2304.02556]. This shift matters because semantically coordinated attacks suppress the very inconsistency signals on which many earlier systems rely.

## 2. Data resources: from DGM$^4$ to SAMM and CAP

The data lineage around RamDG is anchored by two complementary resources. DGM$^4$ introduced the first large-scale dataset for detecting and grounding multi-modal media manipulation, built from VisualNews and focused on human-centric news. It totals **230k** samples: **77,426 pristine** and **152,574 manipulated**. The manipulated set includes **66,722 FS**, **56,411 FA**, **43,546 TS**, and **18,588 TA** pairs, plus **32,693 mixed-manipulation pairs** where image and text manipulations are combined. The construction process also added random perturbations such as JPEG compression and blur to 50% of the pool to simulate noise [2304.02556].

RamDG is introduced together with a new benchmark, SAMM, short for Semantic-Aligned Multimodal Manipulation. SAMM contains **260,970 samples** and is designed to reflect semantically coordinated fake news. It is built from **VisualNews** and **GoodNews**, with filtering to ensure that each news item contains at least one person and that highly redundant images are removed using **CLIP-based similarity** filtering. Its base set is denoted
\[
S = \{P_{s} \mid P_{s} = (I_{s}, T_{s})\},
\]
where \(I_s\) is the source image and \(T_s\) is the source text. The final composition is **111K original news**, **80K swap-manipulated**, and **69K attribute-manipulated** samples [2509.12653].

A second resource, CAP, is external rather than benchmark-internal. CAP, the Celeb Attributes Portfolio, contains multimodal celebrity knowledge gathered from the web via Google Search API, including celebrity images, gender, birth year, occupation, and main achievements. CAP serves two roles simultaneously: it helps drive SAMM construction and it functions as the retrieval source for RamDG [2509.12653].

SAMM’s manipulation pipeline is explicitly two-stage: first manipulate the image, then generate a semantically plausible text. In the swap case, a celebrity face \(I_f\) and corresponding name \(N_f\) are randomly selected from CAP; **SimSwap** or **InfoSwap** generates the manipulated image \(I_m\), and the swapped face bounding box \((x_1,y_1,x_2,y_2)\) is recorded. **Qwen2.5** extracts all names from the original text, **Qwen2-VL** matches the face region to the correct original identity \(N^t\), and all occurrences of \(N^t\) are replaced with \(N_f\), producing \(T_m\) with a word-level tampering label. In the attribute case, **DSFD** detects a face and records its bounding box, **Qwen2-VL** predicts the emotional state, and **HFGI** or **StyleCLIP** produces a manipulated face \(F_m\); emotion-related language is then inserted into the text using an emotion expression set
\[
\mathcal{E}=\{e_i|e_i=[w^i_1,w^i_2,...]\}_{i=1}^{|\mathcal{E}|},
\]
again with word-level tampering labels [2509.12653].

The conceptual significance of SAMM is that it replaces purposeless tampering and artificial cross-modal inconsistency with purposeful tampering and aligned manipulation across image and text. This suggests a stricter evaluation regime for multimodal forensics, because success can no longer be attributed primarily to shallow alignment cues.

## 3. Retrieval augmentation and CAP-aided context-aware encoding

RamDG’s retrieval stage begins from an input image-text pair
\[
P = (I, T).
\]
Person names are extracted from the text, and **string matching** is used to retrieve associated celebrity entries from CAP:
\[
\{P_j = (I_j, T_j) \mid j = 1, 2, \dots, P_j \in \mathit{CAP}\}.
\]
These retrieved entries act as auxiliary knowledge [2509.12653].

The image branch uses a ViT-based image encoder \(E_v\). The news image and the retrieved celebrity images are patchified and encoded as
\[
E_v(I) = \mathit{V} = \{\mathit{V}^{cls}, \mathit{V}^{pat}\},
\]
\[
E_v(I_j) = \mathit{V}_j = \{\mathit{V}_j^{cls}, \mathit{V}_j^{pat}\}.
\]
All retrieved celebrity patch features are concatenated into \(\mathit{V}_{cb}\), and cross-attention injects celebrity knowledge into the image representation:
\[
\mathit{V}_{f} = \text{Attn}(Q = \mathit{V}, K = \mathit{V}_{cb}, V = \mathit{V}_{cb}|\Theta_1).
\]
Here, the query comes from the news image, while keys and values come from retrieved celebrity patches [2509.12653].

The text branch appends retrieved celebrity metadata directly to the news text because the metadata is compact and informative. The enhanced text is then encoded by a text encoder \(E_t\):
\[
\mathit{L}_{f} = E_t(T_{all}) = \{\mathit{L}_{f}^{cls}, \mathit{L}_{f}^{tok}\}.
\]
The model also preserves access to the original text representation \(L\) alongside the celebrity-augmented representation \(L_f\) [2509.12653].

This CAP-aided encoding differs materially from the earlier HAMMER architecture. HAMMER uses two unimodal encoders—an image encoder \(E_v\) and a text encoder \(E_t\)—plus a multi-modal aggregator \(F\), and it performs shallow manipulation reasoning through manipulation-aware contrastive learning followed by deep manipulation reasoning through modality-aware cross-attention. However, HAMMER’s evidence remains internal to the image-text pair and its momentum queues; it does not include external retrieval, evidence selection over a knowledge base, or retrieval-grounding supervision [2304.02556]. RamDG’s augmentation therefore lies not in replacing multimodal fusion, but in supplying an external evidential branch before and during that fusion.

## 4. Joint reasoning modules and optimization objectives

RamDG contains two named modules: **Celebrity-News Contrastive Learning (CNCL)** and the **Fine-grained Visual Refinement Mechanism (FVRM)**. CNCL is the retrieval-alignment component. It defines four contrastive objectives: \( \mathcal{L}_{v2v} \), \( \mathcal{L}_{v2t} \), \( \mathcal{L}_{t2v} \), and \( \mathcal{L}_{t2t} \), which are summed into
\[
\mathcal{L}_{cncl} = \mathcal{L}_{v2v} + \mathcal{L}_{v2t} + \mathcal{L}_{t2v} + \mathcal{L}_{t2t}.
\]
A representative term is
\[
\mathcal{L}_{v2v}(I_{j}, I, \mathcal{I}) = -  
\log \bigg( \frac{\exp(s(I_{j}, I^{+})/\tau)}{\sum_{I_{k} \in \mathcal{I} \exp(s(I_{j}, I_{k})/\tau)} \bigg),
\]
where \(s(\cdot)\) is cosine similarity and \(\tau\) is a learnable temperature. The implementation uses a momentum encoder \(\hat{E}_v\), projection layers \(P_v\), \(\hat{P}_v\), and a queue length of **65,536** [2509.12653].

FVRM addresses the fact that forgery regions are usually small and localized. Global multimodal fusion is first formed by attention over knowledge-enriched image and text features:
\[
M_{\mathit{f}} = \text{Attn}(Q = \mathit{V}_{f}, K = \mathit{L}_{f},V = \mathit{L}_{f}|\Theta_2).
\]
A patch-level classification head over \(\mathit{V}^{pat}\) predicts whether each patch is manipulated, with loss
\[
\mathcal{L}_{pat} = -\sum_{i=1}^{C} \left[ y_{pat} \log(P_{pat}) \right].
\]
The refined patch representation \(\widetilde{V}^{pat}\) is fused with global multimodal information through successive attention operations and residual connections, and the final representation is used for bounding-box regression [2509.12653].

The image grounding loss is written as
\[
\mathcal{L}_\mathit{bbox} = -||P_{gro}-\text{Sigmoid}(y_{box})||_1 \ + \mathcal{L}_\mathit{IoU}(P_{gro}, \text{Sigmoid}(\mathit{y}_{box})).
\]
For text manipulation localization, RamDG attends from text tokens to visual knowledge and then refines token-level representations with the global multimodal [CLS] token. The text localization loss is
\[
\mathcal{L}_\mathit{tok} = -\sum_{i} \left[ y_{tok}^i \log(P_{tok}^i) \right].
\]
Global fake news recognition uses
\[
\mathcal{L}_{2cls} =  -y_{2cls}\log(P_{2cls}),
\]
and manipulation type recognition uses
\[
\mathcal{L}_{\mathit{mcls}} = -\sum_{i=1}^4 \left[ y_{mcls}^i \log(P_{mcls}^i) \right].
\]
The total objective is
\[
\mathcal{L} = \mathcal{L}_{\mathit{cncl}} +  \mathcal{L}_{2\mathit{cls}} + \mathcal{L}_{\mathit{mcls}} + \mathcal{L}_{\mathit{pat}} + \mathcal{L}_{\mathit{bbox}} + \mathcal{L}_{\mathit{tok}}.
\]
As presented, the jointly optimized components are contrastive alignment, binary fake detection, manipulation type recognition, patch-level visual localization, bounding-box regression, and token-level text localization [2509.12653].

A useful architectural comparison is with HAMMER’s hierarchy. HAMMER performs shallow manipulation reasoning via a manipulation-aware contrastive loss,
\[
\mathcal{L}_{MAC} = \frac{1}{4} [\mathcal{L}_{v2t} + \mathcal{L}_{t2v} + \mathcal{L}_{v2v}+\mathcal{L}_{t2t}],
\]
and deep manipulation reasoning via modality-aware cross-attention, with dedicated heads for binary classification, multi-label classification, bounding-box detection, and token detection [2304.02556]. RamDG retains the broad logic of joint detection and grounding, but substitutes celebrity-news retrieval and CAP-conditioned reasoning for HAMMER’s purely pair-internal contrastive setting.

## 5. Evaluation protocol, quantitative performance, and ablations

RamDG is evaluated on four tasks with the same metric families used in DGM$^4$: binary classification with **AUC**, **ACC**, and **EER**; manipulation type recognition with **mAP**, **OF1**, and **CF1**; image grounding with **IoUmean**, **IoU50**, and **IoU75**; and text grounding with **Precision**, **Recall**, and **F1**. The paper reports full-data results, low-data settings with **20,000** and **50,000** training samples, and generalization to unseen entities absent from CAP [2509.12653].

On the full SAMM test set, RamDG achieves **94.66 ACC**, **98.79 AUC**, **5.42 EER**, **98.86 mAP**, **80.90 IoUmean**, and **84.83 F1**. It is reported to outperform existing methods, including the stated **2.06\% higher detection accuracy** on SAMM compared to state-of-the-art approaches. The comparison figures listed in the paper include **HAMMER** at ACC **92.43**, IoUmean **77.68**, and F1 **84.31**; **HAMMER++** at ACC **92.26**, IoUmean **77.66**, and F1 **84.35**; **FKA-Owl** at ACC **92.60** but very poor manipulation localization and multi-label performance; and **Qwen2.5VL-72b** at binary accuracy **55.06** in zero-shot mode [2509.12653].

The low-resource results are emphasized as evidence for the retrieval component. With only **20,000 training samples**, RamDG reaches **88.40 ACC**, compared with **85.74** for HAMMER, **84.03** for HAMMER++, and **81.97** for ViLT. With **50,000 samples**, RamDG reaches **91.07 ACC** [2509.12653]. This suggests that external knowledge is particularly useful when direct supervision is limited.

Generalization is also tested on entities absent from CAP. In that setting, RamDG reports **ACC 94.1**, **mAP 97.3**, **IoUmean 78.7**, and **F1 83.9** [2509.12653]. The stated interpretation is that the model does not merely memorize CAP entries; it learns a reasoning and localization scheme.

The ablations sharpen the functional roles of RamDG’s components. Removing CAP knowledge drops performance by about **3% on average**. Among CAP fields, **textual metadata is much more useful than visual celebrity images**, and **occupation** contributes the most among metadata fields; visual knowledge helps less because images are often redundant for identity reasoning. Removing **CNCL** reduces performance across tasks, while removing **FVRM** hurts image grounding most strongly, with FVRM improving visual localization by about **+2.89%**. Efficiency figures reported for RamDG are **203.24M parameters**, **53.0 GFLOPs**, **12.3 ms latency**, and **7357 MB GPU memory** [2509.12653].

## 6. Relation to adjacent retrieval-grounding research and open distinctions

RamDG occupies a specific position at the intersection of multimodal forensics, grounding, and retrieval augmentation. It is narrower than generic retrieval-augmented generation verification and broader than pair-only deepfake detection. The most immediate predecessor is DGM$^4$, which defined the multi-task detection-and-grounding problem and introduced HAMMER, but did not address retrieval from external knowledge sources, evidence ranking, or grounding against retrieved documents [2304.02556].

A different neighboring line is grounding verification in retrieval-augmented generation, exemplified by "Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generation" [2410.03461]. That work studies an evidence input consisting of the user prompt plus retrieved documents and a generated claim or response produced by an LLM, and frames verification as binary NLI: positive or entailed or grounded when all information in the claim can be inferred directly from the evidence, and negative or non-entailed or ungrounded when the claim contains contradictory or unsupported information. Its Auto-GDA framework performs unsupervised domain adaptation through synthetic data generation, iterative augmentation, weak teacher scoring, and discrete sample selection, and the best **DeBERTaV2 (Auto-GDA)** model reaches average ROC-AUC **0.878** compared with **0.883** for **GPT-4o**, at about **10% of the computational cost** [2410.03461].

The relation between these directions is conceptual rather than task-identical. Auto-GDA verifies whether a generated textual response is supported by retrieved evidence; RamDG determines whether a multimodal news pair is manipulated and localizes the manipulated image region and text tokens. A plausible implication is that both lines treat retrieval as a mechanism for evidential grounding, but they operate at different granularities: claim-level entailment in Auto-GDA, versus joint fake recognition, manipulation-type recognition, image forgery grounding, and text manipulation localization in RamDG.

Several distinctions follow. First, RamDG should not be conflated with ordinary image-text consistency checking; its stated target is the harder case in which the caption is rewritten to remain plausible after the visual edit. Second, retrieval in RamDG is explicit and external through CAP, whereas in HAMMER the nearest analogue to retrieval is internal negative-pair selection and momentum queues rather than evidence retrieval over a knowledge repository [2304.02556]. Third, the reported weakness of **Qwen2.5VL-72b** on zero-shot binary accuracy and the statement that VLLMs often fail at fine-grained localization indicate that external knowledge alone is insufficient without localization-specific design [2509.12653].

In the literature summarized here, RamDG’s importance lies in shifting multimodal manipulation detection from artificial misalignment detection to semantic-coordinated forgery detection. SAMM provides the benchmark for that setting, CAP supplies the external evidence source, CNCL enforces celebrity-news consistency reasoning, and FVRM preserves the fine-grained localization capability needed when manipulated regions and manipulated text spans are small and subtle [2509.12653].

Source: https://www.emergentmind.com/topics/retrieval-augmented-manipulation-detection-and-grounding-ramdg