DTE-FDM: Domain Tag-Guided Forgery Detection

• DTE-FDM is a module that combines domain-aware classification with visual token encoding and LLM capabilities to detect and localize image forgeries.
• It classifies images into forgery domains (Photoshop-style, DeepFake, AIGC-inpainting) and fuses domain tags with deep features for unified analysis.
• The system overcomes black-box limitations by providing evidence-backed, interpretable rationales that enhance forensic investigation and credibility.

A Domain Tag-guided Explainable Forgery Detection Module (DTE-FDM) is a central component in the FakeShield framework for explainable image forgery detection and localization. DTE-FDM addresses two critical limitations of existing image forgery detection and localization (IFDL) methods: the inherent black-box nature of detection models and the limited generalization across diverse manipulation methods, including Photoshop-style editing, DeepFake face manipulations, and AIGC-based inpainting. By integrating domain-aware classification with multimodal LLMs, DTE-FDM facilitates unified, interpretable detection and human-centric rationales for forensic analysis (Xu et al., 2024).

1. Functional Overview

DTE-FDM acts as the frontend of FakeShield, orchestrating three primary responsibilities: (1) classifying the input image into one of three forgery “domains” (Photoshop-style, DeepFake, or AIGC-inpainting), (2) fusing the assigned domain tag with image-derived features, and (3) providing both a binary authenticity verdict and a natural-language explanation that includes localized and evidence-based justification for the detection. This modular approach ensures that the system can not only flag manipulations but also rationalize its decisions for downstream analysts or end-users.

2. Architectural Components

DTE-FDM is composed of three tightly integrated submodules:

1. Domain Tag Generator (DTG): Implements a lightweight convolutional neural network classifier, denoted $\mathcal{G}_{\mathrm{dt}}(\cdot)$, which receives the RGB image $I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$ and outputs a discrete tag $$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$. The head consists of a small CNN with a terminal softmax over three classes.
2. Image Token Encoder & Projector: Utilizes a frozen visual encoder $\mathcal{F}_{\mathrm{enc}}$ (ViT- or CLIP-style backbone) to map $I_{\mathrm{ori}}$ to a sequence of patch tokens, followed by a linear projection layer $\mathcal{F}_{\mathrm{proj}}$ that adapts the visual tokens to the LLM embedding space, producing image tokens $T_{\mathrm{img}}\in\mathbb{R}^{L\times D}$ (where $L$ is token count and $D$ the embedding dimension).
3. LoRA-fine-tuned Multimodal LLM: A LLM, such as LLaVA, augmented via Low-Rank Adaptation (LoRA) to facilitate task-specific tuning. The LLM receives, sequentially, a textual prompt $T_{\mathrm{ins}}$, the domain tag $I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$0, and the image tokens $I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$1. The output $I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$2 comprises a real/fake decision, region localization, and an explicit rationale grounded in both pixel-level and semantic artifacts.

This pipeline enables structured data flow, as illustrated below:

Input	Module	Output
$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$3	$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$4	$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$5
$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$6	$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$7	$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$8
$I_{\mathrm{ori}}\in\mathbb{R}^{H\times W\times 3}$9	LLM	$$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$0

3. Mathematical Formalism

DTE-FDM’s formulation is characterized by explicit mappings at every stage:

• Domain Tag Assignment:

$$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$1

• Image Tokenization:

$$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$2

• Multimodal LLM Decoding:

$$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$3

where text tokens are concatenated, and cross-modal attention is adopted over image tokens.

4. Optimization Objectives

The training of DTE-FDM employs two primary loss terms:

1. Domain Tag Classification Loss:

$$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$4

which promotes correct assignment of domain tags by DTG.

2. Explainable Detection Loss:

$$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$5

where $$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$6 is the LLM’s output token sequence, $$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$7 is the ground truth explanation from the MMTD-Set, and $$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$8 is a balancing factor (typically $$T_{\mathrm{tag}}\in\{\text{"Photoshop"}, \text{"DeepFake"}, \text{"AIGC"}\}$$9). The aggregate loss forces robust classification of the domain and accurate, faithful explanation with both binary verdict and human-consumable justification.

A combined expression for the total loss is: $\mathcal{F}_{\mathrm{enc}}$0

5. Training Regime and Data Utilization

DTE-FDM is trained on the Multi-Modal Tamper Description Set (MMTD-Set), which aggregates:

• Photoshop-style tampering samples (Fantastic Reality, CASIAv2: copy-move, splicing, removal),
• DeepFake instances (FaceApp/FFHQ),
• AIGC-style inpainting (COCO + Stable Diffusion inpainting).

The dataset comprises approximately 50,000 tampered and 50,000 authentic images, with each image annotated by a triplet $\mathcal{F}_{\mathrm{enc}}$1, where $\mathcal{F}_{\mathrm{enc}}$2 (the explanation) is curated using GPT-4o prompt engineering.

Training specifics include:

• Adam optimizer ($\mathcal{F}_{\mathrm{enc}}$3, weight decay $\mathcal{F}_{\mathrm{enc}}$4), batch size 24, $\mathcal{F}_{\mathrm{enc}}$5 epochs.
• Compute: $\mathcal{F}_{\mathrm{enc}}$6 NVIDIA A100 $\mathcal{F}_{\mathrm{enc}}$7GB GPUs.
• Only the DTG and LoRA weights are trained; the LLM and encoder remain frozen.
• No explicit augmentations are used beyond the inherent domain diversity.

6. Inference Process and Qualitative Behavior

At inference, a novel image $\mathcal{F}_{\mathrm{enc}}$8 undergoes DTG-based domain tagging: $\mathcal{F}_{\mathrm{enc}}$9 The vision encoder and projector yield $I_{\mathrm{ori}}$0. These, together with an instruction prompt $I_{\mathrm{ori}}$1 and the domain tag, are provided to the LLM. The LLM then autoregressively generates $I_{\mathrm{ori}}$2—typically structured as follows:

• Detection: Real/Fake determination
• Location: Region(s) of tamper
• Basis: Evidence citing pixel-level or semantic artifacts (e.g., “edge jaggies,” “local blurring,” “lighting inconsistencies”)

Because the LLM is primed with the domain tag, it prioritizes relevant artifact classes (e.g., blur for DeepFake), enhancing the fidelity and specificity of its explanations compared to generic multimodal LLMs.

7. Representative Outputs and Interpretability

Qualitative analysis demonstrates DTE-FDM’s ability to generate explanatory outputs, as exemplified below:

• Photoshop-style Editing (CASIA1+, copy-move):
  • DTG: “This is a suspected Photoshop-style tampered picture.”
  • LLM: “Detection: Tampered. Location: The two yellow petals at the center appear superimposed on the red rose bush in the mid-frame. Basis: The pasted petals’ edges are unnaturally sharp and exhibit pixel-level jaggies; resolution and color tone differ slightly from the background.”
• DeepFake (FaceApp, glasses addition):
  • DTG: “This is a suspected DeepFake tampered picture.”
  • LLM: “Detection: Tampered. Location: The subject’s glasses region (around the eyes) shows local blurring and unnatural reflections. Basis: Facial skin texture is inconsistent around the frames; the specular highlight on the lenses violates expected lighting.”

In both cases, the system leverages the domain tag to produce explanations that are not only technically grounded but also aligned to the specific tamper class, enhancing interpretability.

---

By disambiguating domain-specific artifacts, tightly coupling image features with domain labels, and generating LoRA-guided rationales, DTE-FDM elevates the capabilities of forensic AI for Photoshop-style, DeepFake, and AIGC-inpainting manipulations, setting a benchmark for explainable image forgery detection in the era of multimodal LLMs (Xu et al., 2024).