MMTD-Set: Multi-Modal Tamper Dataset

• MMTD-Set is a large-scale dataset that unifies multi-modal image tamper detection using pixel-level masks and natural language annotations.
• It integrates forensic evidence from Photoshop-style, DeepFake, and AIGC-Editing, supporting detailed localization and cross-domain analysis.
• The dataset facilitates robust model training with modern techniques such as LoRA and SAM, and provides comprehensive evaluation metrics for detection and explanation.

The Multi-Modal Tamper Description dataSet (MMTD-Set) is a large-scale, multi-modal benchmark designed to facilitate explainable image forgery detection and localization. Developed in the context of multi-modal LLM–based tamper analysis and utilized in the FakeShield framework, MMTD-Set introduces granular pixel-level localization, cross-domain sampling, and natural language explanation for multiple tamper types, including traditional image compositing (e.g., Photoshop), DeepFake, and AI-generated content (AIGC-Editing). The dataset provides a unified protocol for jointly training and evaluating models to detect, localize, and interpret digital image forgeries (Xu et al., 2024).

1. Dataset Construction and Scope

MMTD-Set is composed of both authentic and manipulated imagery sourced from established tampering datasets and augmented with generative techniques. The dataset combines three major tampering paradigms:

• Photoshop-style: Includes copy-move, splicing, and removal manipulations from CASIAv2, Fantastic Reality, IMD2020, Columbia, COVERAGE, DSO, and Korus.
• DeepFake: Utilizes FaceAPP-generated facial composites (from DFFD) with FFHQ as the pristine reference set.
• AIGC-Editing: Leverages 20,000 COCO real images and generates 20,000 manipulated versions using Stable-Diffusion inpainting.

For each manipulated image, MMTD-Set provides the tampered image, a binary tampering mask, and a text-based triplet annotation via GPT-4o using tamper-type–specific prompts. Authentic (non-tampered) images are provided with a corresponding authenticity confirmation prompt and annotation. Domain tags are assigned through a CNN-based Domain Tag Generator, classifying each image into {Photoshop-style, DeepFake, AIGC-Editing} and prepending an explicit tag to the LLM input for cross-domain disambiguation.

2. Modalities and Annotation Schema

MMTD-Set encapsulates three primary annotation channels per image:

• Raw Images: Stored as JPEG or PNG at original dataset resolution.
• Tampering Masks: PNG-encoded binary masks (0/255) denoting manipulated pixels per image.
• Natural Language Descriptions: JSON-formatted triplets with the following fields:
  • "label": "tampered" or "authentic"
  • "domain_tag": Domain identifier (e.g., "Photoshop-style")
  • "localization": Region-specific textual localization (absolute and/or relative, e.g. “upper left corner” or “over the book cover”)
  • "basis": Sentence(s) identifying key anomalies—e.g., edge artifacts, lighting inconsistency, semantic errors.

Text descriptions for tampered images are 3–5 sentences, systematically detailing detection verdict, precise region(s), and artifact rationale using specialized tamper- and domain-aware prompts. No fixed schema is imposed beyond the core triplet keys, ensuring scalability and adaptability for new tamper types.

3. Dataset Statistics and Splits

The dataset is stratified into extensive training and testing partitions, with cross-domain coverage and rigorously balanced authentic and tampered sets.

Partition	Source(s) / Domains	Real Images	Tampered Images	Tamper Subtypes – Fake
Training	Photoshop, DeepFake, AIGC	≈ 53,000	≈ 52,000	Copy-move, Splicing, Removal, DeepFake, Stable-Diffusion
Testing	Benchmarks per domain	Varies	Varies	Benchmarks: CASIAv1+, Columbia, COVERAGE, NIST16, IMD2020, DSO, Korus, FFHQ, FaceAPP, COCO, AIGC

Notable statistics:

• Photoshop-style: e.g., CASIAv2 (7,491 real, 5,123 fake), Fantastic Reality (16,592 real, 19,423 fake)
• DeepFake: FFHQ (10,000 real), FaceAPP (7,308 fake)
• AIGC-Editing: COCO (20,000 real), AIGC (20,000 fake)

Cross-domain partitions and domain tags enforce broad generalization and facilitate robust per-domain evaluation protocols.

4. Tampering Description Protocol and Examples

Annotations use rigorous, LLM-assisted instructions to surface interpretable and tamper-type–aware rationales.

A typical annotation comprises:

1. Detection verdict (“the image appears tampered” / “authentic”)
2. Localization (“a small patch in the upper-right corner over the book cover” / “center-left region”)
3. Basis of judgment (“edges too sharp”, “lighting inconsistent”, “unnatural resolution”, “symbol glitch”, “semantic error”)

Example (Photoshop splicing, CASIAv1+):

{ "label": "tampered", "domain_tag": "Photoshop-style", "localization": "Center-left region where the penguin overlaps the desk surface", "basis": "The pasted region has a distinct resolution and unnatural shadow direction; edges are sharp." }

Example (Authentic, IMD2020):

{ "label": "authentic", "domain_tag": "Photoshop-style", "localization": "N/A", "basis": "No inconsistent edges or resolution changes detected; overall lighting and perspective are coherent." }

Prompts are refined per tamper domain to elicit artifact details relevant to each manipulation technology.

5. Quality Control and Evaluation Metrics

Annotation quality and task performance are evaluated through a combination of model-level and text-based semantic metrics:

• Detection metrics: Image-level Accuracy (ACC), F1 score (default threshold = 0.5).
• Localization metrics: Intersection-over-Union (IoU), mask F1, measuring congruency between predicted and ground-truth masks.
• Textual explanation quality: Cosine Semantic Similarity (CSS) between GPT-4o ground-truth annotations and model outputs.
• Training losses (for LoRA fine-tuning):
  • Detection: $$\ell_{\mathrm{det}} = \ell_{\mathrm{ce}}(\hat{O}_{\mathrm{det}}, O_{\mathrm{txt}}) + \lambda \cdot \ell_{\mathrm{ce}}(\hat{T}_{\mathrm{tag}}, T_{\mathrm{tag}})$$
  • Localization: $$\ell_{\mathrm{loc}} = \ell_{\mathrm{ce}}(\hat{y}_{\mathrm{txt}}, y_{\mathrm{txt}}) + \alpha \cdot \ell_{\mathrm{bce}}(\hat{W}_{\mathrm{loc}}, W_{\mathrm{loc}}) + \beta \cdot \ell_{\mathrm{dice}}(\hat{W}_{\mathrm{loc}}, W_{\mathrm{loc}})$$

Quality assurance employs spot-checks of GPT-4o outputs by designed prompts. There are no explicit human-agreement statistics, and a plausible implication is that the primary reliance is on LLM consistency and prompt design.

FakeShield baseline results on MMTD-Set show ACC/F1 scores in the range 0.83–0.99 for detection and IoU/F1 values up to 0.67/0.75 for localization, with CSS scores up to 0.88 for explanations (Xu et al., 2024).

6. Usage Protocols and Model Training

Training on MMTD-Set adheres to the following reference protocol:

1. Detection: Fine-tune a multi-modal LLM (e.g., LLaVA-v1.5-13B) using LoRA (rank=128, α=256) for 10 epochs (batch 24, lr=$2 \times 10^{-4}$, 4×A100 GPUs).
2. Domain Tagging: Domain Tag Generator is trained concurrently.
3. Localization: Fine-tune a Tamper Comprehension Module with Segment-Anything Model (SAM) using LoRA (rank=8, α=16) for 24 epochs (batch 48, lr=$3 \times 10^{-4}$).

Evaluation is domain-discriminative:

• Detection (ACC/F1) and localization (IoU/F1) reported per benchmark split.
• CSS used for explanation quality benchmarking.

7. Comparison to Related Datasets and Unique Features

MMTD-Set extends and unifies prior forensics corpora by:

Dataset	Scale	Domains	Mask	Explanation (Text)	Sources
MMTD-Set	>100K	Multi-domain	Pixel-level	Yes (Triplet)	Photoshop, DeepFake, AIGC
DocTamper (CVPR 2023)	14 K	Documents	Pixel-level	No	N/A
Tampered IC-13 (ECCV)	~300	Scene text	Box	No	N/A
OSFT (arXiv 2024)	3 K	Generative	N/A	No	N/A

MMTD-Set’s distinctive contributions are its multi-domain scope, integration of pixel-level masks with natural language rationales, and broad coverage of both classical and generative tampering modalities. Limitations include the absence of explicit human agreement statistics and domain specialization determined by prompt and mask input design. The dataset and baseline code are hosted at https://github.com/zhipeixu/FakeShield and are referenced in (Xu et al., 2024).