---
title: 'AV-Deepfake1M: Deepfake Benchmark Dataset'
url: https://www.emergentmind.com/topics/av-deepfake1m-dataset-a93274ce-6374-4f23-aaa1-969b19a4dca8
type: topic
---

# AV-Deepfake1M: Deepfake Benchmark Dataset

AV-Deepfake1M is a large-scale benchmark dataset specifically designed for the development, training, and rigorous evaluation of audio-visual deepfake detection and temporal localization systems. The dataset provides synchronized video and audio recordings with granular, frame-level ground truth indicating whether, when, and via which modality (audio, video, or both) manipulations have been performed. Its construction encompasses state-of-the-art audio-text, face-generation, and voice-conversion pipelines and establishes balanced categorical splits for robust comparison under challenging multimodal forgeries and partial-segment attacks [2508.07337][2411.10193].

## 1. Dataset Composition, Classes, and Scale

AV-Deepfake1M consists of over 1 million audiovisual clips, each comprising a talking-head face video aligned with a speech audio track. The design emphasizes four manipulation conditions:

- **RVRA:** Real Video, Real Audio  
- **RVFA:** Real Video, Fake Audio (audio modified by voice conversion or TTS)
- **FVRA:** Fake Video, Real Audio (lip/mouth region re-synthesized to match authentic speech)
- **FVFA:** Fake Video, Fake Audio (simultaneous audio and video manipulation)

Balanced representation is maintained across classes, with training and validation splits containing approximately equal numbers per category. For instance, AV-Deepfake1M specifies  
  $N_{\text{train}} = 746\,180$ (≈186k per class), $N_{\text{val}} = 57\,340$ (≈14.3k per class) [2411.10193].  
AV-Deepfake1M++ and the AV-Deepfake1M 2025 Challenge employed over 2 million clips, further increasing coverage for all manipulations. The typical clip contains a single visible speaker with mean durations ranging from 4 s (AV-Deepfake1M) to ≈9.6 s (AV-Deepfake1M++), but clip lengths can vary from 1.32 s up to 668.48 s in training and 152.40 s in validation [2508.07337].

### Summary Statistics (AV-Deepfake1M++)

| Statistic           | Train               | Validation   |
|---------------------|---------------------|--------------|
| Total clips         | 1,099,217           | 77,326       |
| Real                | 297,389 (27.05%)    | 20,220 (26.15%)  |
| Audio Modified      | 266,116 (24.21%)    | 18,938 (24.49%)  |
| Visual Modified     | 269,901 (24.55%)    | 19,099 (24.70%)  |
| Both Modified       | 265,812 (24.18%)    | 19,069 (24.66%)  |
| Clip duration (s)   | 1.32–668.48 (mean 9.60) | 2.52–152.40 (mean 9.56) |
| Attacks per clip    | 0–5                 | 0–5          |
| Fake segment duration (s) | 0.02–11.58 (mean 0.33) | 0.02–8.10 (mean 0.33) |

This composition is a direct response to the inadequacies of unimodal and coarsely manipulated benchmarks, supporting subtle, short, and temporally local forgeries on a massive scale [2508.07337].

## 2. Manipulation Types and Data Generation Protocols

Audio and video manipulations are performed independently and jointly, leveraging state-of-the-art synthetic techniques:

- **Audio Attacks:**  
  - Advanced text-to-speech (TTS), including zero-shot models and VITS-based prosody cloning.
  - Simulated codec degradations (HTK, GSM, AMR-NB, SPH, Vorbis, OGG, MP3, WAV; multiple bit depths and encodings).
- **Video Attacks:**  
  - Audio-driven lip-sync and talking-head models (LatentSyncTaming, Diff2Lip-style, flow-based talking-head synthesis).
  - Re-synthesis is limited to the mouth region, while the remainder of the face remains unmodified.
- **Joint Attacks:**  
  - Simultaneous TTS-based audio spoofing and temporally correlated video lip-sync.

These manipulations are applied to scripted utterances spoken by real individuals, with manipulation intervals varying per sample. Empirically, manipulated segments in AV-Deepfake1M are, on average, half the length of those in prior LAV-DF benchmarks, with a mean of ≈1.3 fake segments per clip and segment durations averaging 0.33 s [2411.10193][2508.07337].

### Manipulation Pipeline (Pseudocode excerpt)
```
for each original clip v,a:
    if sample in FVFA or FVRA:
        apply deepfake_video_model(v) → v'
    if sample in FVFA or RVFA:
        apply voice_conversion(a) → a'
    store (v',a') with ground-truth intervals
```
[2411.10193]

## 3. Modalities, Feature Extraction, and Augmentation

### Video Modality

- **Input:** Face-centric video (25–30 fps)  
- **Landmark Extraction:** MediaPipe Face Mesh (468 points) to isolate mouth-ROI and non-mouth face.
- **Handcrafted Temporal Features:**
  1. Mouth-ROI blurriness (variance of Laplacian)
  2. Non-mouth MSE between consecutive frames
  3. Lab-space mouth-ROI color shift
  4–8. Lip landmark kinematics: aspect ratio, velocity, acceleration, jerk, and linear-fit jitter
- **Sequence Modeling:**  
  1D TCN with four residual blocks (dilations 1–16), attention pooling, and an MLP head (≈124k parameters for classification; ≈140k with a BILOU tagging head for localization) [2508.07337].

### Audio Modality

- **Input:** Raw speech waveform (16 kHz).  
- **Encoders:**  
  - Wav2Vec 2.0 XLSR-53 (classification, 1024-D output)
  - WavLM-Large SSL (temporal localization at 40 ms frame resolution)
- **Classifier/Head:**  
  - AASIST (graph-attention network for spectro-temporal cues)
  - Boundary-aware Attention Mechanism (BAM) for segment localization
- **Input Context:** 4 s window for both tasks, with wrapping/truncation for variable-length clips.

### Augmentation Protocols

- **Video:**
  - Features: channel-wise normalization, feature-shifting, temporal dropout (random frame drops), index swap, full-channel dropout, additive Gaussian noise.
- **Audio:**  
  - *audiomentations* and *torchaudio/SOX* backend, with room-impulse responses, time/stretch, pitch-shift, Gaussian noise, filtering, and codec simulation, each with $p=0.3$.

## 4. Annotation, Labeling, and Evaluation Protocols

### Labels and Supervision

- **Tasks:**
  1. **Deepfake Classification (DFD):** Clip-level real/fake labels (weak supervision)
  2. **Temporal Forgery Localization (TFL):** Frame-level binary labels and boundary-aware BILOU tags (B=Begin, I=Inside, L=Last, O=Outside of manipulated segment) [2508.07337].
  3. **Distance-to-boundary regression** targets for each frame and each modality [2411.10193].

- **Ground Truth Format:**
  - Binary modality labels ($a^m_j \in \{0,1\}$ per frame $j$ for $m \in \{v,a\}$).
  - Start/end intervals or per-frame distances to each boundary; no pixel-level masks.

### Evaluation Metrics

- **Classification:**  
  Area Under the ROC Curve (AUC):  
  $$ \mathrm{AUC} = \frac{1}{|\mathcal{P}|\;|\mathcal{N}|} \sum_{i\in \mathcal{P}}\sum_{j\in \mathcal{N}} \mathbf{1}[s_i > s_j] $$
  where $\mathcal{P}$ and $\mathcal{N}$ are the sets of fake and real samples, respectively.

- **Localization:**  
  - Average Precision (AP) at IoU thresholds $\{0.5, 0.75, 0.9, 0.95\}$  
  - Average Recall (AR) at $N \in \{50, 30, 20, 10, 5\}$ proposals  
  - Combined Score:  
    $$ \mathrm{Score} = \frac{1}{8}\sum_{\tau}\mathrm{AP}@\tau + \frac{1}{10}\sum_{N}\mathrm{AR}@N $$

### Access and Evaluation

- **Data Availability:**  
  - Train and validation splits: publicly downloadable from the authors’ repository (Cai et al.).
  - Test set: metadata not public; evaluation is managed via Codabench [2411.10193].
- **Licensing:** Non-commercial research use only.

## 5. Benchmarking, Baselines, and Key Results

Performance benchmarking on AV-Deepfake1M and AV-Deepfake1M++ tracks advances in both unimodal and multimodal detection settings. The dataset’s benchmark structure enforces balanced, identity-disjoint splits and supports both real-fake and partial-modality detection scenarios.

### Classification Task (AUC, %)

| Method                           | Val AUC | TestA AUC |
|-----------------------------------|---------|-----------|
| Video TCN (features 1,2)          | 84.29   | 69.31     |
| Video TCN (+color shift)          | 85.48   | 72.00     |
| Video TCN (+landmark kinematics)  | 88.41   | 73.11     |
| Wav2Vec-AASIST (ASV19 baseline)   | 73.48   | 60.53     |
| Wav2Vec-AASIST-codecs             | 99.71   | 82.91     |
| Audio + Video AVG fusion          | 97.86   | 91.97     |
| KLASSify (calibrated max-out)     | 98.04   | 92.78     |

### Temporal Localization (IoU and AP metrics, TestA)

| Method                   | IoU   | AP@0.5 | AP@0.75 | AP@0.9 | AP@0.95 | AR@50 |
|--------------------------|-------|--------|---------|--------|---------|-------|
| Video TCN (4 features)   |0.1139 |   –    |    –    |   –    |    –    |   –   |
| KLASSify-BAM (audio)     |0.3536 | 0.5117 | 0.4017  | 0.1701 | 0.0416  |0.4259 |

### SOTA Performance (AV-Deepfake1M, DiMoDif [2411.10193])

- **Classification AUC:** Up to 96.3% (DiMoDif, audio-visual, test set)
- **Temporal Localization:** AP@0.5 of 86.9%, AP@0.75 of 75.9% (DiMoDif)

Table 2 and Table 3 (above) reflect robust performance improvements between unimodal and multimodal approaches and the impact of feature and network design choices.

## 6. Impact, Limitations, and Research Directions

AV-Deepfake1M establishes a new evaluation paradigm prioritizing (1) balanced, multi-category deepfake coverage, (2) temporally local and subtle manipulation intervals, and (3) support for weak- and fully supervised detection/localization pipelines. Its scale, manipulation diversity, and complete temporal labeling enable the community to:

- Benchmark interpretable models leveraging handcrafted features and lightweight TCNs [2508.07337]
- Develop advanced multimodal fusion and cross-modal discrepancy analysis—e.g., DiMoDif’s hierarchical cross-modal fusion with adaptive alignment and discrepancy mapping [2411.10193]
- Explore resilience to codec-induced degradations and real-world preprocessing noise
- Advance fine-grained temporal localization of deepfake forgeries, a critical capability for content-tracing and forensic applications

Certain implementation details—including exact per-sample frame rates, resolutions, and underlying source datasets—are not specified in all descriptions and may vary between AV-Deepfake1M and AV-Deepfake1M++. Access to test labels is restricted, requiring server-side evaluation. A plausible implication is that reproducibility is very high for model comparisons but that custom downstream uses may require adaptation or reference to the original license [2411.10193].

The dataset is widely adopted for recent state-of-the-art multimodal detection challenges, shaping the direction of practical research in audio-visual deepfake forensics.

Source: https://www.emergentmind.com/topics/av-deepfake1m-dataset-a93274ce-6374-4f23-aaa1-969b19a4dca8