---
title: AV-Deepfake1M 2025 Challenge
url: https://www.emergentmind.com/topics/av-deepfake1m-2025-challenge
type: topic
---

# AV-Deepfake1M 2025 Challenge

Searching arXiv for the most relevant papers on AV-Deepfake1M and the 2025 challenge.
The **AV-Deepfake1M 2025 Challenge**, also termed the **2025 1M-Deepfakes Detection Challenge**, is a community benchmark for **audio-visual deepfake detection and temporal localization** built on **AV-Deepfake1M++**, the 2025 extension of AV-Deepfake1M. It is designed to evaluate whether a clip is real or fake and, more importantly, to identify the manipulated temporal segment(s) within a video under realistic conditions that include partial forgeries, multimodal manipulations, and real-world perturbations [2507.20579]. The challenge inherits the core premise of AV-Deepfake1M—that deepfake analysis should move beyond whole-video binary classification toward fine-grained localization of short manipulated intervals—and extends it to a larger and more distribution-shifted benchmark intended to reflect how manipulated media appears in the wild [2311.15308].

## 1. Origins and benchmark rationale

The challenge emerged from the observation that many existing deepfake datasets and benchmarks assume that a clip is either fully real or fully fake, whereas real attackers may manipulate only **small localized segments** inside otherwise genuine audiovisual content [2409.06991]. This formulation is central to AV-Deepfake1M and remains central in the 2025 challenge, where manipulated portions are often **short partial fakes** embedded inside genuine videos, making both detection and localization substantially harder than conventional whole-utterance spoof settings [2508.07337].

AV-Deepfake1M was introduced as a **large-scale LLM-driven audio-visual deepfake dataset** to support this setting. Its motivating pipeline edits transcripts using structured operations—**replace**, **delete**, and **insert**—and then synthesizes corresponding audio and lip-synced video, thereby emulating content-driven deepfake creation rather than merely generating globally fake clips [2311.15308]. The original challenge paper subsequently formalized these ideas as the **1M-Deepfakes Detection Challenge**, described as a community benchmark and an ongoing benchmarking platform rather than a one-off competition [2409.06991].

The 2025 challenge is hosted on top of **AV-Deepfake1M++**, which explicitly extends the earlier benchmark in three directions: more real-source datasets, more generation methods, and more perturbations. The stated purpose is to provide a benchmark that is **harder, more diverse, and more realistic**, especially under cross-domain evaluation and real-world delivery artifacts such as compression, blur, jitter, and related degradations [2507.20579].

## 2. Dataset foundation: from AV-Deepfake1M to AV-Deepfake1M++

The original **AV-Deepfake1M** dataset contains **1,146,760 total videos**, including **286,721 real** and **860,039 fake**, across **2,068 unique subjects** and **1,886 hours** of audio-visual data [2311.15308]. The challenge paper summarizes the benchmark as containing **more than 1 million manipulated videos** across **more than 2,000 subjects**, with diverse background settings and high-realistic audio-visual content [2409.06991]. Its multimodal composition includes three manipulation categories: **Fake Audio and Fake Visual**, **Fake Audio and Real Visual**, and **Real Audio and Fake Visual** [2311.15308].

AV-Deepfake1M++ is the 2025 benchmark extension. It reports **2,051,154 video clips**, **502,844,910 frames**, **4,655.9 hours**, **7,109 subjects**, **627,936 real samples**, and **1,423,218 fake samples** [2507.20579]. Compared with AV-Deepfake1M, this roughly doubles dataset scale and substantially increases subject diversity [2507.20579]. The real clips are sourced from **VoxCeleb2**, **LRS3**, and **EngageNet**, addressing the earlier limitation that AV-Deepfake1M used only VoxCeleb2 [2507.20579].

The 2025 extension also enlarges the synthesis space. Its generation pipeline includes **nine state-of-the-art generators** spanning audio and visual manipulation. The visual/lip-sync pool includes **TalkLip**, **LatentSync**, and **Diff2Lip**; the audio/TTS pool includes **VITS**, **YourTTS**, **F5TTS**, and **XTTSv2** [2507.20579]. The manipulation process continues to follow an LLM-guided semantic edit plan over ASR transcripts, with token-level JSON output of the form
\[
\{operation, old\_word, new\_word, index\}
\]
where the operations are **replace**, **delete**, and **insert** [2507.20579].

The four modality combinations in AV-Deepfake1M++ are explicitly defined as:

| Combination | Audio | Visual |
|---|---|---|
| 1 | real | real |
| 2 | fake | real |
| 3 | real | fake |
| 4 | fake | fake |

A plausible implication is that the challenge is not reducible to either audio-only or vision-only forensics, because the benchmark includes unimodal and joint manipulations by construction.

## 3. Task definition and challenge protocol

The 2025 challenge includes two tasks on AV-Deepfake1M++ [2507.20579]:

1. **Video-level classification**: determine whether a clip is real or fake.
2. **Temporal localization**: identify the manipulated temporal segment(s) in a video.

The same two-task structure was already present in the earlier 1M-Deepfakes Detection Challenge, where participants had to determine whether an audio-visual sample of a single subject was **real or deepfake** and, separately, identify the **specific time intervals** that were manipulated [2409.06991]. The 2025 formulation preserves this structure while evaluating on **TestA** and **TestB**, with participants required to train **only on the official training split** [2507.20579].

The split structure of AV-Deepfake1M++ is as follows [2507.20579]:

| Split | #Videos | #Subjects |
|---|---:|---:|
| Training | 1,099,217 | 2,606 |
| Validation | 77,326 | shared with training |
| TestA | 828,318 | 4,503 |
| TestB | 46,293 | shared with TestA |
| Overall | 2,051,154 | 7,109 |

The paper emphasizes that training/validation are sample-level random splits, while the test sets are designed to be **domain-shifted**, with different identities, real sources, and generative methods across splits [2507.20579]. This suggests that leaderboard performance is intended to measure not only in-distribution accuracy but also robustness under cross-domain generalization.

The predecessor benchmark used a different split regime: **1,657 subjects** for train/validation and **411 subjects** for test, with **no overlap** between train and test subjects and split proportions of **65% train, 5% validation, 30% test** [2409.06991]. A notable detail there was that the test set used only **high-quality VITS-generated audio manipulations**, making evaluation especially stringent and consistent for audio-related synthesis artifacts [2409.06991].

## 4. Evaluation metrics and scoring

For **classification**, the challenge uses **AUC** (area under the ROC curve) [2409.06991; 2507.20579]. In the earlier challenge paper, the AUC range is given as **[0.5, 1]**, where **0.5** denotes random guessing and **1.0** denotes perfect prediction [2409.06991].

For **temporal localization**, the challenge uses a score based on **Average Precision (AP)** and **Average Recall (AR)** following prior temporal localization work [2409.06991]. The metric notation is preserved in the 2025 benchmark as:
- **AP@\{0.50,0.75,0.90,0.95\}**
- **AR@\{50,30,20,10,5\}** [2507.20579]

The earlier challenge paper specifies the thresholds and proposal counts in detail. For **AP**, the IoU thresholds are **0.5, 0.75, 0.9, 0.95**. For **AR**, the IoU thresholds are **0.5:0.95:0.05**, and the proposal counts are **50, 30, 20, 10, 5** [2409.06991]. The metrics are averaged across thresholds and proposal counts to produce overall AP and AR summary values, which are then used for the final result [2409.06991].

The final score is given as
\[
S = \frac{\sum_{IoU\in\{0.5,0.75,0.9,0.95\}AP@IoU}{8} + \frac{\sum_{N\in\{50,30,20,10,5\}AR@N}{10}
\]
with the clarification that the typesetting is malformed but the intended meaning is a combination of averaged AP and AR across the specified IoU thresholds and proposal counts [2409.06991]. The score lies in **[0,1]**, where **0** corresponds to random guessing and **1** to perfect predictions [2409.06991].

Operationally, the earlier challenge used **CodaBench** as the automated submission and scoring platform [2409.06991]. For the 2025 challenge, the dataset, challenge details, and evaluation scripts are available online under a **research-only license**, and the leaderboard is maintained on the challenge evaluation page [2507.20579].

## 5. Generation pipeline and realism assumptions

The benchmark’s realism derives from the underlying AV-Deepfake1M generation pipeline. A real video is processed through a three-stage procedure: transcript manipulation, audio generation, and video generation [2311.15308].

In the transcript stage, audio is extracted with **FFmpeg**, transcripts are generated using **Whisper**, and then modified by **ChatGPT** with a few-shot prompt [2311.15308]. The number of transcript edits is governed by
\[
M = \mathbf{ceil}(t/10)
\]
where \(M\) is the number of modifications and \(t\) is video duration in seconds [2311.15308]. This makes longer videos contain more edits.

In the audio stage, background noise and speech are separated using **Denoiser** in AV-Deepfake1M [2311.15308], whereas AV-Deepfake1M++ describes speech/background separation using **Demucs** [2507.20579]. Audio is synthesized with **VITS** for a subset of subjects and **YourTTS** for the remaining subjects in the original dataset [2311.15308], while the 2025 extension broadens the TTS pool to include **F5TTS** and **XTTSv2** as well [2507.20579]. The original paper explicitly uses two audio synthesis strategies for replacements and insertions: synthesizing the whole fake transcript and cropping the relevant part, or synthesizing only the new words directly [2311.15308].

In the video stage, AV-Deepfake1M evaluates several reenactment options—**EAMM**, **AVFR-GAN**, **DiffTalk**, **AD-NeRF**, and **ATVGnet**—and selects **TalkLip** because it is identity-independent, lip-sync focused, fast, state-of-the-art, and open-source [2311.15308]. AV-Deepfake1M++ subsequently expands the visual model pool to include **LatentSync** and **Diff2Lip** in addition to **TalkLip** [2507.20579].

A central realism mechanism in AV-Deepfake1M++ is the addition of real-world perturbations. Video perturbations include **Gaussian blur**, **salt-and-pepper noise**, **low bitrate / compression artifacts**, **Gaussian noise**, **Poisson noise**, **speckle noise**, **color quantization**, **random brightness**, **motion blur**, **rolling shutter**, **camera shake**, **lens distortion**, **vignetting**, **exposure variation**, and **chromatic aberration**. Audio perturbations include **compression artifacts**, **pitch/loudness change**, **white noise**, **time stretch**, **reverberation**, **ambient noise**, **clipping**, **frequency filtering**, **Doppler**, **interference**, **room impulse**, and **padding simulation** [2507.20579]. Held-out tests add harder perturbations such as **frame-rate jitter**, **pixelation distortion**, **localized defocus blur**, **frame dropouts**, **random spatial warping**, **random frame shuffle**, **audio stutter/repeat**, **audio stutter**, **audio frame shuffle**, and **padding simulation** [2507.20579].

This perturbation regime is explicitly motivated by the need to model online-video delivery artifacts that may hide forensic traces, create misleading artifacts, alter temporal coherence, and induce brittleness in detectors trained on clean synthetic data [2507.20579].

## 6. Methods, leaderboard results, and technical tendencies

The earlier challenge paper reports participation statistics from the 2024 edition: **191 teams** signed the EULA and registered, **1034 successful submissions** were made by the deadline, and Task 1 received **51% more submissions** than Task 2 [2409.06991]. The paper interprets this imbalance as evidence that the community still focuses more on video-level detection than on temporal localization, despite the importance of partial manipulations [2409.06991].

For the earlier detection task, the top three teams were **Fake-detector (USTC)**, **UQCV (UQ)**, and **FRIdatas (FRI)** [2409.06991]. Their reported methods indicate several technical tendencies:
- **Fake-detector (USTC)** introduced an **Audio-Visual Local-Global Interaction Module (AV-LG Module)** with **Local Intra-Region Self-Attention**, **Global Inter-Region Self-Attention**, and **Local-Global Interaction** [2409.06991].
- **UQCV (UQ)** used **weakly supervised pseudo-labeling** from **audio-based pseudo-labeling** and **mouth-region-based pseudo-labeling**, iteratively adding high-confidence samples for fine-tuning [2409.06991].
- **FRIdatas (FRI)** used a **vision transformer**, **Wav2Vec-XLS-R**, an embedding module, and **temporal convolution** to detect **fake chunks** in video [2409.06991].

For the earlier localization task, the top three teams were **Gradiant**, **Maya (USTC)**, and **nudt24 (NUDT)** [2409.06991]. Their methods included:
- **Gradiant**: **gMLP** for audio features plus **optical flow and RGB** processed by **UMMAFormer**, with output fusion [2409.06991].
- **Maya (USTC)**: **BYOL-A** for audio features, **TSN** and **InternVideo** for visual features; **BYOL-A + InternVideo** for detection and **BYOL-A + TSN** for localization [2409.06991].
- **nudt24 (NUDT)**: **THE-FD (Task Hierarchical Emotion-aware for Fake Detection)**, structured to process **video-level data** first and adapt to **frame-level data** through hierarchical feature inheritance and sharing [2409.06991].

The 2025 AV-Deepfake1M++ benchmark reports leaderboard-style results on **TestA**. For classification (AUC), the listed teams are **XJTU SunFlower Lab: 97.83**, **WHU_SPEECH: 93.07**, **KLASS: 92.78**, **Pindrop Labs: 92.49**, and **Mizhi Labs: 91.78**. The **Xception baseline** scores **55.09** on TestA and **57.29** on TestB [2507.20579]. For temporal localization on TestA, the reported results are **Pindrop Labs: 67.20**, **Mizhi Lab: 55.00**, **Purdue-M2: 50.87**, **WHU_SPEECH: 41.30**, **KLASS: 35.36**, **BA-TFD+ baseline: 14.71**, and **BA-TFD baseline: 13.54**; on TestB, **BA-TFD+ baseline** and **BA-TFD baseline** achieve **15.15** and **11.17**, respectively [2507.20579].

One reported interpretation is especially important: **temporal localization is dramatically harder** under the AV-Deepfake1M++ conditions, with methods that performed strongly on prior datasets collapsing sharply [2507.20579]. The authors highlight **BA-TFD+** as a concrete example, noting **96.30 AP@0.5** on **LAV-DF** but only **14.7 AP@0.5** on AV-Deepfake1M++ [2507.20579].

A specific 2025 challenge system, **KLASSify**, illustrates a different design choice. Rather than using a heavy end-to-end multimodal transformer, it **decouples audio and visual reasoning** [2508.07337]. The visual branch uses **MediaPipe Face Mesh** and eight handcrafted temporal artifact features, including **Blurriness of the Mouth ROI**, **Non-mouth Mean Squared Error (MSE)**, **Color Shift of the Mouth ROI**, and **Landmark Kinematics** such as **Mouth Aspect Ratio**, **Velocity**, **Acceleration**, **Jerk**, and **Jitter** [2508.07337]. These features are processed by a **lightweight 1D TCN** with **124K parameters** for classification and about **140K parameters** for localization; the localization head predicts frame-level tags under the **BILOU** scheme with labels **O, B, I, L** [2508.07337]. The audio branch uses **Wav2Vec-AASIST** for classification and an adapted **Boundary-aware Attention Mechanism (BAM)** with **WavLM-Large** for localization [2508.07337].

KLASSify reports **AUC 92.78%** for classification on **AV-Deepfake1M++ TestA** using multimodal fusion, and **IoU 0.3536** for localization using only the audio modality via **KLASSify-BAM** [2508.07337]. The paper further reports that simple score averaging of the best audio/video classifiers gives **91.97% AUC**, while calibration plus **Max-Out** improves this to **92.78%** [2508.07337]. This suggests that, within that system, multimodal fusion is most effective for classification, whereas audio is the effective modality for localization.

## 7. Significance, difficulty, and research implications

The challenge’s broader methodological message is that deepfake analysis should not be restricted to the question “is this whole video fake?” but should instead include **cross-modal analysis**, **temporal localization**, **partial-forgery scenarios**, **high-realism data**, and **standardized protocols** [2409.06991]. This principle originates directly from the AV-Deepfake1M benchmark, whose main claim is that existing methods experience a substantial performance collapse when confronted with short localized manipulations, multimodal edits, and very limited fake content per clip [2311.15308].

The original dataset paper provides several quantitative indicators of difficulty. On AV-Deepfake1M, temporal localization drops sharply relative to LAV-DF: **BA-TFD** falls from **79.15 → 37.37** AP@0.5, **BA-TFD+** from **96.30 → 44.42**, and **UMMAFormer** from **98.83 → 51.64** [2311.15308]. In detection, many methods remain near chance, with the best frame-level method reported as **Xception: 68.68 AUC**, while the paper explicitly notes that **even with frame-level labels, best AUC remains under 70** due to multimodal manipulations [2311.15308]. Human evaluation also found the dataset much harder than LAV-DF, with accuracy dropping from **84.03 → 68.64** and localization-oriented measures dropping sharply as well [2311.15308].

AV-Deepfake1M++ intensifies this difficulty by increasing source diversity, generation diversity, and perturbation diversity [2507.20579]. The benchmark’s stated conclusion is therefore not merely that the challenge is large, but that it is structured to stress **robustness to unseen manipulations and perturbations**, especially for localization [2507.20579]. This suggests that methods optimized narrowly for generator-specific artifacts or pristine synthesis traces may not transfer effectively.

A recurring misconception in deepfake benchmarking is that strong video-level classification performance implies robust forensic competence. The AV-Deepfake1M challenge family directly counters that assumption. Classification scores in the 90s can coexist with substantially weaker localization behavior, and localization baselines that were competitive on earlier datasets can degrade severely under the AV-Deepfake1M++ regime [2507.20579]. The challenge therefore distinguishes coarse clip-level discrimination from the finer problem of identifying manipulation boundaries in realistic audiovisual streams.

The practical ecosystem around the benchmark is also part of its significance. The associated repository provides **evaluation scripts**, **baseline models**, and **accompanying code**, and the evaluation server remains accessible beyond the challenge deadline to support continued benchmarking and future experimentation [2409.06991]. In this sense, the AV-Deepfake1M 2025 Challenge functions not only as a competition but as a continuing benchmark infrastructure for research on content-driven, temporally localized audio-visual deepfakes [2409.06991; 2507.20579].

Source: https://www.emergentmind.com/topics/av-deepfake1m-2025-challenge