AV-Deepfake1M 2025 Challenge
- AV-Deepfake1M 2025 Challenge is a benchmark for detecting partial audio-visual deepfakes and localizing manipulated segments.
- It expands on the original dataset with increased scale, diverse generation methods, and realistic perturbations to simulate real-world conditions.
- The challenge evaluates both video-level classification and fine-grained temporal localization using metrics like AUC and Average Precision.
Searching arXiv for the most relevant papers on AV-Deepfake1M and the 2025 challenge. The AV-Deepfake1M 2025 Challenge, also termed the 2025 1M-Deepfakes Detection Challenge, is a community benchmark for audio-visual deepfake detection and temporal localization built on AV-Deepfake1M++, the 2025 extension of AV-Deepfake1M. It is designed to evaluate whether a clip is real or fake and, more importantly, to identify the manipulated temporal segment(s) within a video under realistic conditions that include partial forgeries, multimodal manipulations, and real-world perturbations (Cai et al., 28 Jul 2025). The challenge inherits the core premise of AV-Deepfake1M—that deepfake analysis should move beyond whole-video binary classification toward fine-grained localization of short manipulated intervals—and extends it to a larger and more distribution-shifted benchmark intended to reflect how manipulated media appears in the wild (Cai et al., 2023).
1. Origins and benchmark rationale
The challenge emerged from the observation that many existing deepfake datasets and benchmarks assume that a clip is either fully real or fully fake, whereas real attackers may manipulate only small localized segments inside otherwise genuine audiovisual content (Cai et al., 2024). This formulation is central to AV-Deepfake1M and remains central in the 2025 challenge, where manipulated portions are often short partial fakes embedded inside genuine videos, making both detection and localization substantially harder than conventional whole-utterance spoof settings (Kukanov et al., 10 Aug 2025).
AV-Deepfake1M was introduced as a large-scale LLM-driven audio-visual deepfake dataset to support this setting. Its motivating pipeline edits transcripts using structured operations—replace, delete, and insert—and then synthesizes corresponding audio and lip-synced video, thereby emulating content-driven deepfake creation rather than merely generating globally fake clips (Cai et al., 2023). The original challenge paper subsequently formalized these ideas as the 1M-Deepfakes Detection Challenge, described as a community benchmark and an ongoing benchmarking platform rather than a one-off competition (Cai et al., 2024).
The 2025 challenge is hosted on top of AV-Deepfake1M++, which explicitly extends the earlier benchmark in three directions: more real-source datasets, more generation methods, and more perturbations. The stated purpose is to provide a benchmark that is harder, more diverse, and more realistic, especially under cross-domain evaluation and real-world delivery artifacts such as compression, blur, jitter, and related degradations (Cai et al., 28 Jul 2025).
2. Dataset foundation: from AV-Deepfake1M to AV-Deepfake1M++
The original AV-Deepfake1M dataset contains 1,146,760 total videos, including 286,721 real and 860,039 fake, across 2,068 unique subjects and 1,886 hours of audio-visual data (Cai et al., 2023). The challenge paper summarizes the benchmark as containing more than 1 million manipulated videos across more than 2,000 subjects, with diverse background settings and high-realistic audio-visual content (Cai et al., 2024). Its multimodal composition includes three manipulation categories: Fake Audio and Fake Visual, Fake Audio and Real Visual, and Real Audio and Fake Visual (Cai et al., 2023).
AV-Deepfake1M++ is the 2025 benchmark extension. It reports 2,051,154 video clips, 502,844,910 frames, 4,655.9 hours, 7,109 subjects, 627,936 real samples, and 1,423,218 fake samples (Cai et al., 28 Jul 2025). Compared with AV-Deepfake1M, this roughly doubles dataset scale and substantially increases subject diversity (Cai et al., 28 Jul 2025). The real clips are sourced from VoxCeleb2, LRS3, and EngageNet, addressing the earlier limitation that AV-Deepfake1M used only VoxCeleb2 (Cai et al., 28 Jul 2025).
The 2025 extension also enlarges the synthesis space. Its generation pipeline includes nine state-of-the-art generators spanning audio and visual manipulation. The visual/lip-sync pool includes TalkLip, LatentSync, and Diff2Lip; the audio/TTS pool includes VITS, YourTTS, F5TTS, and XTTSv2 (Cai et al., 28 Jul 2025). The manipulation process continues to follow an LLM-guided semantic edit plan over ASR transcripts, with token-level JSON output of the form
where the operations are replace, delete, and insert (Cai et al., 28 Jul 2025).
The four modality combinations in AV-Deepfake1M++ are explicitly defined as:
| Combination | Audio | Visual |
|---|---|---|
| 1 | real | real |
| 2 | fake | real |
| 3 | real | fake |
| 4 | fake | fake |
A plausible implication is that the challenge is not reducible to either audio-only or vision-only forensics, because the benchmark includes unimodal and joint manipulations by construction.
3. Task definition and challenge protocol
The 2025 challenge includes two tasks on AV-Deepfake1M++ (Cai et al., 28 Jul 2025):
- Video-level classification: determine whether a clip is real or fake.
- Temporal localization: identify the manipulated temporal segment(s) in a video.
The same two-task structure was already present in the earlier 1M-Deepfakes Detection Challenge, where participants had to determine whether an audio-visual sample of a single subject was real or deepfake and, separately, identify the specific time intervals that were manipulated (Cai et al., 2024). The 2025 formulation preserves this structure while evaluating on TestA and TestB, with participants required to train only on the official training split (Cai et al., 28 Jul 2025).
The split structure of AV-Deepfake1M++ is as follows (Cai et al., 28 Jul 2025):
| Split | #Videos | #Subjects |
|---|---|---|
| Training | 1,099,217 | 2,606 |
| Validation | 77,326 | shared with training |
| TestA | 828,318 | 4,503 |
| TestB | 46,293 | shared with TestA |
| Overall | 2,051,154 | 7,109 |
The paper emphasizes that training/validation are sample-level random splits, while the test sets are designed to be domain-shifted, with different identities, real sources, and generative methods across splits (Cai et al., 28 Jul 2025). This suggests that leaderboard performance is intended to measure not only in-distribution accuracy but also robustness under cross-domain generalization.
The predecessor benchmark used a different split regime: 1,657 subjects for train/validation and 411 subjects for test, with no overlap between train and test subjects and split proportions of 65% train, 5% validation, 30% test (Cai et al., 2024). A notable detail there was that the test set used only high-quality VITS-generated audio manipulations, making evaluation especially stringent and consistent for audio-related synthesis artifacts (Cai et al., 2024).
4. Evaluation metrics and scoring
For classification, the challenge uses AUC (area under the ROC curve) (Cai et al., 2024, Cai et al., 28 Jul 2025). In the earlier challenge paper, the AUC range is given as [0.5, 1], where 0.5 denotes random guessing and 1.0 denotes perfect prediction (Cai et al., 2024).
For temporal localization, the challenge uses a score based on Average Precision (AP) and Average Recall (AR) following prior temporal localization work (Cai et al., 2024). The metric notation is preserved in the 2025 benchmark as:
- AP@{0.50,0.75,0.90,0.95}
- AR@{50,30,20,10,5} (Cai et al., 28 Jul 2025)
The earlier challenge paper specifies the thresholds and proposal counts in detail. For AP, the IoU thresholds are 0.5, 0.75, 0.9, 0.95. For AR, the IoU thresholds are 0.5:0.95:0.05, and the proposal counts are 50, 30, 20, 10, 5 (Cai et al., 2024). The metrics are averaged across thresholds and proposal counts to produce overall AP and AR summary values, which are then used for the final result (Cai et al., 2024).
The final score is given as
$S = \frac{\sum_{IoU\in\{0.5,0.75,0.9,0.95\}AP@IoU}{8} + \frac{\sum_{N\in\{50,30,20,10,5\}AR@N}{10}$
with the clarification that the typesetting is malformed but the intended meaning is a combination of averaged AP and AR across the specified IoU thresholds and proposal counts (Cai et al., 2024). The score lies in [0,1], where 0 corresponds to random guessing and 1 to perfect predictions (Cai et al., 2024).
Operationally, the earlier challenge used CodaBench as the automated submission and scoring platform (Cai et al., 2024). For the 2025 challenge, the dataset, challenge details, and evaluation scripts are available online under a research-only license, and the leaderboard is maintained on the challenge evaluation page (Cai et al., 28 Jul 2025).
5. Generation pipeline and realism assumptions
The benchmark’s realism derives from the underlying AV-Deepfake1M generation pipeline. A real video is processed through a three-stage procedure: transcript manipulation, audio generation, and video generation (Cai et al., 2023).
In the transcript stage, audio is extracted with FFmpeg, transcripts are generated using Whisper, and then modified by ChatGPT with a few-shot prompt (Cai et al., 2023). The number of transcript edits is governed by
where is the number of modifications and is video duration in seconds (Cai et al., 2023). This makes longer videos contain more edits.
In the audio stage, background noise and speech are separated using Denoiser in AV-Deepfake1M (Cai et al., 2023), whereas AV-Deepfake1M++ describes speech/background separation using Demucs (Cai et al., 28 Jul 2025). Audio is synthesized with VITS for a subset of subjects and YourTTS for the remaining subjects in the original dataset (Cai et al., 2023), while the 2025 extension broadens the TTS pool to include F5TTS and XTTSv2 as well (Cai et al., 28 Jul 2025). The original paper explicitly uses two audio synthesis strategies for replacements and insertions: synthesizing the whole fake transcript and cropping the relevant part, or synthesizing only the new words directly (Cai et al., 2023).
In the video stage, AV-Deepfake1M evaluates several reenactment options—EAMM, AVFR-GAN, DiffTalk, AD-NeRF, and ATVGnet—and selects TalkLip because it is identity-independent, lip-sync focused, fast, state-of-the-art, and open-source (Cai et al., 2023). AV-Deepfake1M++ subsequently expands the visual model pool to include LatentSync and Diff2Lip in addition to TalkLip (Cai et al., 28 Jul 2025).
A central realism mechanism in AV-Deepfake1M++ is the addition of real-world perturbations. Video perturbations include Gaussian blur, salt-and-pepper noise, low bitrate / compression artifacts, Gaussian noise, Poisson noise, speckle noise, color quantization, random brightness, motion blur, rolling shutter, camera shake, lens distortion, vignetting, exposure variation, and chromatic aberration. Audio perturbations include compression artifacts, pitch/loudness change, white noise, time stretch, reverberation, ambient noise, clipping, frequency filtering, Doppler, interference, room impulse, and padding simulation (Cai et al., 28 Jul 2025). Held-out tests add harder perturbations such as frame-rate jitter, pixelation distortion, localized defocus blur, frame dropouts, random spatial warping, random frame shuffle, audio stutter/repeat, audio stutter, audio frame shuffle, and padding simulation (Cai et al., 28 Jul 2025).
This perturbation regime is explicitly motivated by the need to model online-video delivery artifacts that may hide forensic traces, create misleading artifacts, alter temporal coherence, and induce brittleness in detectors trained on clean synthetic data (Cai et al., 28 Jul 2025).
6. Methods, leaderboard results, and technical tendencies
The earlier challenge paper reports participation statistics from the 2024 edition: 191 teams signed the EULA and registered, 1034 successful submissions were made by the deadline, and Task 1 received 51% more submissions than Task 2 (Cai et al., 2024). The paper interprets this imbalance as evidence that the community still focuses more on video-level detection than on temporal localization, despite the importance of partial manipulations (Cai et al., 2024).
For the earlier detection task, the top three teams were Fake-detector (USTC), UQCV (UQ), and FRIdatas (FRI) (Cai et al., 2024). Their reported methods indicate several technical tendencies:
- Fake-detector (USTC) introduced an Audio-Visual Local-Global Interaction Module (AV-LG Module) with Local Intra-Region Self-Attention, Global Inter-Region Self-Attention, and Local-Global Interaction (Cai et al., 2024).
- UQCV (UQ) used weakly supervised pseudo-labeling from audio-based pseudo-labeling and mouth-region-based pseudo-labeling, iteratively adding high-confidence samples for fine-tuning (Cai et al., 2024).
- FRIdatas (FRI) used a vision transformer, Wav2Vec-XLS-R, an embedding module, and temporal convolution to detect fake chunks in video (Cai et al., 2024).
For the earlier localization task, the top three teams were Gradiant, Maya (USTC), and nudt24 (NUDT) (Cai et al., 2024). Their methods included:
- Gradiant: gMLP for audio features plus optical flow and RGB processed by UMMAFormer, with output fusion (Cai et al., 2024).
- Maya (USTC): BYOL-A for audio features, TSN and InternVideo for visual features; BYOL-A + InternVideo for detection and BYOL-A + TSN for localization (Cai et al., 2024).
- nudt24 (NUDT): THE-FD (Task Hierarchical Emotion-aware for Fake Detection), structured to process video-level data first and adapt to frame-level data through hierarchical feature inheritance and sharing (Cai et al., 2024).
The 2025 AV-Deepfake1M++ benchmark reports leaderboard-style results on TestA. For classification (AUC), the listed teams are XJTU SunFlower Lab: 97.83, WHU_SPEECH: 93.07, KLASS: 92.78, Pindrop Labs: 92.49, and Mizhi Labs: 91.78. The Xception baseline scores 55.09 on TestA and 57.29 on TestB (Cai et al., 28 Jul 2025). For temporal localization on TestA, the reported results are Pindrop Labs: 67.20, Mizhi Lab: 55.00, Purdue-M2: 50.87, WHU_SPEECH: 41.30, KLASS: 35.36, BA-TFD+ baseline: 14.71, and BA-TFD baseline: 13.54; on TestB, BA-TFD+ baseline and BA-TFD baseline achieve 15.15 and 11.17, respectively (Cai et al., 28 Jul 2025).
One reported interpretation is especially important: temporal localization is dramatically harder under the AV-Deepfake1M++ conditions, with methods that performed strongly on prior datasets collapsing sharply (Cai et al., 28 Jul 2025). The authors highlight BA-TFD+ as a concrete example, noting 96.30 [email protected] on LAV-DF but only 14.7 [email protected] on AV-Deepfake1M++ (Cai et al., 28 Jul 2025).
A specific 2025 challenge system, KLASSify, illustrates a different design choice. Rather than using a heavy end-to-end multimodal transformer, it decouples audio and visual reasoning (Kukanov et al., 10 Aug 2025). The visual branch uses MediaPipe Face Mesh and eight handcrafted temporal artifact features, including Blurriness of the Mouth ROI, Non-mouth Mean Squared Error (MSE), Color Shift of the Mouth ROI, and Landmark Kinematics such as Mouth Aspect Ratio, Velocity, Acceleration, Jerk, and Jitter (Kukanov et al., 10 Aug 2025). These features are processed by a lightweight 1D TCN with 124K parameters for classification and about 140K parameters for localization; the localization head predicts frame-level tags under the BILOU scheme with labels O, B, I, L (Kukanov et al., 10 Aug 2025). The audio branch uses Wav2Vec-AASIST for classification and an adapted Boundary-aware Attention Mechanism (BAM) with WavLM-Large for localization (Kukanov et al., 10 Aug 2025).
KLASSify reports AUC 92.78% for classification on AV-Deepfake1M++ TestA using multimodal fusion, and IoU 0.3536 for localization using only the audio modality via KLASSify-BAM (Kukanov et al., 10 Aug 2025). The paper further reports that simple score averaging of the best audio/video classifiers gives 91.97% AUC, while calibration plus Max-Out improves this to 92.78% (Kukanov et al., 10 Aug 2025). This suggests that, within that system, multimodal fusion is most effective for classification, whereas audio is the effective modality for localization.
7. Significance, difficulty, and research implications
The challenge’s broader methodological message is that deepfake analysis should not be restricted to the question “is this whole video fake?” but should instead include cross-modal analysis, temporal localization, partial-forgery scenarios, high-realism data, and standardized protocols (Cai et al., 2024). This principle originates directly from the AV-Deepfake1M benchmark, whose main claim is that existing methods experience a substantial performance collapse when confronted with short localized manipulations, multimodal edits, and very limited fake content per clip (Cai et al., 2023).
The original dataset paper provides several quantitative indicators of difficulty. On AV-Deepfake1M, temporal localization drops sharply relative to LAV-DF: BA-TFD falls from 79.15 → 37.37 [email protected], BA-TFD+ from 96.30 → 44.42, and UMMAFormer from 98.83 → 51.64 (Cai et al., 2023). In detection, many methods remain near chance, with the best frame-level method reported as Xception: 68.68 AUC, while the paper explicitly notes that even with frame-level labels, best AUC remains under 70 due to multimodal manipulations (Cai et al., 2023). Human evaluation also found the dataset much harder than LAV-DF, with accuracy dropping from 84.03 → 68.64 and localization-oriented measures dropping sharply as well (Cai et al., 2023).
AV-Deepfake1M++ intensifies this difficulty by increasing source diversity, generation diversity, and perturbation diversity (Cai et al., 28 Jul 2025). The benchmark’s stated conclusion is therefore not merely that the challenge is large, but that it is structured to stress robustness to unseen manipulations and perturbations, especially for localization (Cai et al., 28 Jul 2025). This suggests that methods optimized narrowly for generator-specific artifacts or pristine synthesis traces may not transfer effectively.
A recurring misconception in deepfake benchmarking is that strong video-level classification performance implies robust forensic competence. The AV-Deepfake1M challenge family directly counters that assumption. Classification scores in the 90s can coexist with substantially weaker localization behavior, and localization baselines that were competitive on earlier datasets can degrade severely under the AV-Deepfake1M++ regime (Cai et al., 28 Jul 2025). The challenge therefore distinguishes coarse clip-level discrimination from the finer problem of identifying manipulation boundaries in realistic audiovisual streams.
The practical ecosystem around the benchmark is also part of its significance. The associated repository provides evaluation scripts, baseline models, and accompanying code, and the evaluation server remains accessible beyond the challenge deadline to support continued benchmarking and future experimentation (Cai et al., 2024). In this sense, the AV-Deepfake1M 2025 Challenge functions not only as a competition but as a continuing benchmark infrastructure for research on content-driven, temporally localized audio-visual deepfakes (Cai et al., 2024, Cai et al., 28 Jul 2025).
