Celeb-DF++: Robust DeepFake Video Benchmark

• Celeb-DF++ is a large-scale DeepFake benchmark covering face-swap, face-reenactment, and talking-face forgeries with comprehensive scenario coverage.
• It integrates 22 diverse DeepFake generation methods and employs strict evaluation protocols to measure cross-method, cross-quality, and cross-dataset performance.
• The benchmark highlights significant challenges in real-world detection, emphasizing the need for robust forensic techniques to handle compression and novel forgery types.

Celeb-DF++ is a large-scale, challenging video DeepFake benchmark specifically constructed to facilitate research in generalizable DeepFake forensics. Unlike earlier datasets, Celeb-DF++ emphasizes broad forensic coverage by encompassing a wide variety of forgery types—including face-swap, face-reenactment, and talking-face forgeries—created by diverse state-of-the-art generation techniques. Its rigorous protocols and granular evaluation strategies are designed to address the persistent gaps in cross-method, cross-quality, and cross-dataset generalization of DeepFake detectors, thereby aligning the benchmark with real-world detection needs (Li et al., 24 Jul 2025).

1. Motivation and Scope

The primary motivation for Celeb-DF++ is to address the shortcomings of existing DeepFake video datasets, which generally comprise a limited set of forgery types—most typically early face-swap methods—thus impeding the development of detectors robust to the ever-evolving landscape of generative attacks. The dataset is constructed to serve three key objectives:

• Comprehensive scenario coverage: Including the three most common forgery types—face-swap (FS), face-reenactment (FR), and talking-face (TF)—each prevalent in real-world media manipulation incidents.
• Diversity of methods: Incorporating 22 recent DeepFake generation methods differing in architecture (autoencoders, GANs, diffusion models, neural rendering), target facial regions (whole face, organs, 3D shape, landmarks, audio‐driven motion), and pipeline design.
• Generalizability assessment: Establishing evaluation protocols that stress not only within-method accuracy but also cross-forgery, cross-quality (compression-robustness), and cross-dataset generalization, reflecting real deployment conditions (Li et al., 24 Jul 2025).

2. Dataset Construction and Statistics

Celeb-DF++ expands significantly on previous benchmarks, both in real and forged content volume and in technical heterogeneity.

2.1 Real Video Collection

• 590 celebrity interview videos sourced from YouTube.
• 59 distinct identities, balanced in gender, age, and ethnicity.
• Each video averages approximately 13 s at 30 fps, yielding a total of about 15 million frames.

2.2 Forged Video Generation

Celeb-DF++ includes 53,196 DeepFake videos, spanning three principal manipulation scenarios and 22 DeepFake methods. Each method contributes roughly 1,900–3,000 forged videos.

Scenario	Number of Methods	Approximate Videos	Methods
Face-swap (FS)	8	~16,000	Celeb-DF original, SimSwap, InSwapper, HifiFace, GHOST, UniFace, MobileFaceSwap, BlendFace
Face-reenactment (FR)	7	~14,000	DaGAN, TPSMM, MCNET, HyperReenact, LIA, FSRT, LivePortrait
Talking-face (TF)	7	~20,000	SadTalker, IP-LAP, AniTalker, EDTalk, Real3D-Portrait, EchoMimic, FLOAT

2.3 Preprocessing and Quality Control

• Pipeline: Face detection → cropping and alignment → resizing to 256×256.
• Color-transfer post-processing minimizes donor–target mismatches.
• Detail-preserving masks (eyebrow/lip-driven convex hulls with smooth blending) suppress boundary artifacts.
• Temporal smoothing of facial landmarks via Kalman filtering reduces flickering.

These steps are designed to maximize the realism and technical comparability of forgeries across diverse generation methods (Li et al., 24 Jul 2025).

3. Evaluation Protocols and Metrics

Celeb-DF++ introduces three principal evaluation protocols and rigorously defined metrics to measure detector performance under conditions that stress generalizability.

3.1 Metrics

• Frame-level ROC AUC: Calculated across all sampled frames ($N=32$ per video), using per-frame “fake” probabilities $s_i$.
• Video-level ROC AUC: The average $\bar s = (1/N)\sum_i s_i$ is used per video; AUC is computed over these averages.
• Area Under the Curve (AUC):

$\mathrm{AUC} = \int_{0}^{1} \mathrm{TPR}(f)\,df$

where $f$ is the false-positive rate.

• Equal Error Rate (EER):

$$\mathrm{EER} = \{\epsilon \mid \mathrm{FAR}(\epsilon) = \mathrm{FRR}(\epsilon)\}$$

with $\mathrm{FAR}$, $\mathrm{FRR}$ the false‐accept and false‐reject rates.

3.2 Evaluation Protocols

• GF-eval (Generalized-Forgery): Train on original Celeb-DF (single FS method); test on all 21 other methods (FS, FR, TF).
• GFQ-eval (Generalized-Forgery across Quality): As GF-eval, but with H.264 compression (c35, c45) applied to test videos.
• GFD-eval (Generalized-Forgery across Datasets): Train on FaceForensics++(HQ); test on all 22 methods of Celeb-DF++.

Each protocol isolates distinct aspects of generalization: cross-method, cross-quality, and cross-dataset respectively (Li et al., 24 Jul 2025).

4. Benchmark Results and Analysis

Celeb-DF++ exposes fundamental weaknesses in current DeepFake detectors.

4.1 Zero-shot Difficulty

When detectors are trained on FaceForensics++(HQ) and tested directly (zero-shot) on Celeb-DF++:

• Frame-level AUC drops from 76.5% (Celeb-DF) to 69.6% (Celeb-DF++).
• Video-level AUC drops from 81.6% to 73.8%.

4.2 GF-eval (Protocol 1)

Across 8 representative detectors:

• Frame-level AUC: 71.7%
• Video-level AUC: 72.1%
• Best model (Effort): ≈83% (frame), 84.4% (video), but with sharp drop on FR/TF scenarios.

4.3 GFQ-eval (Protocol 2)

• H.264 compression (c35):
  • Frame-level AUC ≈68.2% (−3.5% drop)
  • Video-level AUC drops by 2.2%
• Compression (c45):
  • Frame-level AUC ≈63.8%
  • Video-level AUC ≈69.9%

4.4 GFD-eval (Protocol 3)

• Frame-level AUC: 69.4%
• Video-level AUC: 73.7%
• Performance highest for FS (similar to FF++), weaker for FR and TF.

4.5 Scenario-specific Observations

• FS-trained detectors generalize poorly to FR and TF.
• FR⇄TF generalization is unexpectedly strong; models trained on one often surpass 95% AUC on the other, indicative of shared motion cues.
• FS is the “easiest” scenario; audio-driven TF (e.g., EchoMimic, SadTalker) presents the greatest challenge, with some AUC values <60%.

5. Analysis of Detector Failings

Even top-performing detectors (ForAda, Effort, ProDet, CFM) exhibit major degradations when confronting methods, compressions, or domains unseen during training.

• High-frequency artifacts and blending-boundary cues, effective for classical FS detection, become unreliable for modern FR/TF forgeries or after compression.
• Realistic TF generation, which involves audio→landmarks→rendering, suppresses many spatial/temporal artifacts and requires new forensic features (e.g., multi-scale motion cues, audio-visual consistency, self-supervised anomaly learning).
• The observed gaps underscore the challenge of forging detectors that are robust to both method diversity and post-processing typical in the wild (Li et al., 24 Jul 2025).

6. Forward Directions and Open Research Problems

Celeb-DF++ illuminates several key trajectories for advancing generalizable DeepFake forensics:

• Joint training on FS, FR, and TF with balanced sampling to avoid overfitting to a single scenario.
• Development of