DF40: Universal Deepfake Detection Benchmark

• DF40 is a deepfake detection benchmark designed to overcome past limits by encompassing 40 diverse manipulation techniques including face-swapping, reenactment, synthesis, and editing.
• It standardizes evaluation with four protocols, measuring cross-forgery, cross-domain, and open-set performance using frame-level and video-level AUC metrics.
• The benchmark highlights asymmetric transfer challenges and robust performance differences between state-of-the-art detectors like CLIP-large and traditional models.

DF40 is a comprehensive face deepfake detection benchmark designed to evaluate detector robustness beyond the FF++-centric regime that had dominated the field. It was introduced to address three problems identified as limiting progress in deepfake detection: insufficient forgery diversity, outdated forgery realism, and evaluation protocols that assess only narrow parts of the problem space. DF40 comprises 40 distinct deepfake techniques spanning face-swapping, face-reenactment, entire face synthesis, and face editing, and it was proposed explicitly as a benchmark for universal deepfake detection under cross-forgery, cross-domain, and open-set conditions (Yan et al., 2024).

1. Motivation and benchmark rationale

The benchmark is motivated by the claim that the dataset itself had become the bottleneck for deepfake detection progress. In the formulation of the original paper, the common recipe of training on a single dataset such as FaceForensics++ and testing on a small number of external datasets had functioned as a “golden compass” for ranking detectors, but this protocol overstated robustness because it was too narrow and too dated relative to contemporary deepfake generation pipelines (Yan et al., 2024).

Three deficiencies are central. First, forgery diversity: prior benchmarks often covered only one or a few manipulation families, especially older face-swapping and face-reenactment pipelines. Second, forgery realism: the prevailing training datasets emphasized older artifacts and therefore did not reliably measure generalization to more recent GAN, diffusion, and commercial systems. Third, evaluation protocol: most prior work tested only one forgery type or a small within-domain slice, which hindered development of detectors intended to operate across manipulation families, source domains, and previously unseen generators (Yan et al., 2024).

This design position matters because DF40 does not present benchmarking as a single held-out accuracy number. It instead treats robustness as the interaction of manipulation family, source domain, and openness of the test condition. The benchmark’s architecture therefore separates same-domain cross-forgery transfer, same-forgery cross-domain transfer, and unknown-forgery unknown-domain evaluation, which is a more demanding definition of generalization than the older FF++-anchored protocol.

2. Composition, taxonomy, and scale

DF40 focuses on face deepfakes and organizes them into four manipulation categories: face-swapping (FS), face-reenactment (FR), entire face synthesis (EFS), and face editing (FE). Formally, it contains 40 distinct deepfake techniques, which the benchmark paper describes as 10 times larger than FF++ in manipulation-method count (Yan et al., 2024).

The category structure is as follows.

Category	Count	Notes
Face-swapping (FS)	10	Includes DeepFaceLab as test only
Face-reenactment (FR)	13	Includes HeyGen as test only
Entire face synthesis (EFS)	12	Includes MidJourney-6 and WhichisReal as test only
Face editing (FE)	5	All 5 are test only in the benchmark protocol

The benchmark describes its scale as over 0.1M fake video clips for the video-based categories FS and FR, and over 1M fake images for the image-based categories EFS and FE (Yan et al., 2024). Its main real domains are FF++ (c23) and Celeb-DF-v2 (CDF). Additional real sources are used for some test-only methods: UADFV for DeepFaceLab, VFHQ for HeyGen, FFHQ for MidJourney-6 and WhichisReal, and CelebA for some face-editing methods (Yan et al., 2024).

A defining design choice is the use of 9 test-only unknown-domain forgeries: DeepFaceLab, HeyGen, MidJourney-6, WhichisReal, CollabDiff, e4e, StarGAN, StarGANv2, and StyleCLIP. These are reserved for open-set evaluation and are not used in training (Yan et al., 2024). This creates an explicit distinction between “known” methods used during benchmark training and “unknown” methods used to approximate deployment conditions.

The method roster is also intentionally modern. The benchmark paper emphasizes the inclusion of recent or widely used systems such as PixArt-$\alpha$, DiT, SiT, DeepFaceLab, HeyGen, and MidJourney-6 (Yan et al., 2024). That choice is directly tied to the realism critique: detectors that rely on the artifact priors of older blending-based or early GAN-era forgeries are expected to fail when confronted with these newer methods.

3. Construction principles and formal task structure

The benchmark is built primarily from FF++ and CDF. For FF++, the paper uses the c23 compression version and follows the official split of 720 videos for training, 140 for validation, and 140 for testing. For CDF-v2, it uses 518 testing videos, derived from the original split of 340 fake videos and 178 real videos, and generates 680 fake videos per method in the CDF domain through bidirectional identity pairing (Yan et al., 2024).

The benchmark paper provides a unified generation view in which a target face is written as $x_t(i_t,a_t,b_t)$, with identity $i_t$, identity-agnostic content $a_t$, and external attributes $b_t$. The four manipulation families are defined as follows (Yan et al., 2024):

• Face-swapping:

$$f_{sw}(x_t(i_t, a_t, b_t), x_s(i_s, a_s, b_s)|\text{swap}) = \tilde{x}_t(\tilde{i}_s,a_t,b_t)$$

• Face-reenactment:

$$f_{re}(x_t(i_t, a_t, b_t)|c_a) = \tilde{x}_t(i_t,\tilde{a}_s, b_t)$$

• Entire face synthesis:

$$f_{sy}(n) = \tilde{x}_t(\tilde{i}_t, \tilde{a}_t, \tilde{b}_t)$$

• Face editing:

$$f_{ed}(x_t(i_t, a_t, b_t)|c_b) = \tilde{x}_t(i_t,a_t,\tilde{b}_s)$$

This formalization is not merely notational. It encodes the benchmark’s central argument that “deepfake detection” is not a single manipulation class but a composite of identity replacement, motion transfer, full synthesis, and attribute editing. A plausible implication is that detectors trained on one of these transformations should not be assumed to generalize symmetrically to the others unless the benchmark demonstrates it.

DF40 standardizes preprocessing and training through DeepfakeBench. Reported settings include $224 \times 224$ input for CLIP, $x_t(i_t,a_t,b_t)$0 for other image detectors, Adam optimization, learning rates of $x_t(i_t,a_t,b_t)$1 for CLIP and $x_t(i_t,a_t,b_t)$2 for non-CLIP detectors, weight decay $x_t(i_t,a_t,b_t)$3, batch size 32, and 15 training epochs (Yan et al., 2024). The primary benchmark metric is frame-level AUC, while video-level AUC is reported for I3D (Yan et al., 2024).

4. Evaluation protocols and principal findings

A major contribution of DF40 is its 4 standard protocols (Yan et al., 2024).

Protocol 1, cross-forgery evaluation, measures transfer across manipulation categories within the same data domain. Protocol 2, cross-domain evaluation, measures domain shift while keeping the manipulation family aligned. Protocol 3, unknown forgery and domain evaluation, is the open-set setting in which both generator and source domain are unseen. Protocol 4, One-Versus-All (OvA) evaluation, provides a fine-grained transfer matrix over individual forgery methods, reported as a $x_t(i_t,a_t,b_t)$4 heatmap because the 9 unknown-domain methods are excluded from the pairwise training matrix (Yan et al., 2024).

The benchmark reports over 2,000 evaluations and studies 8 representative detectors/models: Xception, SRM, SPSL, RECCE, RFM, SBI, CLIP, and I3D. The paper also notes a small inconsistency: the abstract and some sections say 7 representative detection methods, while the evaluation discussion also reports I3D (Yan et al., 2024).

Several quantitative patterns define DF40’s empirical identity. Within-forgery performance can be nearly saturated, often around 0.99 AUC, but cross-forgery transfer drops substantially. For example, training on FS (FF) and testing on FS (FF) yields 0.991 for Xception and 0.996 for CLIP, while training on EFS (FF) and testing on FS (FF) yields 0.665 for Xception and 0.688 for CLIP (Yan et al., 2024). This supports the paper’s claim of asymmetric transfer between manipulation types.

Protocol 3 is the harshest setting. When trained on FS (FF), the average AUC across the unknown methods is 0.546 for Xception and 0.692 for CLIP. When trained on FR (FF), the corresponding averages are 0.349 and 0.588. When trained on EFS (FF), they rise to 0.679 and 0.802 (Yan et al., 2024). The benchmark interprets this as evidence that EFS-trained detectors learn more transferable synthesis cues than detectors trained on older face-swap or reenactment regimes.

The paper crystallizes these observations into 7 findings. These include asymmetric cross-type transfer, the lack of clear superiority of many prior SoTA detectors over plain Xception under harder settings, the consistent strength of CLIP—especially CLIP-large—the insufficiency of blending alone as a universal cue, the relatively minor drop when crossing only domain or only forgery, the importance of the joint interaction between domain and forgery, and the observation that FR methods often share transferable patterns except for Wav2Lip (Yan et al., 2024).

One especially consequential benchmark result is the comparison of models trained on DF40 (FF) across 12 face-domain test sets: Xception 0.644, CLIP-base 0.814, and CLIP-large 0.898 in average AUC (Yan et al., 2024). The benchmark also reports that CLIP-large trained only on face-domain DF40 transfers to the non-face GenImage benchmark with 0.746 average AUC, and that training only on EFS (FF) yields 0.815 average AUC there (Yan et al., 2024). This suggests that at least part of the representation learned from EFS in DF40 is not narrowly face-specific.

5. Subsequent use of DF40 in later research

Later work treats DF40 as a central evaluation bed rather than only as a dataset release. In the explainable detection framework DF-P2E, DF40 is the main benchmark for the end-to-end pipeline. That paper adopts “the recently published DF40 dataset” as its main benchmark, describes it as comprising 40 distinct manipulation techniques, and uses it to compare XceptionNet, CLIP-base, and CLIP-large as detection backbones before selecting CLIP-large for Grad-CAM-based explanation generation. It reports frame-level AUCs of 0.776 for Xception, 0.900 for CLIP-base, and 0.913 for CLIP-large in the reported cross-dataset table, but also notes that many benchmark-specific implementation details remain unspecified, including exact subset sizes, split composition, and training hyperparameters (Tariq et al., 11 Aug 2025).

Another line of work uses DF40 explicitly as an out-of-distribution suite for modern foundation models. A frozen-backbone linear-probe study evaluates RoPE-ViT, DINOv3, and NVIDIA C-RADIOv4-H on four DF40 subsets: CollabDiff, StyleCLIP, MidJourney, and WhichFaceIsReal. Its central DF40 finding is an asymmetry between localized editing and entire face synthesis: localized face editing is markedly harder, with several models collapsing toward near-trivial predictions on CollabDiff and StyleCLIP, while entire face synthesis remains comparatively tractable. The paper further identifies a DF40-specific challenge in the low native resolution of some editing subsets, reporting approximately $x_t(i_t,a_t,b_t)$5 native patch resolution for CollabDiff and StyleCLIP, and argues that globally pooled linear probes discard the local evidence required for these cases (Delibasoglu, 24 May 2026).

DF40 has also been used as a “standard benchmark” in human-versus-AI comparison. A study of 200 human participants and 95 AI detectors evaluates a 1,000-video DF40 sample consisting of 500 real and 500 fake videos. On DF40, individual humans achieve mean accuracy 0.743, while individual AI detectors achieve 0.610. The human ensemble reaches 0.890 accuracy, the AI ensemble 0.869, and the hybrid ensemble 0.941, with catastrophic failure rate reduced to 0.000 for the hybrid system. In that study, DF40 functions as the benchmark-style condition against which a lower-quality, more realistic dataset is contrasted (Postiglione et al., 15 Mar 2026).

These downstream uses show that DF40 has become not only a benchmark for detector ranking but also a diagnostic instrument for interpretability, domain shift, representation analysis, and human-AI complementarity.

6. Limitations, ambiguities, and research directions

DF40’s benchmark paper acknowledges at least one major limitation: comparatively limited analysis of video-level detectors. I3D is included, but the paper’s main emphasis is on frame/image-level detectors and frame-level AUC (Yan et al., 2024). The release also does not provide, in the main text, a full low-level specification of frame extraction rate, exact face cropping and alignment, or per-generator postprocessing settings; those details are deferred to supplementary material and code (Yan et al., 2024).

Some ambiguities arise in downstream usage. The DF-P2E paper states that all pipeline modules are trained and evaluated on subsets of DF40 following its standardized splits and preprocessing pipeline, yet it also leaves unclear whether all captioning and narrative-refinement data come from DF40 alone. Its cross-dataset table caption is described as somewhat confusing relative to its broader earlier description of DF40, and key benchmark details such as exact split counts, subset sizes, and training hyperparameters are not specified (Tariq et al., 11 Aug 2025). This suggests that later reuse of DF40 may not always preserve a uniform evaluation protocol.

The original benchmark paper also raises underexplored questions rather than treating DF40 as a completed endpoint. These include what blending-based training can still contribute, how incremental learning should absorb a growing number of forgery types, whether manipulations should be grouped by broad category or by a more refined artifact-based taxonomy, and whether stronger pretrained models can be extended to video or multimodal detection (Yan et al., 2024). A plausible implication is that DF40’s most durable contribution may be methodological: it redefines deepfake benchmarking as a problem of structured transfer across forgery families, source domains, and previously unseen generators, rather than as a single within-dataset accuracy contest.

In that sense, DF40 is best understood as a benchmark for modern realism robustness and universal deepfake detection. Its scale—40 methods across swapping, reenactment, synthesis, and editing—is important, but the deeper contribution lies in the way it exposes the interaction of manipulation type, domain, and openness of the test condition (Yan et al., 2024).