DFWild-Cup: Deepfake Detection Benchmark

• DFWild-Cup is a deepfake face detection benchmark that evaluates real-versus-fake classification under heterogeneous sources, multiple manipulation methods, and domain shifts.
• It aggregates data from eight public deepfake datasets with uniform preprocessing and anonymization to prevent reliance on dataset-specific shortcuts.
• The benchmark serves as the IEEE SP Cup 2025 competition dataset, driving the development of ensemble methods and adaptive strategies for improved detection generalization.

Searching arXiv for papers on DFWild-Cup and related benchmark descriptions. DFWild-Cup is a deepfake face detection benchmark for the IEEE SP Cup 2025 task, “Deepfake Face Detection in the Wild,” designed to evaluate robust real-versus-fake classification under heterogeneous sources, multiple manipulation methods, and real-world domain shift. It is described as being built from the DeepfakeBench evaluation framework and assembled from eight public deepfake datasets, with consistent preprocessing, anonymized filenames, and removal of source-identifying information so that performance reflects visual generalization rather than dataset-specific shortcuts (Shanto et al., 28 Jan 2025). In both benchmark-oriented descriptions, DFWild-Cup is presented less as a narrow in-distribution dataset than as a cross-generator generalization testbed for unconstrained deepfake detection (Shanto et al., 28 Jan 2025, Anan et al., 15 Feb 2025).

1. Benchmark role and intended evaluation regime

DFWild-Cup 2025 serves as the competition dataset for the 2025 IEEE Signal Processing Cup. Its stated purpose is to assess deepfake image detectors that remain effective across different source datasets, forgery types, and unseen scenarios rather than merely memorizing the artifact profile of a single manipulation pipeline (Shanto et al., 28 Jan 2025).

This positioning is central to the benchmark’s design. One description emphasizes that the key difficulty is not only classifying images as real or fake, but doing so under “in-the-wild” conditions characterized by heterogeneous sources, multiple manipulation methods, and real-world domain shift. Another frames the same benchmark as a deliberate response to a common failure mode of prior detectors: strong performance when test images resemble the training distribution, followed by degradation on new fake generators, different post-processing pipelines, or unseen manipulation methods (Anan et al., 15 Feb 2025).

The public validation portion is treated as the principal benchmark for model comparison and reporting. In one account it is referred to as the “competition validation dataset (DFWild),” and in both accounts it functions as the held-out validation set for reporting accuracy and related metrics (Shanto et al., 28 Jan 2025).

2. Multi-source construction and source datasets

The benchmark aggregates data from eight public deepfake datasets:

1. Celeb-DF-v1
2. Celeb-DF-v2
3. FaceForensics++
4. DeepfakeDetection
5. FaceShifter
6. UADFV
7. Deepfake Detection Challenge Preview
8. Deepfake Detection Challenge

This composition is explicitly intended to produce a diverse deepfake image dataset and to encourage generalizable detectors rather than methods specialized to one manipulation source (Shanto et al., 28 Jan 2025, Anan et al., 15 Feb 2025).

The multi-source construction introduces several forms of domain shift that are repeatedly identified as central to the benchmark’s difficulty: different face manipulation methods, different video or image source characteristics, varying compression and artifact profiles, and inconsistent data origins and imaging conditions (Shanto et al., 28 Jan 2025). A plausible implication is that DFWild-Cup should not be interpreted as a homogeneous corpus with a single dominant forensic signature. Instead, it is structured to expose detectors to multiple forgery subdomains, each potentially carrying distinct low-level and high-level cues.

This interpretation is reinforced by reported embedding analyses. In one study, t-SNE visualizations on 2,000 real and 2,000 fake training images showed heavy overlap before training, whereas after supervised contrastive training the embeddings became well separated and fake samples appeared as multiple clusters, which the authors associate with different deepfake generation methods (Shanto et al., 28 Jan 2025). Another study likewise reports that fake images from different generators form compact clusters after training, supporting the view that the benchmark contains multiple generator-dependent modes while still permitting learning of more transferable decision boundaries (Anan et al., 15 Feb 2025).

3. Curation, preprocessing, and fairness constraints

A defining feature of DFWild-Cup is its organizer-controlled preprocessing. The component datasets were uniformly preprocessed so that competitors would not benefit from idiosyncratic preprocessing pipelines or from artifacts tied to the original source datasets (Shanto et al., 28 Jan 2025, Anan et al., 15 Feb 2025). Filenames were anonymized, and any information that could identify dataset origin was removed.

These constraints are consequential for benchmark validity. By suppressing metadata cues and source identifiers, DFWild-Cup is designed to measure visual forensic generalization rather than shortcut learning. The benchmark therefore attempts to isolate the actual detection problem: discriminating manipulated from authentic facial imagery when superficial indicators of provenance are unavailable (Shanto et al., 28 Jan 2025).

The supplied descriptions do not present a single benchmark-wide input geometry beyond the organizer preprocessing. Instead, input handling is model-dependent in the submitted systems. One method resizes all input images to $256 \times 256$ pixels and applies offline augmentation to real images together with online augmentation during training (Shanto et al., 28 Jan 2025). Another states that images were resized according to the input requirements of each architecture and normalized using the ImageNet mean and standard deviation; it also applies random horizontal flip, random rotation in $0^\circ$–$10^\circ$, color jitter with range $0.2$ for brightness, contrast, saturation, and hue, and random resized crop to $224 \times 224$ with crop area scaling between $80\%$ and $120\%$ for two of its branches (Anan et al., 15 Feb 2025). These are method-level responses to the dataset rather than properties of the benchmark specification itself.

4. Label space, split structure, and class imbalance

DFWild-Cup is a binary benchmark with two labels: Real and Fake. The reported split is as follows (Shanto et al., 28 Jan 2025, Anan et al., 15 Feb 2025):

Split	Real	Fake
Training set	42,690	219,470
Validation set	1,548	1,524

The training set contains 262,160 images in total, and the validation set contains 3,072 images in total (Shanto et al., 28 Jan 2025). The training distribution is highly imbalanced, with fake samples outnumbering real samples by about $5{:}1$ (Anan et al., 15 Feb 2025).

This imbalance is not a peripheral property; it is explicitly treated as a key practical and methodological issue. One account notes that a model trained directly on the raw distribution could become biased toward the fake class, and the corresponding training strategy therefore samples and rebalances during backbone training (Shanto et al., 28 Jan 2025). Another formalizes the same challenge through a multistage disjoint-set training procedure in which the real-image set $\mathcal{R}$ is paired sequentially with five disjoint fake subsets $\{\mathcal{F}_1,\ldots,\mathcal{F}_5\}$, using

$0^\circ$0

and then

$0^\circ$1

This formulation is intended to ensure that every fake image is eventually seen while preventing the majority class from dominating optimization at a single stage (Anan et al., 15 Feb 2025).

A common misconception is to treat the benchmark as difficult only because of forgery diversity. The reported evidence indicates that its difficulty is joint: class imbalance, heterogeneous generation sources, and generalization to unseen generators are all presented as core factors (Anan et al., 15 Feb 2025).

5. Methodological adaptations induced by the benchmark

The benchmark’s structure has directly shaped the design of submitted detection systems. In one approach, three backbones—MaxViT, CoAtNet, and EVA-02—are fine-tuned with supervised contrastive loss to improve separation between real and fake embeddings (Shanto et al., 28 Jan 2025). To address the skewed class distribution during backbone training, that system uses all 42,690 original real images together with 21,335 offline-augmented real images, for a total of 64,025 real images, and constructs fake training subsets of 73,157 images each, supplemented by 12,200 generated fake images from external deepfake synthesis methods. Each backbone is reported as being trained on 64,025 real images and 85,357 fake images (Shanto et al., 28 Jan 2025).

The same study further augments the competition data with seven secondary synthesis methods—E4S, FaceDancer, BlendFace, InSwapper, HyperReenact, e4e, and StyleCLIP—contributing about 2,000 fake images per method, for a total of 12,200 additional fake samples (Shanto et al., 28 Jan 2025). This is explicitly described as part of the training strategy rather than part of the official DFWild-Cup benchmark.

Its final decision rule is a majority-voting ensemble over MaxViT, CoAtNet, and EVA-02:

$0^\circ$2

If the final decision is real, the returned probability is the highest probability among the models; if the final decision is fake, it is the lowest probability among the models (Shanto et al., 28 Jan 2025).

A second response to DFWild-Cup uses a weighted ensemble designed to combine local spatial, global contextual, and frequency-domain cues. In the detailed system description, the branches are ResNet-34 with Squeeze-and-Excitation, DeiT-B, and XceptionNet with Haar wavelet transform, with weights $0^\circ$3, $0^\circ$4, and $0^\circ$5, respectively (Anan et al., 15 Feb 2025). The same source summary notes a discrepancy between the abstract-level characterization—EfficientNet, DeiT, and ConvNeXt with wavelet transform—and the detailed competition model description as a weighted ensemble of ResNet-34 + SE, DeiT, and XceptionNet with wavelet features (Anan et al., 15 Feb 2025). The discrepancy is best read as a reporting inconsistency in the source material rather than as a property of the benchmark itself.

Across both systems, the benchmark elicits the same underlying design principle: no single artifact family is assumed to be sufficient. One method emphasizes local, multi-scale, and global features through MaxViT, CoAtNet, and EVA-02 (Shanto et al., 28 Jan 2025); the other combines local spatial, global attention-based, and frequency-domain representations (Anan et al., 15 Feb 2025). This suggests that DFWild-Cup operationalizes generalization pressure strongly enough that complementary cues become a practical necessity.

6. Reported results and benchmark significance

The reported validation results indicate that DFWild-Cup is challenging but tractable with robust representation learning and model fusion. In one benchmark comparison, classical CNN baselines reach accuracies in the high-$0^\circ$6 range: ResNet50 at $0^\circ$7, ResNet152 at $0^\circ$8, ResNet101 at $0^\circ$9, InceptionV3 at $10^\circ$0, and InceptionResNetV2 at $10^\circ$1 (Shanto et al., 28 Jan 2025). The same study reports substantially stronger single-model results for selected pretrained backbones: MaxViT with accuracy $10^\circ$2 and AUC $10^\circ$3, CoAtNet with accuracy $10^\circ$4 and AUC $10^\circ$5, and EVA-02 with accuracy $10^\circ$6 and AUC $10^\circ$7. Its final ensemble reaches accuracy $10^\circ$8, F1 $10^\circ$9, precision $0.2$0, recall $0.2$1, and AUC $0.2$2 on DFWild (Shanto et al., 28 Jan 2025).

A second study reports validation accuracy $0.2$3, precision $0.2$4, recall $0.2$5, F1-score $0.2$6, AUROC $0.2$7, Equal Error Rate $0.2$8, and Average Precision $0.2$9 for its proposed weighted ensemble (Anan et al., 15 Feb 2025). It also provides single-model accuracies of $224 \times 224$0 for ResNet-34 + SE, $224 \times 224$1 for Xception + Wavelet, and $224 \times 224$2 for DeiT, along with lower-performing comparison configurations including Xception-Net at $224 \times 224$3, VGG-19 at $224 \times 224$4, ResNet-50 at $224 \times 224$5, DCT + ResNet-34 at $224 \times 224$6, ResNet-34 at $224 \times 224$7, ResNet-34 + CBAM at $224 \times 224$8, majority voting ensemble at $224 \times 224$9, and a two-branch weighted ensemble at $80\%$0 (Anan et al., 15 Feb 2025).

Several benchmark-level conclusions are explicitly drawn from these results. First, standard CNN baselines plateau around $80\%$1–$80\%$2 accuracy in one report, indicating that the task exposes weaknesses in conventional architectures (Shanto et al., 28 Jan 2025). Second, transformer or hybrid backbones generalize better, surpassing $80\%$3 and $80\%$4 accuracy in the stronger systems (Shanto et al., 28 Jan 2025). Third, AUC values near $80\%$5 indicate strong separability even when exact classification remains nontrivial (Shanto et al., 28 Jan 2025). Fourth, the improvement from the best single model to the ensemble—from MaxViT at $80\%$6 to the three-model ensemble at $80\%$7—is described as modest but meaningful, implying complementary error patterns across models (Shanto et al., 28 Jan 2025).

Taken together, these findings position DFWild-Cup as a realistic mixed-source benchmark for deepfake detection under domain shift, class imbalance, and multiple manipulation types. Its significance lies in the way its construction and curation force detectors away from metadata exploitation and toward transferable visual evidence. A plausible implication is that DFWild-Cup is best understood not as a mere aggregation of existing datasets, but as a benchmark whose principal scientific value is the controlled measurement of cross-generator robustness under deliberately adversarial data heterogeneity (Shanto et al., 28 Jan 2025, Anan et al., 15 Feb 2025).