HydraFake Deepfake Benchmark
- HydraFake is a deepfake benchmark designed to evaluate detectors using hierarchical out-of-distribution testing across unseen models, forgery types, and image domains.
- The dataset comprises 100K images (50K real and 50K fake) curated from diverse sources with meticulous quality filtering and prompt-based refinement.
- Its hierarchical protocol distinguishes cross-model, cross-forgery, and cross-domain scenarios, revealing nuanced robustness and advocating reasoning-based detection approaches.
Searching arXiv for papers mentioning “HydraFake” and closely related names to disambiguate the term. HydraFake is a deepfake benchmark introduced in "Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning" (Tan et al., 28 Aug 2025). It is designed to simulate real-world challenges with hierarchical generalization testing rather than a single train/test split. HydraFake-100K contains 100,000 images, with 50K real images and 50K fake images, and its stated purpose is to evaluate whether a detector can cope with unseen model architectures, emerging forgery techniques, novel data domains, and mixed image qualities and resolutions.
1. Motivation and problem setting
HydraFake was created to address a mismatch between standard academic deepfake evaluation and deployment conditions. The benchmark is motivated by the observation that existing academic benchmarks often use homogeneous training sources and low-quality testing images, and that many detectors are trained on a small set of forgery sources and then tested on other benchmarks that are not representative of the rapidly changing in-the-wild landscape (Tan et al., 28 Aug 2025).
The design premise is that detector performance must be decomposed by novelty type. In HydraFake, the central issue is not only whether a detector transfers across datasets, but whether it remains reliable when the generator architecture changes, when the manipulation family itself changes, or when both real and fake samples come from unfamiliar domains. This framing makes HydraFake a benchmark for out-of-distribution generalization rather than a conventional corpus for closed-world discrimination.
2. Dataset composition and curation
HydraFake-100K is built from real and fake image sources selected to expose different generalization regimes. The real images are collected from 8 public datasets: LFW, CelebA, FF++, FFIW, FFHQ, VFHQ, UADFV, and CelebAHQ. These are divided into low-resolution sources—LFW, CelebA, FF++, FFIW—and high-resolution sources—FFHQ, VFHQ, UADFV, CelebAHQ. For training, the benchmark uses 5 real subsets, yielding 24K real training images (Tan et al., 28 Aug 2025).
The fake images are drawn from three source categories. The first is classic deepfake data from FF++, DF40, and FFIW, mainly covering face swapping and face reenactment and spanning 10 generative models. The second is public deepfake data from WILD, seeprettyface, and TalkingHeadBench, containing carefully synthesized faces from 16 popular generators. The third is a self-constructed and crawled set of about 10K deepfake images from 10 advanced generators, including face restoration, face relighting, face personalization, generative face swapping, VAR-based generation, and commercial or social-media sources such as GPT-4o, Dreamina, and Hailuo AI. After filtering and balancing, the final dataset contains 50K real and 50K fake images (Tan et al., 28 Aug 2025).
Curation is a defining part of HydraFake. The benchmark excludes some low-quality classic datasets such as DFDC, DFDCP, and WDF because of unwanted blur or quality issues in real images. It uses Qwen2.5-VL-72B to generate tailored prompts for personalization instead of simple templates, manually selects high-quality relighting results, and manually inspects and filters social-media crawls. This suggests that HydraFake is structured to reduce confounds from dataset quality and to make resolution bias visible rather than silently absorbed into the benchmark.
3. Hierarchical out-of-distribution protocol
HydraFake is organized around a restricted-but-abundant training regime and a test regime that introduces distinct forms of novelty. Training uses 48K images total, with real training images from 5 subsets and fake training images from 21 subsets, but only 3 forgery types are seen in training: FS, FR, and EFG (Tan et al., 28 Aug 2025).
| Split | Novelty type | Size |
|---|---|---|
| In-Domain (ID) | Same source datasets as training, but different identities | 14K |
| Cross-Model (CM) | Unseen model architectures / generators | 11K |
| Cross-Forgery (CF) | Unseen manipulation techniques | 12K |
| Cross-Domain (CD) | Both real and fake images from unseen domains | 15K |
The CM split includes unseen generators such as Adobe Firefly, StarryAI, FLUX1.1 Pro, MAGI-1, HART, and Infinity. The CF split includes attribute editing, generative face swapping, IP-preserved personalization, face relighting, and face restoration, with examples such as StarGANv2, CodeFormer, IC-Light, FaceAdapter, PuLID, and InfiniteYou. The CD split is described as the harshest setting because both the forgery technique and the data source change simultaneously, including unseen datasets such as VFHQ, UADFV, and FFIW, along with social-media and commercial fake images of varying quality and resolution (Tan et al., 28 Aug 2025).
This hierarchy separates four failure modes that are often conflated in prior work. A detector that performs well on CM but fails on CF is capturing architecture-transfer regularities without learning invariants across manipulation families; a detector that also fails on CD is additionally sensitive to domain and resolution shifts. HydraFake’s protocol is designed precisely to make those distinctions explicit.
4. Formal task framing and connection to reasoning-based detection
HydraFake is paired with an explicit formalization of the deepfake task. Let the input image be , and let the authenticity label satisfy , where $1$ means fake. The benchmark distinguishes explainable detection from reasoning-based detection. In the former, a model first produces an answer token and then a post-hoc explanation; in the latter, the model produces reasoning and then the answer, changing the mapping from to (Tan et al., 28 Aug 2025).
Within the same paper, HydraFake motivates Veritas, a detector based on pattern-aware reasoning. The argument is that strong performance under HydraFake requires more than memorizing artifact signatures: the model must reason about what kind of manipulation is likely present and adapt reasoning depth to sample difficulty. The reasoning patterns extracted in the paper are fast judgment, reasoning, conclusion, planning, and self-reflection. This suggests that HydraFake functions not only as a stress test for discriminative classifiers, but also as a benchmark for whether multimodal models can internalize forensic reasoning instead of merely emitting post-hoc rationales.
5. Empirical findings and benchmark significance
HydraFake is constructed to reveal a specific empirical pattern: many previous detectors show strong generalization on cross-model scenarios but fall short on unseen forgeries and data domains. The paper reports that methods such as Effort, D, Co-SPY, and ProDet are often very strong on CM, with many near-98% results, yet performance drops sharply on CF and CD. The benchmark therefore exposes a difference between cross-model transfer and broader robustness (Tan et al., 28 Aug 2025).
The same study reports that the Veritas cold-start model reaches 96.8% ID, 99.6% CM (average), 84.0% CF, and 55.9% CD, while the final model improves these to 97.3% ID, 98.4% CM, 87.4% CF, and 80.1% CD in the pattern-ablation table. In the full main table, the paper reports an overall average of 90.7 and claims an overall 6.0% average gain over the previous best method. It further states that vanilla MLLMs are generally weak as detectors, that GPT-4o often predicts “real” too often and has extremely low recall, that Gemini-2.5-Pro is stronger than most MLLMs and sometimes competitive with specialized detectors, and that Veritas surpasses Gemini-2.5-Pro by 11.8% average. These findings make HydraFake significant less as a larger dataset than as an evaluative instrument that distinguishes superficial transfer from durable OOD robustness.
A related benchmark conclusion is that cross-forgery and cross-domain evaluation are materially harder than cross-model transfer. In HydraFake’s framing, good CM performance does not imply true robustness. That observation is central to how the benchmark is interpreted and to why it is paired with a reasoning-oriented detector.
6. Terminological ambiguity and adjacent usages
HydraFake should not be conflated with HyperFake, Hydra, or Hydra-themed systems in unrelated security domains. In "HyperFake: Hyperspectral Reconstruction and Attention-Guided Analysis for Advanced Deepfake Detection" (Shekar et al., 24 May 2025), the paper explicitly states that HydraFake is not a separate method in that work; the proposed system is HyperFake, a hyperspectral deepfake detection pipeline. In "Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models" (Wang et al., 19 May 2026), Hydra is a framework for multi-concept backdoor injection under cumulative and decentralized model reuse, not a deepfake benchmark.
The name also overlaps with cybersecurity work outside media forensics. "Hydras and IPFS: A Decentralised Playground for Malware" (Patsakis et al., 2019) uses the hydra metaphor for resilient botnet management via Resource Identifier Generation Algorithms over IPFS, while "Detecting integrity attacks in IoT-based Cyber Physical Systems: a case study on Hydra testbed" (Battisti et al., 2019) studies tampered data injection on the Hydra industrial CPS testbed. These usages are unrelated to HydraFake’s role as a deepfake benchmark.
In current arXiv usage, HydraFake most precisely denotes the dataset introduced alongside Veritas. Its defining contribution is a hierarchical OOD evaluation framework that separates unseen generators, unseen forgery types, and unseen data domains, thereby making generalization failure modes legible at a granularity that conventional deepfake benchmarks often do not provide.