OpenFake: Political Deepfake Benchmark
- OpenFake is a large-scale, politically focused benchmark for synthetic image detection, pairing 3 million real images with nearly 1 million synthetic images.
- It employs a prompt-conditioned generation pipeline using both open-source and proprietary models to ensure high visual fidelity and real-world relevance.
- OpenFake Arena, a continuously updated adversarial platform, refines detection benchmarks by incorporating successful synthetic submissions via community engagement.
OpenFake is an open dataset and platform for large-scale deepfake detection centered on contemporary, politically sensitive synthetic imagery. It was introduced to address a gap in existing benchmarks, which the paper characterizes as limited by outdated generation methods, low realism, or single-face imagery, and therefore insufficient for general synthetic image detection. OpenFake combines 3 million real images paired with descriptive captions and 963,342 high-quality synthetic images generated from a mix of proprietary and open-source models, and it couples this corpus with a continuously updated crowdsourced adversarial system, OpenFake Arena, intended to keep detection benchmarks aligned with evolving generative capabilities (Livernoche et al., 11 Sep 2025).
1. Definition, scope, and problem setting
OpenFake is designed for benchmarking deepfake detection under conditions that are explicitly framed as closer to contemporary misinformation practice than earlier face-centric datasets. Its stated emphasis is political and emotionally charged visual content, reflecting the observation that synthetic media increasingly propagates through multiple modalities of social-media misinformation. The dataset therefore moves beyond a narrow “face swap” conception of deepfakes and instead treats synthetic image detection as a broader authenticity assessment problem involving varied scene types and visual formats (Livernoche et al., 11 Sep 2025).
A central premise of the resource is that recent proprietary image generators have become difficult for the general public to distinguish from real photographs. The associated human perception study is used to motivate the benchmark: OpenFake is not organized around legacy GAN artifacts alone, but around a mixture of newer open and proprietary models, higher visual fidelity, and prompt-conditioned generation from captions derived from real-world images. This makes OpenFake a dataset about modern synthetic image detection rather than only about traditional facial manipulation (Livernoche et al., 11 Sep 2025).
A common misconception is that OpenFake is primarily a facial deepfake benchmark. The released description contradicts that interpretation. The dataset includes politically relevant imagery spanning faces, crowds, protests, disaster scenes, manipulated news, signage, social media screenshots, and everyday objects, and the retained source images are filtered for humans or catastrophes rather than for faces alone (Livernoche et al., 11 Sep 2025).
2. Dataset composition and organization
OpenFake pairs real imagery with prompt-conditioned synthetic counterparts. The real side contains 3 million real images, each paired with an automatically generated descriptive caption. The synthetic side contains 963,342 high-quality synthetic images, generated to correspond to those captions. The public release includes the prompts, the images, and model-attribution metadata, and the assets are hosted on the HuggingFace Hub in Parquet and CSV formats for streaming-friendly access (Livernoche et al., 11 Sep 2025).
The dataset is organized into training and testing splits. The test set is constructed by sampling 1,000 images per generative model, with 200–600 for rare/OOD models, and matching them with real images to form a ~60,000-image test set, described as roughly 3% of the dataset. The remainder is assigned to the training set. This split is intended to support both standard supervised evaluation and more general cross-domain testing (Livernoche et al., 11 Sep 2025).
| Component | OpenFake specification |
|---|---|
| Real images | 3,000,000 |
| Synthetic images | 963,342 |
| Test set | ~60,000 images |
| Storage format | Parquet and CSV |
| Metadata | Prompts and model attribution |
| Platform extension | OpenFake Arena |
The image distribution is deliberately broad. Rather than collapsing the problem into portrait classification, OpenFake includes photographs, posters, screenshots, and other media forms encountered in online discourse. This breadth is part of its benchmark philosophy: the resource is intended to capture “the full spectrum of visual misinformation” rather than a single manipulation subtask (Livernoche et al., 11 Sep 2025).
3. Caption extraction and synthetic generation pipeline
The real-image source is described as LAION-400M, from which images and rough captions are scraped and then filtered for political relevance. A vision-LLM, specified as Qwen2.5-VL, first evaluates whether an image contains real human faces or politically salient and catastrophic content. Only images receiving “yes” for humans or catastrophes are retained. Qwen2.5-VL then generates a concise, model-ready caption for each retained image; these captions become both descriptors of the real images and prompts for text-to-image synthesis (Livernoche et al., 11 Sep 2025).
The generation side mixes open-source and proprietary systems. The open-source list includes Stable Diffusion (v1.5, v2.1, XL, v3.5), Flux (1.0-dev, 1.1-Pro, Schnell), HiDream-I1 Full, Chroma, and various LoRA and community finetunes. The proprietary list includes Midjourney (v6, v7), DALL·E 3, Imagen (3.0, 4.0), GPT Image 1, Ideogram 3.0, Grok 2, and Recraft v3. Images are generated at approximately 1MP resolution with varied aspect ratios such as 1:1, 16:9, and 9:16, as well as varied guidance scales and inference steps, with the stated aim of matching social-media presentation and maximizing fidelity (Livernoche et al., 11 Sep 2025).
This prompt-extraction strategy is foundational to OpenFake’s realism claim. By generating synthetic images from captions derived from real social-media photographs and events, the benchmark attempts to preserve prompt-image plausibility and align synthetic content with actual online narratives. The paper notes overlap in CLIP embedding space via t-SNE visualizations, which it uses as evidence that the synthetic distribution is close to the real prompt-conditioned distribution (Livernoche et al., 11 Sep 2025).
4. OpenFake Arena and continual adversarial updating
OpenFake is not only a dataset; it is also a platform for adversarial data collection. OpenFake Arena is described as an interactive web-based system built as a Gradio app on HuggingFace Spaces. Users receive a prompt sampled from the prompt bank, generate a synthetic image using any model or editing tool, and submit it to the platform. The submission then undergoes a CLIP-based similarity gating stage to verify prompt-image alignment and is evaluated against a live SwinV2-based deepfake detector (Livernoche et al., 11 Sep 2025).
The success criterion is adversarial: if a submitted synthetic image fools the detector and is misclassified as real, the contributor earns a point. Successful samples are then added back into the benchmark, and the detector is periodically retrained on the new adversarial data. The platform also maintains a real-time leaderboard, tracks user points and badges, and records model attribution for contributors. In this sense, OpenFake operationalizes benchmark maintenance as a community-driven coevolution between generation and detection (Livernoche et al., 11 Sep 2025).
This makes OpenFake unusual among forensic datasets. A fixed benchmark can become stale as generator capabilities shift; OpenFake Arena is explicitly structured to reduce that failure mode. A plausible implication is that OpenFake treats distribution shift not as a downstream nuisance but as a first-class design constraint, embedding benchmark refresh directly into the resource itself.
5. Benchmark protocols and empirical usage
The benchmark supports supervised, semi-supervised, zero-shot, and cross-domain transfer evaluations. The supervised reference detector is a SwinV2-Small Vision Transformer fine-tuned on OpenFake with inputs and binary cross-entropy for real/fake classification. The training procedure includes geometric, color, blur, and compression-mimicking augmentations intended to improve robustness against synthetic artifacts and JPEG signal leakage. The paper also lists CLIP-based probe, InternVL, and handcrafted detectors under zero-shot and semi-supervised protocols (Livernoche et al., 11 Sep 2025).
The reported evaluation metrics are True Positive Rate (TPR), True Negative Rate (TNR), Accuracy, F1-score, ROC-AUC, and PR-AUC, with the paper giving
These metrics are used not only for in-domain performance but also for transfer tests between OpenFake and other datasets, which the paper uses to argue that OpenFake is particularly valuable for out-of-distribution detection (Livernoche et al., 11 Sep 2025).
Subsequent work has used OpenFake as a benchmark for detector generalization and efficiency. A training-free detector based on representation sensitivity to structured frequency perturbations reports an average AUC of 0.881 on OpenFake, compared with 0.779 for DTAD, while also reporting substantially lower computational cost (Sonoda et al., 23 Mar 2026). Another study, SSAFE, evaluates frozen multimodal encoders on OpenFake and reports that a linear classifier trained on curated subsets of 5K, 10K, 30K, or 100K samples can match or exceed a SwinV2 detector trained on the full 4M-image OpenFake corpus; with 30K samples, SSAFE reports 99.3 Overall F1, 100.0 ROC AUC, and 100.0 PR AUC, versus 99.2 Overall F1 for the full-data SwinV2 baseline (Lee et al., 7 Jun 2026). These later results indicate that OpenFake has become a testbed not only for raw detection accuracy but also for data efficiency, zero-shot robustness, and training-free methods.
6. Position within the open-forensics landscape
OpenFake occupies a specific niche among recent forensic resources. OpenForensics focuses on multi-face forgery detection and segmentation in-the-wild, with 115,325 images, 334,136 human faces, and face-wise annotations such as bounding boxes, segmentation masks, forgery boundaries, and landmarks (Le et al., 2021). By contrast, OpenFake is not primarily a face-localization resource; it is a large-scale benchmark for politically salient synthetic-image authenticity, with prompt-conditioned generation and a broad scene taxonomy.
It also differs from OmniFake, introduced with OmniDFA, which is a class-aware synthetic image dataset containing 1.17M images from 45 distinct generative models and is constructed to support open-set and few-shot attribution rather than primarily adversarially refreshed authenticity detection (Wu et al., 30 Sep 2025). OpenFake’s design is closer to a continuously extensible detection benchmark, whereas OmniFake emphasizes structurally distinct synthetic classes for attribution.
At the platform level, OpenFake is complementary to DeepFake-O-Meter v2.0, an online service that integrates 6 image, 6 video, and 5 audio detectors and provides secure user-facing evaluation across media types (Ju et al., 2024). DeepFake-O-Meter emphasizes detector integration, user interaction, and multimodal service deployment; OpenFake emphasizes large-scale data curation, politically focused synthetic generation, and crowdsourced adversarial expansion. This suggests that OpenFake is best understood not merely as another dataset, but as one component of a broader shift toward open, continuously updated, and deployment-oriented deepfake forensics infrastructure.
7. Significance, misconceptions, and research implications
OpenFake’s main significance lies in three linked design choices: scale, recency, and extensibility. The resource is described as the largest publicly available dataset with rich real and synthetic classes and prompt metadata, and it explicitly incorporates both open-source and proprietary generators. This matters because several recent works argue that detector failure increasingly stems from generalization gaps to unseen generators and distribution shifts rather than from insufficient performance on older benchmarks (Livernoche et al., 11 Sep 2025, Lee et al., 7 Jun 2026).
Three misconceptions are particularly important to avoid. First, OpenFake is not limited to face swaps or portraits; it includes a much broader visual misinformation distribution. Second, OpenFake is not a fixed benchmark snapshot; the Arena is designed so that successful adversarial submissions are reintroduced into the dataset. Third, OpenFake is not restricted to fully supervised training; its protocols explicitly include semi-supervised, zero-shot, and transfer settings (Livernoche et al., 11 Sep 2025).
The broader research implication is that OpenFake reframes deepfake detection as a continually moving target. Later work on OpenFake has emphasized either training-free frequency probing (Sonoda et al., 23 Mar 2026) or frozen-encoder linear probing with curated subsets (Lee et al., 7 Jun 2026), indicating that the benchmark now supports competing views of robustness: one emphasizing adaptation through continual adversarial refresh, the other emphasizing strong general-purpose representations and careful curation. Taken together, these uses position OpenFake as a benchmark for studying not just whether a detector works, but how it remains effective as generators, prompts, and deployment conditions change.