OmniGen Benchmark: Two-Axis Evaluation

• OmniGen Benchmark is a held-out evaluation suite for synthetic image detection that assesses detector performance across both unseen generators and semantic domains.
• It combines diverse synthetic images from 12 modern generators with balanced real images, reporting metrics like Accuracy, Average Precision, Mean, and Standard Deviation.
• The benchmark stresses detector robustness through perturbations such as JPEG compression and Gaussian blur, highlighting stability and brittleness in detection.

The OmniGen Benchmark is a held-out evaluation benchmark for synthetic image detection introduced in the context of the two-axis generalization framework of "Redefining Generalization in Visual Domains: A Two-Axis Framework for Fake Image Detection with FusionDetect" (Amanzadi et al., 7 Oct 2025). It is designed to evaluate whether a detector can generalize simultaneously across unseen generators and across semantic domains, rather than only across generators. In that formulation, OmniGen combines synthetic images from 12 state-of-the-art generators with semantically diverse prompts and balanced real-image comparisons from Unsplash, and reports Accuracy, Average Precision, and stability statistics such as Mean and Standard Deviation across generators (Amanzadi et al., 7 Oct 2025).

1. Terminology and scope

In the literature, the exact phrase “OmniGen Benchmark” most specifically refers to the fake-image-detection benchmark introduced alongside FusionDetect (Amanzadi et al., 7 Oct 2025). Its purpose is detector evaluation under realistic shifts in both generator identity and visual semantics.

This usage is distinct from several similarly named artifacts. OmniGenBench denotes a benchmark for omnipotent multimodal image generation across 57 sub-tasks and six capability dimensions (Wang et al., 24 May 2025). In genomics, OmniGenBench denotes a standardized or modular benchmarking framework for genomic foundation models, integrating multiple benchmark suites, datasets, and automated evaluation pipelines (Yang et al., 2024, Yang et al., 20 May 2025). By contrast, the image-generation paper "OmniGen: Unified Image Generation" states that it does not introduce a single named suite called “OmniGen Benchmark” and instead evaluates on established public benchmarks plus targeted controllability tests (Xiao et al., 2024). Similarly, "OmniGen-AR: AutoRegressive Any-to-Image Generation" states that it does not define or introduce an “OmniGen Benchmark” and evaluates on existing public benchmark suites such as GenEval and VBench (Wang et al., 8 Jun 2026).

A further nearby usage appears in autonomous driving. "OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving" defines an evaluation suite for unified multimodal sensor generation on nuScenes, covering reconstruction, multimodal generation, controllability, and downstream utility; the paper’s details describe this evaluation suite as the OmniGen benchmark for that setting (Tang et al., 16 Dec 2025). This suggests that the term is context-dependent and should be disambiguated by domain.

2. Two-axis generalization framework

The conceptual core of the benchmark is the claim that fake-image detection should be treated as a two-axis generalization problem. One axis is cross-generator generalization, where detectors are tested on generators not seen during training. The other is cross-domain generalization, where detectors are tested on disjoint semantic domains, including shifts in content, styles, and contexts (Amanzadi et al., 7 Oct 2025).

The paper formalizes this as follows:

$$G_{train} \cap G_{test}=\emptyset \;\; \text{and} \;\; S_{train} \cap S_{test} = \emptyset$$

Here, $G$ denotes the set of generators and $S$ denotes semantic domains. The benchmark therefore evaluates a detector trained on one distribution of generators and domains and tested on a disjoint distribution along both axes (Amanzadi et al., 7 Oct 2025).

The reported evaluation measures are Accuracy (Acc) and Average Precision (AP), together with Mean and Standard Deviation (STD) across datasets or generators. The explicit role of STD is to quantify generalization consistency; a lower STD indicates more stable behavior under distribution shift (Amanzadi et al., 7 Oct 2025).

In the same framework, the proposed detector FusionDetect is trained with binary cross-entropy:

$$L(\theta) = - \frac{1}{N} \sum_{i=1}^{N} [y_i \log(p_i) + (1 - y_i) \log(1 - p_i)]$$

where $y_i \in \{0,1\}$ is the ground-truth label and $p_i = \sigma(f_{\theta}(z_{f,i}))$ is the predicted probability (Amanzadi et al., 7 Oct 2025). Although this loss belongs to the detector rather than the benchmark itself, it is part of the benchmarked experimental setup in the original paper.

3. Dataset construction and composition

OmniGen is presented as a benchmark intentionally constructed to stress both axes of generalization by combining modern generators with high semantic diversity (Amanzadi et al., 7 Oct 2025). Its design principles are to include state-of-the-art, widely used, and recently released generators, maximize semantic variety through a structured randomized prompt template, and provide per-generator evaluation so that stability or brittleness on specific generators is visible (Amanzadi et al., 7 Oct 2025).

The synthetic component contains 11,550 synthetic images in total, all at 1024×1024 resolution (Amanzadi et al., 7 Oct 2025). The included generators are divided into three groups: closed-source, open-source, and community fine-tuned (Amanzadi et al., 7 Oct 2025).

Generator group	Generators	Images
Closed-Source	GPT-4o, Imagen 4, Imagen 4 Ultra, MidJourney v7	550; 1000 each except GPT-4o
Open-Source	FLUX 1, Kandinsky 3, PixArt-$\delta$, SD3.5-medium, HiDream-I1, CogView4-6B	1000 each
Community Fine-tuned	Juggernaut v11, Dreamshaper	1000 each

The benchmark prioritizes semantic diversity rather than a fixed category taxonomy. Prompts draw from “over 400 different subjects” and use the following structured, randomized template:

“A richly detailed, high-resolution and photorealistic image depicting: {subject\ during the {time}. The scene includes {setting}, {visual}, and lifelike rendering. The image style resembles {style}. Use {light}."}

This template varies subjects, settings, styles, times, and lighting conditions and is intended to induce domain shifts without relying on fixed class labels (Amanzadi et al., 7 Oct 2025).

For real images, Unsplash is the source. The main text states that, for each generator, fake images are evaluated against a set of 1000 real images from Unsplash; in practice, per-generator evaluations are balanced against matched real samples, such as 1000 real vs. 1000 fake or 550 real vs. 550 fake for GPT-4o (Amanzadi et al., 7 Oct 2025). Labels are binary—real versus synthetic—and synthetic samples also retain their generator of origin (Amanzadi et al., 7 Oct 2025). The paper specifies resolution but does not specify file formats (Amanzadi et al., 7 Oct 2025).

OmniGen is used as a test-only evaluation benchmark. FusionDetect itself is trained on a separate 60,000-image custom dataset built from GenImage, ImagiNet, and prompts derived from Chameleon, using only SD1.4 and SD2.1 for synthetic training images to isolate cross-generator generalization (Amanzadi et al., 7 Oct 2025).

4. Evaluation protocol and robustness testing

The central protocol is a cross-generator test in which the detector is trained on SD1.4 and SD2.1 and then evaluated on the 12 OmniGen generators (Amanzadi et al., 7 Oct 2025). Performance is reported per generator, and then summarized using Mean and STD. This generator-wise reporting is a deliberate benchmark feature: it exposes stable versus brittle behavior that could be obscured by a single pooled score (Amanzadi et al., 7 Oct 2025).

The benchmark is paired with complementary cross-domain evaluations on GenImage, ImagiNet, and Chameleon, which represent distinct semantic distributions. The paper states that these domain shifts are demonstrated in t-SNE projections, and that Mean and STD across these datasets quantify cross-semantic generalization (Amanzadi et al., 7 Oct 2025).

Robustness testing is also part of the benchmarked setting. The reported perturbations are:

• JPEG compression: QF=95, QF=75, QF=50
• Gaussian blur: $\sigma=1.0$, $\sigma=2.0$, $\sigma=3.0$

For each perturbation setting, the benchmark reports Acc and AP (Amanzadi et al., 7 Oct 2025). This matters because many artifact-centric detectors degrade sharply under common image-processing operations; the benchmark explicitly measures whether a detector remains effective when those cues are weakened (Amanzadi et al., 7 Oct 2025).

The authors recommend a broad baseline suite for comparison on OmniGen, including DIF, UNIFD, DNF, LASTED, BiLoRA, AIDE, SSP, and NPR (Amanzadi et al., 7 Oct 2025). This makes OmniGen not only a dataset but also a comparative evaluation protocol with reference detectors and stability-oriented reporting.

5. Reported findings on OmniGen

On OmniGen, FusionDetect is reported to achieve the highest mean accuracy and lowest STD across the 12 generators, with Mean Acc 97.38% and STD (Acc) 3.26 (Amanzadi et al., 7 Oct 2025). The paper states that this is a +4.48% increase in mean accuracy over the second-best baseline, SSP, which has Mean Acc=92.90% (Amanzadi et al., 7 Oct 2025).

Per-generator accuracies highlight that OmniGen contains both easy and difficult cases. Reported FusionDetect accuracies include 99.3 on Kandinsky 3, 99.6 on CogView4-6B, 98.5 on FLUX 1 dev, 99.2 on Juggernaut v11, 98.4 on Dreamshaper, 98.2 on SD3.5-medium, 97.3 on GPT-4o, 97.5 on Imagen 4, 96.4 on Imagen 4 Ultra, 99.0 on PixArt-$G$0, and 97.9 on HiDream-I1; MidJourney v7 is reported as a harder case at 87.5 (Amanzadi et al., 7 Oct 2025). This generator-level spread is precisely the kind of brittleness the benchmark is designed to expose.

The appendix results further report mean real accuracy ≈ 99.13% and mean fake accuracy ≈ 95.58% for FusionDetect on OmniGen (Amanzadi et al., 7 Oct 2025). This suggests that the benchmark is not only measuring overall discrimination but also class-wise balance under substantial generator diversity.

Under perturbations, FusionDetect is reported to exhibit minimal degradation. The benchmark reports 80.91 / 90.43 (Acc / AP, %) with no degradation, 80.94 / 91.04 at JPEG QF=95, 81.03 / 91.10 at QF=75, 80.45 / 91.08 at QF=50, 80.92 / 92.64 with blur $G$1, 79.32 / 92.34 with blur $G$2, and 78.87 / 92.00 with blur $G$3 (Amanzadi et al., 7 Oct 2025). The paper contrasts this with artifact-focused baselines such as SSP, NPR, and DNF, which show large accuracy drops under stronger JPEG compression (Amanzadi et al., 7 Oct 2025).

The authors also argue that OmniGen reveals limitations of detectors that appear strong on older standards but fail on newer generators or more diverse semantics. The paper attributes this to the benchmark’s combination of modern generators, semantic diversity, and stability-focused reporting through Mean and STD (Amanzadi et al., 7 Oct 2025).

6. Limitations, future extensions, and naming ambiguity

The paper identifies several limitations of OmniGen itself. Although it spans 12 SOTA generators and enforces semantic variety through randomized prompts, the generative and semantic landscapes continue to evolve rapidly, so both generator coverage and domain breadth can be expanded (Amanzadi et al., 7 Oct 2025). The benchmark focuses on photographic and stylistic diversity, and task-specific forensic subdomains may require additional curated subsets (Amanzadi et al., 7 Oct 2025). The paper also states that OmniGen is open-source, but specific licensing terms are not detailed (Amanzadi et al., 7 Oct 2025).

The proposed future directions are to extend OmniGen with additional generators and richer domain taxonomies, to further improve fusion-based architectures, and to encourage adoption of the two-axis perspective as a general evaluation method for universal image detectors (Amanzadi et al., 7 Oct 2025). These proposals are consistent with the benchmark’s role as a stress test for real-world detector robustness rather than a closed, fixed leaderboard.

A recurrent source of confusion is the overlap between OmniGen Benchmark, OmniGenBench, and evaluation suites associated with models named OmniGen. In multimodal image generation, OmniGenBench is a separate benchmark spanning 57 diverse sub-tasks with a dual-mode evaluation protocol using off-the-shelf visual parsing tools and an LLM-based judger (Wang et al., 24 May 2025). In autonomous driving, the OmniGen paper defines a benchmarked evaluation suite for unified multimodal sensor generation on nuScenes, including reconstruction, multimodal generation, controllability, and downstream perception and planning (Tang et al., 16 Dec 2025). In unified image generation and autoregressive any-to-image generation, the papers explicitly state that they do not define a single named “OmniGen Benchmark”, instead relying on established benchmarks such as GenEval, Emu-Edit, DreamBench, ADE20K, MultiGen-Depth-Eval, and VBench (Xiao et al., 2024, Wang et al., 8 Jun 2026). In genomics, OmniGenBench refers to a distinct benchmarking framework for genomic foundation models, not to image detection or image generation (Yang et al., 2024, Yang et al., 20 May 2025).

Within this broader naming landscape, the specific contribution of the OmniGen Benchmark is the formalization of synthetic-image detection as a two-axis generalization problem and the provision of a benchmark that operationalizes that view through modern generators, diverse semantics, per-generator reporting, and perturbation-aware evaluation (Amanzadi et al., 7 Oct 2025).