---
title: 'OmniGen Benchmark: Two-Axis Evaluation'
url: https://www.emergentmind.com/topics/omnigen-benchmark
type: topic
---

# OmniGen Benchmark: Two-Axis Evaluation

The **OmniGen Benchmark** is a held-out evaluation benchmark for synthetic image detection introduced in the context of the two-axis generalization framework of "Redefining Generalization in Visual Domains: A Two-Axis Framework for Fake Image Detection with FusionDetect" [2510.05740]. It is designed to evaluate whether a detector can generalize simultaneously across unseen generators and across semantic domains, rather than only across generators. In that formulation, OmniGen combines synthetic images from 12 state-of-the-art generators with semantically diverse prompts and balanced real-image comparisons from Unsplash, and reports Accuracy, Average Precision, and stability statistics such as Mean and Standard Deviation across generators [2510.05740].

## 1. Terminology and scope

In the literature, the exact phrase **“OmniGen Benchmark”** most specifically refers to the fake-image-detection benchmark introduced alongside FusionDetect [2510.05740]. Its purpose is detector evaluation under realistic shifts in both generator identity and visual semantics.

This usage is distinct from several similarly named artifacts. **OmniGenBench** denotes a benchmark for omnipotent multimodal image generation across 57 sub-tasks and six capability dimensions [2505.18775]. In genomics, **OmniGenBench** denotes a standardized or modular benchmarking framework for genomic foundation models, integrating multiple benchmark suites, datasets, and automated evaluation pipelines [2410.01784]; [2505.14402]. By contrast, the image-generation paper "OmniGen: Unified Image Generation" states that it **does not introduce a single named suite called “OmniGen Benchmark”** and instead evaluates on established public benchmarks plus targeted controllability tests [2409.11340]. Similarly, "OmniGen-AR: AutoRegressive Any-to-Image Generation" states that it **does not define or introduce an “OmniGen Benchmark”** and evaluates on existing public benchmark suites such as GenEval and VBench [2606.09156].

A further nearby usage appears in autonomous driving. "OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving" defines an evaluation suite for unified multimodal sensor generation on nuScenes, covering reconstruction, multimodal generation, controllability, and downstream utility; the paper’s details describe this evaluation suite as the **OmniGen benchmark** for that setting [2512.14225]. This suggests that the term is context-dependent and should be disambiguated by domain.

## 2. Two-axis generalization framework

The conceptual core of the benchmark is the claim that fake-image detection should be treated as a **two-axis** generalization problem. One axis is **cross-generator generalization**, where detectors are tested on generators not seen during training. The other is **cross-domain generalization**, where detectors are tested on disjoint semantic domains, including shifts in content, styles, and contexts [2510.05740].

The paper formalizes this as follows:

$$
G_{train} \cap G_{test}=\emptyset \;\; \text{and} \;\; S_{train} \cap S_{test} = \emptyset
$$

Here, $G$ denotes the set of generators and $S$ denotes semantic domains. The benchmark therefore evaluates a detector trained on one distribution of generators and domains and tested on a disjoint distribution along both axes [2510.05740].

The reported evaluation measures are **Accuracy (Acc)** and **Average Precision (AP)**, together with **Mean** and **Standard Deviation (STD)** across datasets or generators. The explicit role of STD is to quantify generalization consistency; a lower STD indicates more stable behavior under distribution shift [2510.05740].

In the same framework, the proposed detector FusionDetect is trained with binary cross-entropy:

$$
L(\theta) = - \frac{1}{N} \sum_{i=1}^{N} [y_i \log(p_i) + (1 - y_i) \log(1 - p_i)]
$$

where $y_i \in \{0,1\}$ is the ground-truth label and $p_i = \sigma(f_{\theta}(z_{f,i}))$ is the predicted probability [2510.05740]. Although this loss belongs to the detector rather than the benchmark itself, it is part of the benchmarked experimental setup in the original paper.

## 3. Dataset construction and composition

OmniGen is presented as a benchmark intentionally constructed to stress both axes of generalization by combining modern generators with high semantic diversity [2510.05740]. Its design principles are to include **state-of-the-art, widely used, and recently released generators**, maximize semantic variety through a structured randomized prompt template, and provide **per-generator evaluation** so that stability or brittleness on specific generators is visible [2510.05740].

The synthetic component contains **11,550 synthetic images** in total, all at **1024×1024** resolution [2510.05740]. The included generators are divided into three groups: closed-source, open-source, and community fine-tuned [2510.05740].

| Generator group | Generators | Images |
|---|---|---:|
| Closed-Source | GPT-4o, Imagen 4, Imagen 4 Ultra, MidJourney v7 | 550; 1000 each except GPT-4o |
| Open-Source | FLUX 1, Kandinsky 3, PixArt-$\delta$, SD3.5-medium, HiDream-I1, CogView4-6B | 1000 each |
| Community Fine-tuned | Juggernaut v11, Dreamshaper | 1000 each |

The benchmark prioritizes semantic diversity rather than a fixed category taxonomy. Prompts draw from **“over 400 different subjects”** and use the following structured, randomized template:

> “A richly detailed, high-resolution and photorealistic image depicting: \{subject\ during the \{time\}. The scene includes \{setting\}, \{visual\}, and lifelike rendering. The image style resembles \{style\}. Use \{light\}."}

This template varies subjects, settings, styles, times, and lighting conditions and is intended to induce domain shifts without relying on fixed class labels [2510.05740].

For real images, **Unsplash** is the source. The main text states that, for each generator, fake images are **evaluated against a set of 1000 real images from Unsplash**; in practice, per-generator evaluations are balanced against matched real samples, such as **1000 real vs. 1000 fake** or **550 real vs. 550 fake for GPT-4o** [2510.05740]. Labels are binary—real versus synthetic—and synthetic samples also retain their generator of origin [2510.05740]. The paper specifies resolution but **does not specify file formats** [2510.05740].

OmniGen is used as a **test-only** evaluation benchmark. FusionDetect itself is trained on a separate **60,000-image custom dataset** built from GenImage, ImagiNet, and prompts derived from Chameleon, using only **SD1.4** and **SD2.1** for synthetic training images to isolate cross-generator generalization [2510.05740].

## 4. Evaluation protocol and robustness testing

The central protocol is a cross-generator test in which the detector is trained on **SD1.4 and SD2.1** and then evaluated on the **12 OmniGen generators** [2510.05740]. Performance is reported per generator, and then summarized using **Mean** and **STD**. This generator-wise reporting is a deliberate benchmark feature: it exposes stable versus brittle behavior that could be obscured by a single pooled score [2510.05740].

The benchmark is paired with complementary cross-domain evaluations on **GenImage**, **ImagiNet**, and **Chameleon**, which represent distinct semantic distributions. The paper states that these domain shifts are demonstrated in t-SNE projections, and that Mean and STD across these datasets quantify cross-semantic generalization [2510.05740].

Robustness testing is also part of the benchmarked setting. The reported perturbations are:

- **JPEG compression**: QF=95, QF=75, QF=50
- **Gaussian blur**: $\sigma=1.0$, $\sigma=2.0$, $\sigma=3.0$

For each perturbation setting, the benchmark reports **Acc** and **AP** [2510.05740]. This matters because many artifact-centric detectors degrade sharply under common image-processing operations; the benchmark explicitly measures whether a detector remains effective when those cues are weakened [2510.05740].

The authors recommend a broad baseline suite for comparison on OmniGen, including **DIF, UNIFD, DNF, LASTED, BiLoRA, AIDE, SSP,** and **NPR** [2510.05740]. This makes OmniGen not only a dataset but also a comparative evaluation protocol with reference detectors and stability-oriented reporting.

## 5. Reported findings on OmniGen

On OmniGen, FusionDetect is reported to achieve the **highest mean accuracy** and **lowest STD** across the 12 generators, with **Mean Acc 97.38%** and **STD (Acc) 3.26** [2510.05740]. The paper states that this is a **+4.48%** increase in mean accuracy over the second-best baseline, **SSP**, which has **Mean Acc=92.90%** [2510.05740].

Per-generator accuracies highlight that OmniGen contains both easy and difficult cases. Reported FusionDetect accuracies include **99.3** on Kandinsky 3, **99.6** on CogView4-6B, **98.5** on FLUX 1 dev, **99.2** on Juggernaut v11, **98.4** on Dreamshaper, **98.2** on SD3.5-medium, **97.3** on GPT-4o, **97.5** on Imagen 4, **96.4** on Imagen 4 Ultra, **99.0** on PixArt-$\delta$, and **97.9** on HiDream-I1; **MidJourney v7** is reported as a harder case at **87.5** [2510.05740]. This generator-level spread is precisely the kind of brittleness the benchmark is designed to expose.

The appendix results further report **mean real accuracy ≈ 99.13%** and **mean fake accuracy ≈ 95.58%** for FusionDetect on OmniGen [2510.05740]. This suggests that the benchmark is not only measuring overall discrimination but also class-wise balance under substantial generator diversity.

Under perturbations, FusionDetect is reported to exhibit minimal degradation. The benchmark reports **80.91 / 90.43 (Acc / AP, %)** with no degradation, **80.94 / 91.04** at JPEG QF=95, **81.03 / 91.10** at QF=75, **80.45 / 91.08** at QF=50, **80.92 / 92.64** with blur $\sigma=1.0$, **79.32 / 92.34** with blur $\sigma=2.0$, and **78.87 / 92.00** with blur $\sigma=3.0$ [2510.05740]. The paper contrasts this with artifact-focused baselines such as SSP, NPR, and DNF, which show large accuracy drops under stronger JPEG compression [2510.05740].

The authors also argue that OmniGen reveals limitations of detectors that appear strong on older standards but fail on newer generators or more diverse semantics. The paper attributes this to the benchmark’s combination of modern generators, semantic diversity, and stability-focused reporting through Mean and STD [2510.05740].

## 6. Limitations, future extensions, and naming ambiguity

The paper identifies several limitations of OmniGen itself. Although it spans **12 SOTA generators** and enforces semantic variety through randomized prompts, the generative and semantic landscapes continue to evolve rapidly, so both generator coverage and domain breadth can be expanded [2510.05740]. The benchmark focuses on **photographic and stylistic diversity**, and task-specific forensic subdomains may require additional curated subsets [2510.05740]. The paper also states that OmniGen is open-source, but **specific licensing terms are not detailed** [2510.05740].

The proposed future directions are to extend OmniGen with **additional generators** and **richer domain taxonomies**, to further improve fusion-based architectures, and to encourage adoption of the two-axis perspective as a general evaluation method for universal image detectors [2510.05740]. These proposals are consistent with the benchmark’s role as a stress test for real-world detector robustness rather than a closed, fixed leaderboard.

A recurrent source of confusion is the overlap between **OmniGen Benchmark**, **OmniGenBench**, and evaluation suites associated with models named **OmniGen**. In multimodal image generation, OmniGenBench is a separate benchmark spanning **57 diverse sub-tasks** with a dual-mode evaluation protocol using off-the-shelf visual parsing tools and an LLM-based judger [2505.18775]. In autonomous driving, the OmniGen paper defines a benchmarked evaluation suite for **unified multimodal sensor generation** on nuScenes, including reconstruction, multimodal generation, controllability, and downstream perception and planning [2512.14225]. In unified image generation and autoregressive any-to-image generation, the papers explicitly state that they do **not** define a single named **“OmniGen Benchmark”**, instead relying on established benchmarks such as GenEval, Emu-Edit, DreamBench, ADE20K, MultiGen-Depth-Eval, and VBench [2409.11340]; [2606.09156]. In genomics, OmniGenBench refers to a distinct benchmarking framework for genomic foundation models, not to image detection or image generation [2410.01784]; [2505.14402].

Within this broader naming landscape, the specific contribution of the **OmniGen Benchmark** is the formalization of synthetic-image detection as a **two-axis** generalization problem and the provision of a benchmark that operationalizes that view through modern generators, diverse semantics, per-generator reporting, and perturbation-aware evaluation [2510.05740].

Source: https://www.emergentmind.com/topics/omnigen-benchmark