GenEval 2: Benchmark for Compositional T2I Models

Updated 19 December 2025

The paper introduces GenEval 2, a novel evaluation benchmark that uses an atom-level VQA-based judge to assess compositional accuracy in T2I models.
It expands concept coverage with 800 prompts featuring 40 objects, 18 attributes, and 9 relations to test models beyond basic skills.
It employs advanced protocols like Soft-TIFA and multi-generation metrics that improve human alignment and diagnose compositional gaps.

GenEval 2 is a compositional, atomized evaluation benchmark for text-to-image (T2I) models, designed to address saturation and alignment drift of prior benchmarks in the face of accelerating model capabilities. It expands concept coverage and compositional difficulty, utilizes a robust atom-level VQA-based judge (Soft-TIFA), and provides fine-grained diagnostic signals for compositional image generation. Empirical analyses demonstrate significant advances in benchmark design, measurement of human alignment, and robustness against semantic drift in new model regimes (Kamath et al., 18 Dec 2025).

1. Motivation and Benchmark Drift

Recent advances in T2I models, such as Stable Diffusion 3, Gemini 2.5, and Qwen-Image, have outpaced the original GenEval 1 benchmark, rendering it insufficiently challenging and increasingly misaligned with human preferences. “Benchmark drift” refers to the breakdown of benchmark validity as static judge models fail to keep pace with distributional change in generated outputs. While GenEval 1 initially agreed with human judgments on basic skills (≈83%), large-scale studies reveal that state-of-the-art models now score up to 17.7% higher with human annotators versus GenEval’s automated detector score (e.g., Gemini 2.5: GenEval 75.4% vs. human 93.1%). The saturation and misalignment signal the end of the benchmark’s discriminative power, demanding expanded scope and more resilient evaluation protocols (Kamath et al., 18 Dec 2025).

Model	GenEval 1 Score	Human Score	Absolute Deviation
SD 2.1	44.8%	42.5%	–2.4%
SD XL	53.5%	56.6%	+3.1%
Gemini 2.5	75.4%	93.1%	+17.7%

2. Task Design: Vocabulary, Templates, and Compositionality

GenEval 2 generalizes the previous benchmark’s focus on “basic” skills, expanding the vocabulary and introducing graded compositionality across prompts. The benchmark comprises:

Objects: 40 types (balanced between animate/inanimate; 20 COCO classes + 20 new),
Attributes: 18 (colors, materials, patterns),
Relations: 9 (six spatial prepositions, three transitive verbs),
Counts: Ranging from 2 to 7, and constructs prompts as compositions of “atoms.” Atomicity $N$ enumerates the total number of concept instances (objects, attributes, relations, counts) appearing in a prompt. GenEval 2 provides 100 prompts at each atomicity $N = 3..10$ , totaling 800 prompts. Each template prompt is optionally “rewritten” into longer, context-rich forms via GPT-4o, maintaining relevance for real-world T2I pipelines (Kamath et al., 18 Dec 2025).

3. Evaluation Protocols: Soft-TIFA Judge and Metric Aggregation

GenEval 2 is evaluated with Soft-TIFA, an atom-level, soft-probabilistic judge built atop open-source VQA models (Qwen3-VL 8B). For each prompt, its constituent atoms (concept instances) generate templated VQA questions; the judge computes per-atom probabilities $P_M(a_i \mid \text{image}, q_i)$ . Two aggregation metrics are used:

Atom-mean (AM): $\frac{1}{N} \sum_{i=1}^N P_M(a_i)$ ,
Atom-geo (GM): $\left( \prod_{i=1}^N P_M(a_i) \right)^{1/N}$ , with the GM score acting as a soft logical “all-must-be-correct” metric. Soft-TIFA shows enhanced alignment with human ratings (AUROC 94.5%), outperforming VQAScore (92.4%) and TIFA (91.6%), and is robust to VQA backbone drift (Kamath et al., 18 Dec 2025).

Metric/Backbone	AUROC vs. Human
Soft-TIFA GM (Qwen3-VL)	94.5%
VQAScore (Qwen3-VL)	92.4%
TIFA (Qwen3-VL)	91.6%

4. Empirical Results and Compositional Analysis

Human-judged and Soft-TIFA-scored results reveal persistent compositional gaps even among top T2I models. Atom-level accuracy for best models approaches 85%, but prompt-level accuracy under rewritten conditions remains below 35.8%. Accuracy substantially drops with increasing atomicity—models maintain $\sim 60\%$ correctness at $N=3$ and collapse to near zero at $N=9$ -$10$. Per-skill breakdowns highlight nearly complete mastery of objects ( $\sim$ 99%) and attributes ( $\sim$ 90%), but significant errors for counts ( $\sim$ 68%), spatial relations ( $\sim$ 65%), and verb relations ( $\sim$ 63%) (Kamath et al., 18 Dec 2025).

Model	Atom (%)	Prompt (%)	Notable Weaknesses
Gemini 2.5 I	84.4	31.0	High atomicity, relations, counts
Qwen-Image	82.0	26.8	Multi-object combinations
SD 3.5-large	67.8	14.3	Spatial reasoning, attribute binding

5. Multi-Generation GenEval and Semantic Drift Assessment

The Multi-Generation GenEval (MGG) metric [Editor's term] extends standard GenEval to a cyclic evaluation regime, alternating image-to-text and text-to-image via unified models across $G$ generations. At each step, GenEval compliance is scored, and the average across generations quantifies compound semantic loss. MGG reveals cross-modal compounding errors that are not diagnosed by single-pass metrics and discriminates stability among unified vision–LLMs (Mollah et al., 4 Sep 2025). In practice, MGG exposes failures on compositional concepts that become amplified through multiple cross-modal alternations.

MGG score (for $G$ generations): $\mathrm{MGG} = \frac{1}{G} \sum_{g=1}^{G} A(g)$ where $A(g)$ is the standard GenEval accuracy at generation $g$ .

6. Evolving Verification Protocols and Reward Models

Recent work associates substantial GenEval gains with advanced reward modeling and reasoning strategies. Chain-of-Thought (CoT) verification, Direct Preference Optimization (DPO), and the Potential Assessment Reward Model (PARM/PARM++) substantially strengthen GenEval results. PARM adaptively scores intermediate decoding steps for potential success, while PARM++ introduces a reflection-driven self-correction loop. These mechanisms, evaluated on GenEval, yield state-of-the-art performance: Show-o baseline (53%), Stable Diffusion 3 (62%), Show-o + PARM + It DPO + PARM (77%) (Guo et al., 23 Jan 2025). Gains are most pronounced in challenging compositional subtasks (position, attribute binding, multi-object scenes).

Model Setup	Two-Obj	Counting	Position	Attr. Bind.	Overall
Show-o baseline	0.52	0.49	0.11	0.28	0.53
Stable Diffusion 3	0.74	0.63	0.34	0.36	0.62
Show-o + PARM + It DPO + PARM	0.86	0.67	0.66	0.64	0.77

7. Implications, Limitations, and Future Directions

GenEval 2 demonstrates that compositional understanding in T2I models remains unsolved, with strong atom-level scores but poor high-atomicity prompt accuracy. Atom-based prompt construction, robust VQA-based judges, and continuous audit of alignment are necessary to maintain benchmark relevance and resist saturation. Authors recommend decomposable, compositional benchmark design, use of open VLMs (such as Qwen3-VL), and continual audit and refresh cycles. Noted challenges include richer spatial/physical relations, multi-step temporal dynamics, multi-sentence/contextual prompts, and adversarial or real-world scenarios. Soft-TIFA and MGG provide forward-looking frameworks for robust evaluation as model regimes continue to shift (Kamath et al., 18 Dec 2025, Mollah et al., 4 Sep 2025, Guo et al., 23 Jan 2025).

Summary Table: Key Features of GenEval 2

Dimension	Details	Significance
Coverage	800 prompts, N=3–10 atoms, objects/attributes/relations	Measures scalability/compositionality
Judge Model	Soft-TIFA (GM/AM), Qwen3-VL backbone	Strong human alignment, reduced drift
Metrics	Atom accuracy, prompt accuracy, MGG	Fine-grained, stability over multi-modal cycles
Current SOTA	Show-o + PARM/DPO/verification: 77%; Gemini 2.5: 31% prompt-level	High variance, open compositional gaps