UniGenBench++: Unified T2I Evaluation Benchmark

• UniGenBench++ is a unified semantic evaluation benchmark that assesses text-to-image models across hierarchical themes and subthemes.
• It leverages a bilingual, multi-length prompt corpus with detailed test points to evaluate style, world knowledge, and logical reasoning.
• The evaluation pipeline combines automated MLLM-based and offline methods to provide robust diagnostics for both proprietary and open-source models.

UniGenBench++ is a unified semantic evaluation benchmark targeting text-to-image (T2I) generation systems. Developed to address the limitations of prior benchmarks—specifically the lack of prompt diversity, insufficient multilingual coverage, and lack of fine-grained semantic evaluation—UniGenBench++ provides a systematically hierarchical, multilingual, and multi-length corpus of prompts and a comprehensive, fine-grained evaluation protocol. The benchmark is designed to assess the semantic consistency and generalization of T2I models across real-world and imaginative scenarios, leveraging both automated multi-modal LLM (MLLM)-based and efficient offline evaluation pipelines.

1. Hierarchical Benchmark Structure

UniGenBench++ organizes its evaluation corpus as a hierarchy, with 600 prompts distributed across two axes:

• Primary Themes (5 total): These are high-level categories encompassing broad domains such as Creative Divergence, Art, Illustration, Film/Story, and Design.
• Subthemes (20 in total): Each theme is further subdivided; for example, the Design theme covers advertising graphics, spatial/game/UX design, poster design, and logo/icon creation.

Each prompt is meticulously curated to maximize coverage and diagnostic efficiency. Instead of amassing thousands of shallow, non-overlapping prompts, each UniGenBench++ prompt is engineered to probe several semantically fine-grained "test points," targeting specific dimensions and sub-dimensions relevant to semantic image fidelity. This structure, coupling thematic breadth with analytic depth, enables effective and efficient evaluation with a tractable number of prompts.

2. Multi-Dimensional Semantic Evaluation Criteria

The benchmark defines a set of 10 primary evaluation dimensions, each decomposed into relevant subcriteria, as detailed in the following table:

Primary Dimension	Subcriteria (Examples)
Style	Visual style, artistic qualities
World Knowledge	Factual accuracy, physical and cultural correctness
Attribute	Quantity, Expression, Material, Color, Shape, Size
Compound	Imagination, Feature Matching
Action	Contact Interaction, Non-contact, Hand, Full-body, State, Animal Actions
Entity Layout	2D Space, 3D Space
Relationship	Composition, Similarity, Comparison, Inclusion
Logical Reasoning	Causality, contrast, complex semantic relations
Grammar	Pronoun Reference, Consistency, Negation
Text Generation	Text accuracy and legibility in generated images

Each prompt is annotated with a set of fine-grained test points (e.g., verifying a specific object's color, attribute, spatial configuration, or logical relationship as described in the prompt). Evaluation is not a single holistic judgment but rather a set of binary (correct/incorrect) decisions at each test point, accompanied by natural-language explanations.

3. Multilingual and Prompt Length Variations

UniGenBench++ implements explicit multilinguality and input complexity controls:

• Bilingual Support: Every prompt exists in both English and Chinese versions, ensuring the benchmark can assess a T2I model's invariance and robustness across language boundaries.
• Prompt Length Variation: Both short and long prompt forms are included. Short forms deliver instructions concisely; long forms, generated via MLLM-driven expansion, add detailed attribute and scene descriptions while preserving semantic intent. This dichotomy validates model performance under both terse and information-rich conditions. Empirical results indicate long prompts systematically introduce more attribute-related test points, increasing fine-grained evaluation rigor.

4. Automated Evaluation Pipeline and Offline Model

Prompt construction and evaluation leverage Gemini-2.5-Pro, a proprietary multi-modal LLM. The workflow comprises:

• Prompt Generation: Short prompts, parameterized by (theme, subject category, test points), are created in English and Chinese. Long prompts are automatically synthesized by expanding short prompts via MLLM-based rewriting, with corresponding test point realignment.
  • LaTeX notation: $$(p^{en}, p^{zh}, \{d_1, \dots, d_k\}) \sim \mathrm{MLLM}_{gen}(t, s, \{c_1, \dots, c_k\})$$; prompt expansion $\tilde{p} \sim \mathrm{MLLM}_{expand}(p \mid r)$.
• Evaluation: For each prompt–image pair, Gemini-2.5-Pro receives the prompt, test point descriptions, and generated image, sequentially rendering binary correctness judgments and explanations for each test point.
• Score Aggregation: Sub-dimension scores are computed as:

  $$R_c = \frac{\sum_{i,j} \mathbb{1}\{d_{ij} \in c \land r_{ij} = 1\}}{\sum_{i,j} \mathbb{1}\{d_{ij} \in c\}}$$

  with primary dimension scores as averages over constituent sub-dimensions, yielding both granular and aggregate system diagnostics.

To facilitate cost-effective and reproducible evaluation at scale, an offline evaluation model is trained via supervised fine-tuning using Gemini-2.5-Pro’s binary/explanation outputs as targets, with language modeling loss:

$$\mathcal{L}(\theta) = - \sum_t \log P_{\theta}\left( y^{(t)} \mid y^{(<t)}, x_i, p_i, \{d_i\} \right)$$

This offline evaluator supports robust local and community benchmarking.

5. Model Benchmarking Methodology

UniGenBench++ was deployed to evaluate both proprietary (closed-source) and open-source T2I models, encompassing GPT-4o, Imagen-4.0 (multiple variants), Seedream-3.0/4.0, Nano Banana (closed-source), and Qwen-Image, Hunyuan-Image-2.1, Lumina-DiMOO (open-source), among others. Each model was systematically tested on all combinations of prompt language (English, Chinese) and prompt length (short, long).

This controlled protocol enables detection of model sensitivity to both linguistic and descriptive complexity, and exposes variability in semantic fidelity, style adherence, attribute depiction, logical reasoning, and grammatical accuracy across the heterogeneous landscape of current-generation T2I systems.

6. Empirical Insights from Benchmarking

Systematic results produced with UniGenBench++ reveal several salient trends:

• Closed-source models, such as GPT-4o and Imagen-4.0-Ultra, exhibit high scores for style, world knowledge, and overall semantic alignment. However, they manifest notable weaknesses in logical reasoning, especially under prompts with complex causal or relational structures.
• Open-source models demonstrate strengths in world knowledge and attribute synthesis, but consistently underperform relative to closed-source models in grammatical faithfulness (pronoun resolution, inter-prompt consistency, negation) and advanced logical/relation reasoning tasks.
• Model performance is sensitive to both language and prompt length. Some models display marked performance differentials between short/long prompts; in certain cases, the introduction of more detailed prompt content in the long form exposes additional weaknesses or strengths not evident in the short form alone.
• The benchmark’s sub-dimension breakdown allows researchers to target specific weaknesses, such as action rendering or complex semantic relationship depiction, for subsequent model improvement.

7. Synthesis and Outlook

UniGenBench++ constitutes a significant advance in semantic evaluation for text-to-image generation. Its hierarchical prompt structure, multidimensional and fine-grained evaluation protocol, bilingual and multi-length support, and efficient—yet rigorous—MLLM-based and offline pipelines collectively address major gaps in prior T2I evaluation methodologies. The benchmark’s analytic granularity and comprehensive coverage afford developers, researchers, and practitioners a powerful diagnostic framework for both cross-model and cross-linguistic assessment, directly supporting the principled advancement of T2I generative modeling.

A plausible implication is that future benchmarks may further extend this paradigm by incorporating additional languages, more intricate logical constructs, or richer real-world scenario diversity, continuing the trajectory of increasing robustness and diagnostic utility in T2I evaluation.