Qwen-Image-Bench: Benchmark for Creative T2I Evaluation

• Qwen-Image-Bench is a benchmark for text-to-image evaluation that adds real-world fidelity and creative generation to traditional quality measures.
• It utilizes a three-level hierarchical taxonomy of 5 pillars, 23 sub-capabilities, and 56 rubrics to provide detailed, rubric-grounded diagnostics.
• The framework leverages Q-Judger, a unified judge model trained on 130K annotated image-prompt pairs, to offer actionable insights for production-level T2I development.

Searching arXiv for the benchmark and associated evaluation papers. Qwen-Image-Bench is a creator-centric benchmark for text-to-image (T2I) evaluation introduced to measure capabilities that extend beyond coarse prompt adherence and generic visual quality, particularly faithful real-world reconstruction and creative expression in professional creative workflows (Li et al., 27 May 2026). It organizes evaluation through a three-level hierarchical taxonomy spanning Quality, Aesthetics, Alignment, Real-world Fidelity, and Creative Generation, and uses a unified judge model, Q-Judger, to produce rubric-grounded diagnostics rather than a single opaque score (Li et al., 27 May 2026). In subsequent use, Qwen-Image-Bench also served as the principal automated benchmark for assessing Qwen-Image-2.0-RL, where improvements were reported both in aggregate benchmark score and in human-preference Elo ratings (Xu et al., 25 Jun 2026).

1. Origin, scope, and motivation

Qwen-Image-Bench was proposed in response to the claim that contemporary T2I systems have largely saturated conventional benchmarks centered on semantic correctness, CLIP-style alignment, and coarse visual plausibility, while remaining inadequate on capabilities that matter in authentic artistic practice (Li et al., 27 May 2026). The benchmark’s central premise is that professional creators require not only prompt following, but also faithful reconstruction of real-world detail and genuinely creative, stylistically coherent imagery (Li et al., 27 May 2026).

The benchmark therefore adds two application-driven dimensions—Real-world Fidelity and Creative Generation—alongside the traditional pillars of Quality, Aesthetics, and Text-Image Alignment (Li et al., 27 May 2026). The paper characterizes this shift as moving “from generation to creation”, emphasizing higher-level creative capabilities in addition to basic prompt adherence (Xu et al., 25 Jun 2026).

This design responds to several deficiencies identified in prior evaluation practice. Existing suites are described as relying heavily on CLIP scores or a single off-the-shelf multimodal LLM as judge, which leads to saturated score bands, benchmark drift, and poor localization of failure modes in high-value creative workflows (Li et al., 27 May 2026). Qwen-Image-Bench instead aims to provide a taxonomy that mirrors the staged reasoning of actual artistic workflows and a judging pipeline that is rubric-grounded and attributable (Li et al., 27 May 2026).

A plausible implication is that the benchmark is intended not only as a leaderboard instrument, but also as a diagnostic tool for production-level model development. That interpretation is consistent with the claim that it provides a trustworthy optimization signal for production-level T2I development (Li et al., 27 May 2026).

2. Hierarchical taxonomy and evaluation target

At the top level, Qwen-Image-Bench is structured around 5 first-level pillars: Quality, Aesthetics, Alignment, Real-world Fidelity, and Creative Generation (Li et al., 27 May 2026, Xu et al., 25 Jun 2026). The benchmark paper describes a three-level hierarchical taxonomy that decomposes evaluation into 23 mid-level sub-capabilities and 56 atomistic rubrics (Li et al., 27 May 2026). A related evaluation summary describes the same overall design as 5 first-level pillars, 16 second-level sub-pillars, and 56 third-level facets (Xu et al., 25 Jun 2026). Both descriptions agree on the role of the 56 fine-grained evaluation units.

The taxonomy is explicitly grounded in the staged reasoning inherent in professional artistic workflows, including ideation → styling → iterative refinement (Li et al., 27 May 2026). The design is top-down: prompts invoke a mix of third-level facets, these scores aggregate to second-level sub-capabilities, and then to first-level pillars (Li et al., 27 May 2026).

Examples given for the second and third levels illustrate the scope of the benchmark. Under Real-world Fidelity, the benchmark includes sub-capabilities such as Fairness, Safety & Compliance, World Knowledge (Li et al., 27 May 2026). Under Creative Generation, it includes Imagination, Text Rendering, Design Applications, Visual Storytelling (Li et al., 27 May 2026). At the rubric level, examples include Physical Logic, Color Harmony, Text Accuracy, Game Design, Contact Interaction, Temporal Characteristics, Font, and Cinematic Style (Li et al., 27 May 2026). The evaluation summary further characterizes third-level facets as including attributes such as Color Harmony, Shadow Realism, Facial Proportions, among others (Xu et al., 25 Jun 2026).

Level	Structure	Examples
Level 1	5 pillars	Quality; Aesthetics; Alignment; Real-world Fidelity; Creative Generation
Level 2	23 mid-level sub-capabilities	Fairness; Safety & Compliance; World Knowledge; Imagination; Text Rendering
Level 3	56 verifiable rubrics	Physical Logic; Color Harmony; Text Accuracy; Contact Interaction

This hierarchy is intended to localize strengths and weaknesses to identifiable sub-skills rather than collapse performance into a single undifferentiated number (Li et al., 27 May 2026). The benchmark paper argues that this is especially important for frontier systems, where the most consequential performance differences increasingly lie in application-driven abilities rather than in basic semantic alignment.

3. Prompt curation and dataset design

Qwen-Image-Bench uses 1,000 bilingual prompts in Chinese/English, curated through a four-stage expert-in-the-loop pipeline (Li et al., 27 May 2026). The prompts are designed so that each prompt jointly exercises multiple facets across pillars. One description states that prompts each jointly exercise 3–5 facets across pillars (Li et al., 27 May 2026), while the abstract states that each prompt jointly exercises more than four fine-grained facets across multiple pillars (Li et al., 27 May 2026). The benchmark also enforces broad coverage and balance across the taxonomy.

The first stage is facet-targeted sampling, where for each prompt $p$ a subset of third-level facets is sampled under a global balance constraint:

$C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$

The benchmark further ensures that most sampled facet sets span at least three pillars (Li et al., 27 May 2026).

The second stage is bilingual drafting, formalized as

$$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$

where each $d_i$ records how facet $c_i$ is explicitly realized (Li et al., 27 May 2026). The third stage is expert review and rewrite, in which professional artists gate each draft and rewrite or discard it until every selected facet is genuinely exercised (Li et al., 27 May 2026). The fourth stage is length-variant expansion, which introduces short and long prompt variants through an LLM-based expansion and re-alignment process (Li et al., 27 May 2026).

The resulting prompt set contains 500 short + 500 long prompts (Li et al., 27 May 2026). The benchmark paper defines the split using 70 Chinese / 235 English characters as the threshold for short versus long prompts (Li et al., 27 May 2026). A later evaluation report describes the test set more generally as a curated collection of several thousand artist-written prompts spanning product design, portrait, 3D modeling, text rendering, cartoon, photorealism, art, and other genres (Xu et al., 25 Jun 2026). This suggests that Qwen-Image-Bench has also been used in an expanded evaluation configuration beyond the original 1,000-prompt benchmark definition.

A plausible implication is that the benchmark’s bilingual and length-variant construction is meant to reduce overfitting to narrow prompt styles and to test robustness under prompt compression and elaboration. That interpretation is consistent with the explicit balancing and re-alignment procedures described in the construction pipeline (Li et al., 27 May 2026).

4. Q-Judger: judge model, annotation protocol, and supervision

The benchmark is operationalized through Q-Judger, a unified judge model used to score generated images against the benchmark’s rubric inventory (Li et al., 27 May 2026). In the benchmark paper, Q-Judger is described as being based on Qwen3.6-27B (Li et al., 27 May 2026). In a later technical report, it is summarized as a unified vision-LLM trained on ≈130 K human-annotated image-prompt pairs by 80 professional artists, using a chains-of-thought prompt plus a Likert-scale output that maps to a continuous [0,100] score (Xu et al., 25 Jun 2026). The benchmark paper gives the corresponding supervision volume as 130 000+ bilingual, densely supervised prompt–image–facet triples (Li et al., 27 May 2026).

The judge model takes as input a prompt-image pair together with the taxonomy-aware checklist:

$(p,I)\ \text{plus}\ \{r_c\}.$

Its outputs are independent classifications for each active third-level facet $c$, predicting

$s_c\in\{0,1,2,\mathrm{N/A}\}$

in the benchmark paper’s formulation (Li et al., 27 May 2026).

The annotation protocol is unusually explicit. The benchmark paper states that supervision was provided by 80 professional annotators with backgrounds including photography, directing, fine arts, under blind labeling and triple-review, with each sample scored by at least 3 independent experts (Li et al., 27 May 2026). The abstract further specifies that these annotators were drawn from global art academies and that labeling followed blind labeling and triple-review protocols (Li et al., 27 May 2026). During annotation, evaluators were given facet-by-facet, rubric-by-rubric checklists (Li et al., 27 May 2026).

Training minimizes the sum of cross-entropy losses over all active facets:

$$\mathcal{L} \;=\; -\sum_{(p,I)}\;\sum_{c\in \mathcal{C}(p)}\;\sum_{y\in\{0,1,2\}} \mathbf{1}[s_c^{\text{(gold)}}=y]\;\log P_J\bigl(s_c=y\mid p,I,r_c\bigr).$$

This formulation reflects the benchmark’s commitment to independent rubric-grounded supervision rather than monolithic scalar preference labels (Li et al., 27 May 2026).

The reported agreement with expert judgment is high: Spearman $\rho=0.92$ overall, with 0.92 on Real-world Fidelity & Creative Generation and 0.89 on Quality/Aesthetics/Alignment (Li et al., 27 May 2026). This is presented as evidence that the judge model tracks professional human assessments particularly well on the newly introduced application-driven dimensions.

5. Scoring methodology and metric design

Qwen-Image-Bench uses a transparent bottom-up scoring pipeline. At the facet level, the judge infers

$C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$0

These discrete outputs are then normalized to $C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$1 using

$C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$2

with the explicit note that Pass → 60 places the passing threshold at 60\% (Li et al., 27 May 2026).

Aggregation proceeds bottom-up. For a level-2 sub-capability $C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$3,

$C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$4

For a level-1 pillar $C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$5,

$C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$6

For the overall per-sample score,

$C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$7

Model-level scores are then averages over the relevant prompt sets, including $C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$8, $C_p = \{c_1,\dots,c_k\}\subseteq C,\quad k\ge3.$9, $$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$0, and $$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$1 (Li et al., 27 May 2026).

A later evaluation report uses a closely related summary notation. It describes each generated image as receiving a score for every third-level facet $$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$2, then defines the second-level pillar score

$$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$3

and the overall score

$$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$4

All scores are reported on $$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$5 (Xu et al., 25 Jun 2026).

This scoring architecture is central to the benchmark’s claimed interpretability. Because each output is attributable to explicit rubrics, the system can expose whether a model fails on Physical Logic, Text Accuracy, World Knowledge, or Visual Storytelling, rather than only reporting an undifferentiated global reward (Li et al., 27 May 2026). This suggests a diagnostic role that is closer to structured assessment than to pure ranking.

6. Empirical findings, discriminative power, and downstream use

The benchmark paper reports evaluation of 18 frontier T2I models end-to-end (Li et al., 27 May 2026). It states that GPT-Image-2 leads with overall 64.7, followed by Nano Banana 2.0 (59.8) and GPT-Image-1.5 (59.6), while the bottom model sits at ≈48.2, yielding a 16.5-point spread that earlier benchmarks could not reveal (Li et al., 27 May 2026). The benchmark’s strongest separation reportedly appears on the two application-driven dimensions, where existing evaluation provides limited insight (Li et al., 27 May 2026).

Variance analysis is one of the paper’s main arguments for the usefulness of the new taxonomy. It reports that, at level 1, Creative Generation >11× Quality and >4× Aesthetics in variance (Li et al., 27 May 2026). At level 2, the top variances occur in Text Rendering, World Knowledge, and Visual Storytelling, while at level 3 the most variable facets include Text Accuracy, Information Visualization, and Cross-lingual Generation—all located in the application-driven pillars (Li et al., 27 May 2026). The paper also reports inter-tier gaps of +8.68 on Creative Generation and +3.28 on Real-world Fidelity for T1–T2, and +4.29 Creative and +2.48 Fidelity for T2–T3 (Li et al., 27 May 2026).

The benchmark additionally identifies a set of systemic ceilings. It highlights five facets—Physical Logic, Anatomical Fidelity, Animals, Objects, Contact Interaction—for which even the best model scores <44 (Li et al., 27 May 2026). These are described as spanning four pillars and marking a “perception-to-cognition” frontier requiring latent world knowledge (Li et al., 27 May 2026). This characterization indicates that the benchmark is intended to diagnose not only rendering limitations but also failures in implicit physical and semantic reasoning.

Qwen-Image-Bench has already been used as the main automated benchmark in the Qwen-Image-2.0-RL Technical Report (Xu et al., 25 Jun 2026). There, Qwen-Image-2.0-RL is reported to improve the overall benchmark score from 55.23 to 57.84 (+2.61) relative to Qwen-Image-2.0-Base (Xu et al., 25 Jun 2026). The report also gives a pillar-wise breakdown:

Model	Quality	Aesthetics	Alignment	Real-world	Creative	Overall
Qwen-Image-2.0-Base	52.29	57.10	57.64	47.54	58.22	55.23
Qwen-Image-2.0-RL	54.39	58.67	59.28	51.83	64.94	57.84

The largest reported gain is on Creative Generation (58.22 → 64.94, +6.72), followed by Real-world Fidelity (47.54 → 51.83, +4.29) (Xu et al., 25 Jun 2026). The same report also states that these automated gains carried over to human preference, with Text-to-Image Overall Elo: 1115 → 1193 (+78) and Image-Editing Elo: 1256 → 1349 (+93) (Xu et al., 25 Jun 2026). The Elo update is given in standard form:

$$(p_{\rm en},\,p_{\rm zh},\{d_i\}_{i=1}^k)\sim{\rm LLM}_{\rm gen}(C_p),$$6

Taken together, these findings position Qwen-Image-Bench as a benchmark whose main contribution is not merely broader coverage, but finer attribution of model capability across a taxonomy aligned with real creation scenarios (Li et al., 27 May 2026). A plausible implication is that its long-term significance depends on whether the newly emphasized axes—especially Real-world Fidelity and Creative Generation—continue to resist the saturation effects that affected earlier prompt-alignment benchmarks.