T2I-CompBench: Compositional T2I Benchmark

• T2I-CompBench is a benchmark suite designed to test text-to-image models on compositional alignment and coherent scene generation.
• It evaluates models across varied challenges including color, texture, shape, 2D/3D spatial relations, numeracy, and multi-object compositions.
• The suite employs parameterized templates and detector-driven scoring to diagnose issues like attribute swaps, spatial misalignments, and object hallucinations.

T2I-CompBench is a structured benchmark suite for evaluating compositional alignment in text-to-image (T2I) generation, rigorously assessing how well generative models translate natural language specifications—including object identities, fine-grained attributes, spatial relationships, and cardinalities—into coherent images. The benchmark is designed to diagnose and quantify error modes that photorealistic outputs often conceal, such as attribute swaps, spatial misalignment, and object hallucination. Multiple versions exist; T2I-CompBench++ (sometimes T2I-CompBench-plus-plus) represents an expanded scope that introduces 3D spatial relations, richer action semantics, and explicit numeracy tasks (Shahabadi et al., 12 Dec 2025).

1. Design Objectives and Scope

T2I-CompBench evaluates T2I models on their ability to ground linguistic constraints in image generation, with a primary focus on compositional generalization. Its objectives are:

• To isolate common compositional failure modes masked by high perceptual fidelity (attribute swaps, spatial inversions, count errors, omission/hallucination).
• To supply a unified, detector-driven evaluation pipeline that delivers per-category and aggregate compositional alignment scores using automated (and, in some variants, VLM-based) analysis.
• To stress-test models along axes granularly varied in isolation, including attribute binding (color, texture, shape), non-spatial and spatial relations, numeracy, and combinatorially complex scene layouts.

By systematically varying each compositional degree of freedom, T2I-CompBench exposes architectural weaknesses and trade-offs of diffusion, diffusion–transformer hybrid, and visual autoregressive (VAR) models (Shahabadi et al., 12 Dec 2025).

2. Task Categories and Linguistic Constraints

T2I-CompBench++ is organized into eight evaluation categories, each corresponding to a distinct linguistic and visual compositional challenge:

1. Color Binding: Tests correct assignment of colors to objects (e.g., “a red cube”). Often reveals color swapping or hallucination when color palettes overlap.
2. Texture Binding: Evaluates model’s rendering of specified surface textures (e.g., “a furry bear” vs. “a metallic bear”), requiring recognition beyond chromatic cues.
3. Shape Binding: Probes whether models resolve non-prototypical geometric descriptors (e.g., “a triangular prism,” “a cylinder”).
4. Non-Spatial Relations: Assesses semantic role alignment for action/interactions that are not tied to position (e.g., “a cat holding a fish”, “a person wearing a hat”).
5. 2D Spatial Relations: Enforces correct planar positioning (e.g., “the cup to the left of the plate”, “the ball above the box”).
6. 3D Spatial Relations: Introduces occlusions and front/back ordering challenges (e.g., “the chair behind the table”) to interrogate depth reasoning.
7. Numeracy (Counting): Demands that models match exact cardinalities of objects (e.g., “three apples”), with established diffusion models showing marked deficiencies.
8. Complex Multi-Object/Attribute Compositions: Compositional prompts with multiple objects, unique attribute bindings, and inter-object relations (e.g., “a blue cube to the right of a furry cat wearing sunglasses”) place joint pressure on the model’s attribute and relational alignment (Shahabadi et al., 12 Dec 2025).

Each category is motivated by distinct failure modalities identified in large-scale generative T2I systems (Huang et al., 2023).

3. Dataset Construction and Structure

The dataset is constructed from parameterized templates for each compositional axis:

• Templates: Attribute templates (“[COLOR] [OBJECT]”), relation templates (“[SUBJECT] is [RELATION] the [OBJECT]”), numeracy templates (“[COUNT] [OBJECT]s”), and complex conjunctions (e.g., “[COLOR1] [OBJECT1] to the [SPATIAL] of a [COLOR2] [OBJECT2] wearing a [TEXTURE]”).
• Vocabulary Coverage: Instantiated over controlled vocabularies: ~10 colors, ~10 textures, ~8 shapes, ~6 non-spatial relations, ~6 2D spatial relations, ~4 3D spatial relations, and counts 1–5.
• Combinatorial Expansion: Enumerative filling yields 1,000–2,000 unique prompts in T2I-CompBench++, with earlier versions containing 6,000 prompts split evenly across six primary subcategories (color, shape, texture, spatial, non-spatial, and complex composition) (Shahabadi et al., 12 Dec 2025Huang et al., 2023).
• Prompt Distribution: Approximate per-category prompt counts: color/texture/shape (~100 each), non-spatial/2D/3D spatial relations (~150 each), counting (~100), and complex compositions (~200), giving a total in the 1,100–1,200 range per benchmark instance (Shahabadi et al., 12 Dec 2025).

This design guarantees balanced and fine-grained compositional stress-testing across diverse scenarios.

4. Evaluation Protocols and Metrics

T2I-CompBench employs detector-driven and rule-based scoring for granular, reproducible evaluation:

• Per-Category Accuracy ($\mathrm{Accuracy}_c$):

$$\mathrm{Accuracy}_c = \frac{1}{N_c} \sum_{i=1}^{N_c} 1[\mathrm{correct}_i]$$

where $N_c$ is the number of prompts in category $c$, and $1[\mathrm{correct}_i]$ indicates exact alignment for prompt $i$.

• Overall Alignment Score:

$$\mathrm{AlignmentScore} = \frac{1}{C} \sum_{c=1}^{C} \mathrm{Accuracy}_c$$

• Detection-Based and Attribute Metrics:
  • Bounding box detectors (e.g., UniDet) for object localization and spatial relation verification.
  • VQA/BLIP-based attribute binding: for prompts such as “a green bench and a red car,” the evaluator prompts the VQA model: “Is there a green bench? Is there a red car?” and multiplies the associated Yes probabilities or checks binary correctness.
  • Precision/Recall/F1 for multi-object detection, if needed.
• Spatial/3D Relation Verification: For planar arrangements, centroids $(x_A, y_A)$ and $(x_B, y_B)$ are compared; correct if $x_A + \delta \leq x_B$ for “A to the left of B” with a configurable tolerance $\delta$. Depth ordering for 3D configurations is checked analogously.
• Numeracy Verification: Quantitative matching via object counts with success indicated by $$|\text{detected objects of type X}| = \text{specified count}$$.
• Complex Composition: Aggregate correctness only if all specified constraints are met.

Scores are typically reported as seed-averaged means with standard deviations, ensuring statistical reliability (Shahabadi et al., 12 Dec 2025Huang et al., 2023).

5. Position Within the T2I Benchmarking Landscape

T2I-CompBench addressed deficiencies in previous benchmarks, which were limited to constrained phenomena like color binding or low-arity spatial relations. Unlike these, T2I-CompBench delivers broad compositional coverage and standardized pipeline baselines, supporting reproducible, multifactorial T2I evaluation (Huang et al., 2023).

T2I-CompBench++ extends the original framework by:

• Adding non-spatial relation tasks (e.g., agent–patient, “holding,” “wearing”) for semantic role binding assessment.
• Introducing explicit 3D spatial evaluation (front/back, occlusion).
• Expanding complex-compositional templates (involving three or more constraints in a single prompt).

This progression enables the community to track and compare architectural innovations—including VAR, diffusion, and mixed/hybrid models—at levels not attainable via traditional perceptual quality measures (e.g., FID, IS) (Shahabadi et al., 12 Dec 2025).

6. Baseline Results and Model Diagnostics

Systematic comparison of T2I systems on T2I-CompBench and T2I-CompBench++ has established performance stratification across model classes (Shahabadi et al., 12 Dec 2025Huang et al., 2023Lee et al., 4 Feb 2025). Illustrative results include:

Model	Color	Shape	Texture	Spatial	Non-Spatial	Complex
SDXL (base)	0.59	—	—	—	—	—
Infinity-8B	0.83	—	—	—	—	—
Attend-Excite	0.643	0.48	0.63	0.127	—	0.35
GORS	0.662	0.48	0.63	0.145	—	0.35
FLUX-dev-12B	0.74	0.49	0.65	0.22	0.31	0.48
CaPO+SDXL	0.646	0.54	0.63	0.17	0.31	0.49

All scores: representative per-category accuracy (normalized, 0–1); cells are omitted (—) where not explicitly reported in the cited sources (Shahabadi et al., 12 Dec 2025Lee et al., 4 Feb 2025Huang et al., 2023).

From the cited reports:

• Diffusion models (e.g., SDXL, Attend-and-Excite, GORS) achieve moderate attribute-binding, but underperform on spatial relations and complex compositions; highest scores for color, followed by texture and shape.
• FLUX and VAR-based models (Infinity-8B) exhibit stronger compositional alignment, especially in attribute and color binding (Shahabadi et al., 12 Dec 2025).
• State-of-the-art methods using calibrated reward models or inference-time latent guidance further improve baseline compositional fidelity by 2–10 percentage points across categories (Lee et al., 4 Feb 2025).
• 3D spatial and numeracy subtasks remain the most challenging overall.

Performance deltas between baselines and new methods offer direct signals for where architectural improvements—or detector advances—yield substantive gains.

7. Limitations, Adoption, and Future Directions

T2I-CompBench and its variants are crucial for tracking detailed architectural advances in T2I alignment, but several limitations persist:

• No single unified metric yet fully encapsulates all compositional requirements—current pipelines combine VQA, object detection, and manual rules.
• Early versions only supported 2D spatial relations; later releases add—but do not yet saturate—3D and numeracy evaluation.
• LLM-based image scoring (e.g., via GPT-4V, ShareGPT4v) offers promise, but currently lags specialized detectors in human alignment correlation (Huang et al., 2023).
• Pretrained detection backbones and VQA models introduce evaluative bias, and potential for compositional misgeneration to reinforce social or semantic biases exists.

The dataset and pipeline are publicly released, providing transparent baselines and supporting reproducible evaluation for future research in T2I compositional alignment (Shahabadi et al., 12 Dec 2025Huang et al., 2023).

---

References:

• “Infinity and Beyond: Compositional Alignment in VAR and Diffusion T2I Models” (Shahabadi et al., 12 Dec 2025)
• “T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation” (Huang et al., 2023)
• “Calibrated Multi-Preference Optimization for Aligning Diffusion Models” (Lee et al., 4 Feb 2025)