T2I-CoReBench: Benchmark for T2I Compositionality

• T2I-CoReBench is a high-complexity benchmark that evaluates explicit compositionality and implicit reasoning in text-to-image models using a unified 12-dimensional taxonomy.
• It integrates scene graph-based composition checks with multi-step reasoning metrics across deductive, inductive, and abductive inference types.
• Empirical analyses reveal strong compositional performance yet a consistent reasoning gap, highlighting the need for architectural innovations in T2I systems.

T2I-CoReBench is a comprehensive and high-complexity benchmark for evaluating both explicit compositionality and implicit reasoning in contemporary text-to-image (T2I) models. Its design integrates a unified, fine-grained evaluation taxonomy that systematically assesses scene graph-based composition and multi-type reasoning skills under challenging prompt scenarios, probing models’ abilities to “set the stage” (compose explicit visual elements) as well as to “direct the play” (infer and manifest implicit, inferential scene requirements) (Li et al., 3 Sep 2025).

1. Benchmark Taxonomy and Structure

T2I-CoReBench is structured around a 12-dimensional taxonomy, capturing explicit composition and diverse forms of visual reasoning. The taxonomy decomposes into:

• Composition Dimensions (4)
  • Multi-Instance (MI): Checks if the model renders all specified instances (e.g., producing each of 25 named objects).
  • Multi-Attribute (MA): Assesses whether each object manifests all assigned attributes (color, size, material, etc.).
  • Multi-Relation (MR): Verifies accurate generation of specified spatial or semantic relationships between objects.
  • Text Rendering (TR): Evaluates fidelity and formatting of rendered text, treated as a visual scene element.
• Reasoning Dimensions (8), aligned with inference types:
  • Deductive Reasoning:
  • Logical Reasoning (LR): Multi-hop, premise-conclusion mapping.
  • Behavioral Reasoning (BR): Inferring observable effects from described actions.
  • Hypothetical Reasoning (HR): Propagation of counterfactuals (e.g., “if wheels were square…”).
  • Procedural Reasoning (PR): Multi-step procedure integration, given only end-state.
  • Inductive Reasoning:
  • Generalization Reasoning (GR): Extrapolating from examples to new cases.
  • Analogical Reasoning (AR): Rule transfer between source and target scenarios.
  • Abductive Reasoning:
  • Commonsense Reasoning (CR): Extrapolating expected but unmentioned real-world details.
  • Reconstructive Reasoning (RR): Inferring the most plausible event history from static observations.

Composition evaluation leverages established scene graph theory (instances as nodes, attributes as properties, and relations as edges), while reasoning is rigorously anchored in philosophical taxonomy, mapping each dimension to a canonical inference challenge.

2. Dataset Complexity and Prompt Design

T2I-CoReBench deliberately escalates complexity compared to prior work:

• High compositional density: Prompts routinely contain up to 25 visual entities, each with multiple and mutually dependent attributes and relational constraints, replicating the density and diversity found in real-world scenes.
• Multi-step reasoning: Prompts embed inference chains such as “if…, then…, and therefore…” or “given these examples, extrapolate…” that demand synthesis across composition and reasoning axes.
• Fine-grained evaluation: Each prompt is paired with a detailed checklist of independent yes/no questions (≈13,500 in total across 1,080 prompts), covering every explicit and implicit requirement.

Checklist Example:

Dimension (Taxonomy)	Checklist Question Example
MI	Are there at least 25 distinct objects?
MA	Is the color/material of each object correct?
MR	Are spatial relations (e.g., “on the curb”) accurate?
TR	Does the rendered text content match in layout/font?
BR	Does the image show a glass fallen and water spilled?

This fine-grained, atomic checklist structure enables per-element assessment rather than global text-image similarity, greatly increasing both evaluation reliability and interpretability.

3. Evaluation Protocol and Metrics

T2I-CoReBench’s protocol:

• Each prompt is mapped to a checklist of $N$ binary assertions (elements to verify).
• Model output is assessed on a per-assertion basis: $$\mathcal{S} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\text{model correctly renders } i]$$.
• CR and RR prompts additionally require inferring unstated (yet necessary) scene elements, increasing depth and reliability of reasoning evaluation.

Notably, compositional questions center on explicit visible structure, while reasoning questions probe for the correct rendering of inferred or implicit elements (e.g., logical outcomes, plausible causes, or generalized patterns).

Evaluation is performed across 27 diverse T2I models, spanning diffusion, autoregressive, and multimodal LLM-driven architectures.

4. Experimental Analysis and Model Insights

The benchmark’s rigorous protocol surfaces clear capability stratification:

• Composition: State-of-the-art open-source and closed-source models have made meaningful progress—Imagen 4 Ultra attains 82.4 (composition score), Qwen-Image reaches 78.0; object, attribute, and relation placement are generally robust for moderate density.
• Reasoning: All tested models, regardless of generative backbone, exhibit a pronounced “reasoning bottleneck,” lagging over 9 points behind their compositional performance. Tasks involving logical consequence, causal reasoning, counterfactual imagination, or commonsense enrichment consistently yield subpar results, even for recent MLLM-integrated systems.
• “Thinking before painting” (i.e., LLM-mediated prompt rewriting or intermediate explicit reasoning as in “BAGEL w/ Think”) yields tangible gain on reasoning dimensions, but sometimes at the cost of compositional fidelity.

Interpretation: These findings demonstrate that, while contemporary T2I systems are approaching human-level explicit scene arrangement, their ability to embed inference-driven visual implications remains a critical unsolved problem. Architectural and training advances remain necessary for bridging this gap.

5. Comparison to Prior Benchmarks

Relative to existing evaluation suites, T2I-CoReBench is uniquely distinguished by:

• Coverage: It unifies composition and explicit reasoning under a single, densely populated taxonomy.
• Granularity: The independently-scored checklist system enables precise identification of strengths and weaknesses at both local (elementwise) and global (scene-level) scales.
• Complexity: Arms-length prompts with multiple cross-cutting dependencies represent a step-change in difficulty versus prior one-to-one or low-density benchmarks.
• Unified reasoning typology: Whereas earlier benchmarks evaluated only compositional control or limited forms of in-context commonsense (Huang et al., 2023, Fu et al., 11 Jun 2024), T2I-CoReBench provides dedicated, high-complexity tracks for deduction, induction, and abduction.

6. Implications for Model Development and Future Research

T2I-CoReBench has direct and indirect ramifications for T2I research trajectories:

• Training Data: There is a demonstrated need for training datasets and augmentation pipelines that place greater emphasis on non-trivial, reasoning-relevant scene compositions and inferred phenomena.
• Architectural Innovation: Integration of explicit reasoning stages (e.g., chain-of-thought modules, intermediate LLM planning, or hybrid symbolic circuits) is recommended to address the gap between surface compositionality and visual reasoning (Goswami et al., 8 Dec 2024, Jiang et al., 1 May 2025, Liu et al., 6 Jul 2025).
• Metric Design: Task-specific, atomic-level evaluation—rather than holistic text-image scoring—can guide both model selection and error diagnosis.
• Applications: For domains requiring higher-order visual inference (e.g., VQA, interactive agents, simulation-to-reality transfer), reliance solely on compositional trends is insufficient; explicit reasoning evaluation is essential.

A plausible implication is that future state-of-the-art T2I models may need to combine large-scale multimodal pretraining with explicit, modularized reasoning engines, aiming to close the reasoning–composition gap identified by T2I-CoReBench.

7. Conclusions and Roadmap

T2I-CoReBench delivers a high-complexity, high-coverage, and methodologically rigorous standard for evaluating the true upper-bound of T2I model capability. Its dual focus—composition and reasoning grounded in formal inference theory—exposes the persistent challenges of implicit information encoding, compositional control, and real-world scene understanding in generative systems. By providing both a robust suite of test cases and a reproducible, atomic evaluation protocol, it lays the groundwork for the next era of research in visually-grounded language understanding and reasoning-integrated image synthesis (Li et al., 3 Sep 2025).