7Bench: Layout-Guided Text-to-Image Benchmark

• 7Bench is a benchmark for layout-guided text-to-image generation that jointly assesses semantic and spatial fidelity.
• It evaluates models using seven distinct scenarios such as object binding, small boxes, overlapping, color and attribute binding, object relationships, and complex composition.
• It employs an automated evaluation protocol using TIFA for semantic scoring and an OWL-ViT based approach with IoU thresholds for precise layout assessment.

Searching arXiv for the 7Bench paper and closely related benchmark context. 7Bench is a benchmark for layout-guided text-to-image generation whose central purpose is to evaluate whether a model is faithful simultaneously to the textual prompt and to the input layout specified as bounding boxes. It is presented as the first benchmark specifically designed to jointly assess semantic alignment and spatial alignment for layout-guided text-to-image models, using text-and-layout pairs spanning seven challenging scenarios and an automated protocol with separate text-alignment and layout-alignment scores (Izzo et al., 18 Aug 2025).

1. Scope and rationale

Layout-guided text-to-image generation extends ordinary text-conditioned synthesis by conditioning generation on both a prompt and an explicit spatial specification. In 7Bench, that specification is a set of bounding boxes, chosen because bounding boxes are simple, common, and easy to annotate. They encode object identity, approximate position, scale, and spatial arrangement. The benchmark is motivated by the observation that current models often satisfy only part of this contract: they may generate the right objects in the wrong places, follow the boxes but miss attributes or relations, or omit and mix objects entirely (Izzo et al., 18 Aug 2025).

The benchmark is framed against two shortcomings in prior evaluation practice. Traditional text-to-image benchmarks assess semantic fidelity—such as color, attribute, and relation following—but do not include layout inputs. Conversely, existing layout-guided evaluation is described as heterogeneous, with different papers using different datasets, sample sizes, and layout-quality protocols. The paper further argues that COCO/Flickr-based subsets are problematic because such datasets are commonly used in model training, and that a recent spatial-skills benchmark focuses mainly on spatial control rather than joint semantic-and-spatial evaluation. 7Bench therefore treats semantic fidelity and spatial fidelity as distinct but inseparable dimensions of controllability.

This joint treatment is especially important for synthetic data generation. If generated images are used as downstream training data, poor layout adherence introduces label noise: objects may not appear in their assigned boxes, requested objects may be missing, multiple objects may be merged, and object-box correspondences may become unreliable. In that sense, 7Bench is not merely a benchmark for image plausibility; it is a benchmark for whether layout-guided generation is dependable when spatial annotations matter.

2. Benchmark composition and the seven scenarios

7Bench contains 224 samples, each consisting of a textual prompt and a set of bounding boxes, with 32 prompts per scenario. Across scenarios, the number of objects varies as $N \in \{1,2,3,4\}$ depending on the setting. Unless otherwise noted, the bounding boxes are manually collected, with the stated goal of creating realistic layouts that preserve object proportions and avoid overlap except where overlap is intentional (Izzo et al., 18 Aug 2025).

The benchmark is organized around seven scenarios, each targeting a distinct failure mode in layout-guided generation.

Scenario	Capability stressed	Typical failure
Object Binding	Object presence and localization	Omitted, confused, or merged objects
Small Bboxes	Generation in tight spatial regions	Missing or unrecognizable small objects
Overlapping Bboxes	Spatial control under overlap	Collapsed or poorly localized layered objects
Color Binding	Color-to-object assignment	Color swap or leakage
Attribute Binding	General attribute binding	Wrong or weak modifier grounding
Object Relationship	Pairwise relation realization	Relation inversion or visually absent relation
Complex Composition	Multi-constraint compositional control	Cascading semantic and spatial failures

The Object Binding scenario contains no attributes and no relations; it isolates the elementary question of whether all requested objects are generated distinctly and placed correctly. Small Bboxes reuses the same prompt style but constrains object area to small regions, probing whether models can maintain recognizability and placement when the spatial conditioning signal is weak. Overlapping Bboxes intentionally violates disjointness, testing whether models can handle layered or co-located content rather than treating overlap as an error condition.

Color Binding and Attribute Binding shift the focus from object presence to modifier attachment. The former isolates color as an attribute class; the latter broadens the attribute vocabulary to include material, appearance, dimension, and other descriptive properties. Object Relationship tests relation words such as “above,” “under,” and “to the left of,” connecting language-level relational semantics to image geometry. Complex Composition combines the difficulties of object multiplicity, attributes, relations, small boxes, and overlapping boxes into a more open-ended compositional stress test.

A common misconception is that layout-guided generation can be assessed adequately by asking only whether the right objects appear. 7Bench is organized around the opposite view: object generation, attribute binding, relation following, and spatial placement are separate competence axes, and failures on any one of them compromise controllability.

3. Prompt formalism and layout representation

The prompt design is described as being inspired by disentangled representation ideas. Each prompt is constructed from objects $\mathcal{o}_i \in \mathcal{O}$, optional attributes $\mathcal{a}_i \in \mathcal{A}$, and optional relations $\mathcal{r}_{ij} \in \mathcal{R}$. For a two-object prompt, the paper gives the general template

$t = \text{"det($\mathcal{o}_1$,$\mathcal{a}_1$)$\mathcal{a}_1$$\mathcal{o}_1$rel(and,$\mathcal{r}_{12}$) det($\mathcal{o}_2$,$\mathcal{a}_2$)$\mathcal{a}_2$$\mathcal{o}_2$"}.$

Each object is paired with a target box

$$bbox_i = [x_{min}^i,y_{min}^i,x_{max}^i,y_{max}^i],$$

with normalized coordinates satisfying

$$0 \leq x_{min} < x_{max} \leq 1,\quad 0 \leq y_{min} < y_{max} \leq 1.$$

The Small Bboxes scenario introduces an explicit area constraint. If

$$A_i = (x_{max}^i - x_{min}^i)\cdot(y_{max}^i-y_{min}^i)$$

and the image area is $A_{image}=w\cdot h$, then the box area is constrained to be between 3% and 10% of image area:

$$0.03 \cdot A_{image} \leq A_{image} \cdot A_i \leq 0.1 \cdot A_{image}.$$

This makes the scenario a direct stress test of spatial conditioning under small target regions.

The color set $\mathcal{o}_i \in \mathcal{O}$0 contains 11 universal basic colors: black, blue, brown, gray, green, pink, purple, red, white, yellow, and orange (Izzo et al., 18 Aug 2025). The attribute set is given as

$\mathcal{o}_i \in \mathcal{O}$1

The text states that “In total, we investigate 30 different attributes,” but the explicit set shown appears to contain 29 items. This discrepancy is part of the benchmark description rather than an external correction. The relation set is

$\mathcal{o}_i \in \mathcal{O}$2

For overlapping layouts, the intended condition is that every box overlaps at least one other box. The notation in the paper is described as slightly malformed, but the intended meaning is clear: overlap is a deliberate scenario variable rather than an annotation accident.

4. Evaluation protocol

7Bench uses two complementary automated metrics: a text-alignment score $\mathcal{o}_i \in \mathcal{O}$3 and a layout-alignment score $\mathcal{o}_i \in \mathcal{O}$4. The benchmark does not collapse them into a single scalar. This design choice is substantive: a model may be semantically faithful while spatially inaccurate, or spatially consistent while semantically wrong, and the benchmark treats these as analytically distinct outcomes (Izzo et al., 18 Aug 2025).

The text score is defined as the TIFA score, with $\mathcal{o}_i \in \mathcal{O}$5. TIFA operates through a VQA-based pipeline: an LLM generates question-answer pairs from the prompt, those questions are posed to a VQA model conditioned on the generated image, and the VQA outputs are compared to the reference answers generated from the prompt. The resulting score is the VQA accuracy. In 7Bench, this semantic axis is used to assess object presence, attributes, relations, and prompt-level fidelity.

The more distinctive component is the detector-based layout score. Given a generated image $\mathcal{o}_i \in \mathcal{O}$6, target objects $\mathcal{o}_i \in \mathcal{O}$7, and target boxes $\mathcal{o}_i \in \mathcal{O}$8, an object detector such as OWL-ViT produces detections

$\mathcal{o}_i \in \mathcal{O}$9

where $\mathcal{a}_i \in \mathcal{A}$0 is a predicted box, $\mathcal{a}_i \in \mathcal{A}$1 a predicted label, and $\mathcal{a}_i \in \mathcal{A}$2 a confidence score. For each target object $\mathcal{a}_i \in \mathcal{A}$3, detections are filtered by label:

$\mathcal{a}_i \in \mathcal{A}$4

The highest-confidence matching box is then selected:

$\mathcal{a}_i \in \mathcal{A}$5

The overlap with the target box is measured by

$\mathcal{a}_i \in \mathcal{A}$6

For thresholds $\mathcal{a}_i \in \mathcal{A}$7, the benchmark computes

$\mathcal{a}_i \in \mathcal{A}$8

The layout score is then defined as the area under this $\mathcal{a}_i \in \mathcal{A}$9 curve:

$\mathcal{r}_{ij} \in \mathcal{R}$0

This procedure evaluates whether objects are not only detected, but detected in the correct spatial regions. Because it sweeps multiple IoU thresholds, it captures both loose and strict forms of spatial alignment. The paper does not specify non-max suppression details, detector confidence thresholds, or synonym/category normalization; the core protocol consists of label filtering, highest-confidence selection, IoU computation, threshold sweeping, and AUC aggregation.

5. Experimental setting and empirical findings

The paper evaluates four layout-guided methods—GLIGEN, Attention Refocusing, BoxDiff, and Cross Attention Guidance—plus Stable Diffusion v1.4 as a text-only reference. GLIGEN is the trained-with-grounding model; Attention Refocusing and BoxDiff are training-free methods built on GLIGEN; Cross Attention Guidance is a training-free method built on Stable Diffusion. For each of the 224 samples, the study generates 16 images per model with seeds 1 through 16 at resolution $\mathcal{r}_{ij} \in \mathcal{R}$1, for a total of $\mathcal{r}_{ij} \in \mathcal{R}$2 images. Evaluation uses pretrained TIFA and pretrained OWL-ViT (Izzo et al., 18 Aug 2025).

The reported score ranges are approximately 0.55 to 0.9 for text alignment and approximately 0.05 to 0.5 for layout alignment. The empirical message is therefore asymmetric: current models can often achieve moderate-to-strong semantic plausibility, but their spatial fidelity remains much weaker. GLIGEN obtains the best layout results in the majority of scenarios, and GLIGEN, BoxDiff, and Attention Refocusing outperform SD_CAG on layout alignment. On the semantic axis, the pattern is different: BoxDiff generally outperforms the other models on text alignment, and SD_CAG achieves text results comparable to the other training-free methods despite being built on Stable Diffusion rather than GLIGEN.

Scenario-level analysis shows marked heterogeneity. Small Bboxes is identified as one of the hardest scenarios, especially for layout fidelity, because small boxes provide weak conditioning signals and force recognizability within tight regions. Overlapping Bboxes is less difficult than might be expected; the paper notes that models often find “creative solutions” that satisfy overlap constraints. Complex Composition is the hardest compositional scenario, and a notable finding is that all layout-guided models tend to converge to similar text-alignment performance there, suggesting a shared failure regime under combined semantic and spatial constraints.

The number of objects also matters systematically. Both text alignment and layout alignment decrease as the number of objects increases. This is expected from the combinatorics of multi-object conditioning, but 7Bench quantifies it within a shared protocol. The degradation is less sharp for GLIGEN-based methods than for SD_CAG and plain Stable Diffusion, reinforcing the paper’s interpretation that explicit grounding during training remains important for spatial control as compositional load increases.

Qualitative examples in the paper compare dashed ground-truth boxes with solid boxes produced by the detector-based layout evaluation pipeline. Their role is diagnostic rather than illustrative in a generic sense: they show that visually plausible images can still fail the benchmark because objects are displaced, merged, or only loosely aligned with the requested layout.

6. Significance, limitations, and release

7Bench is best understood as a benchmark for controllable generation rather than as a generic text-to-image benchmark. Its novelty lies not in evaluating whether images look plausible in isolation, but in evaluating whether models realize both what to generate and where to place it. That framing makes it particularly relevant for applications in which spatial fidelity is operationally consequential, especially synthetic data generation, where poor layout adherence can corrupt downstream labels and degrade detector or grounding models trained on generated images (Izzo et al., 18 Aug 2025).

The benchmark also clarifies an important methodological point: semantic fidelity and spatial fidelity should not be conflated. A single scalar score would obscure cases in which a model performs well semantically but poorly spatially, or vice versa. By keeping $\mathcal{r}_{ij} \in \mathcal{R}$3 and $\mathcal{r}_{ij} \in \mathcal{R}$4 separate, 7Bench makes it possible to compare training-aware grounding methods and training-free guidance methods without forcing their strengths into one aggregate number.

Its limitations are mostly implicit in the benchmark design. Results depend on pretrained external evaluators, namely TIFA for semantic scoring and OWL-ViT for layout scoring. The benchmark size is modest at 224 samples. Bounding-box layouts are manually collected unless otherwise specified, which supports curation quality but limits scale. The layout score depends on detector label matching, so detector errors can affect reported spatial fidelity. These are not peripheral caveats: they define the operational envelope within which 7Bench should be interpreted.

The benchmark is publicly released at https://github.com/Elizzo/7Bench. For reproduction, the paper identifies the key ingredients as the 224 text-and-bounding-box pairs, the 7-scenario split with 32 samples each, generation of 16 images per sample at $\mathcal{r}_{ij} \in \mathcal{R}$5, seeds 1 to 16, the evaluated model implementations, pretrained TIFA, pretrained OWL-ViT, and the IoU-threshold-sweep layout-score computation pipeline. Within those bounds, 7Bench provides a compact but technically pointed instrument for measuring a capability that many layout-guided systems claim but do not yet robustly achieve: joint semantic-spatial controllability.