PosEval: Spatial Reasoning Benchmark

• PosEval is a systematic benchmark that assesses position-based spatial understanding and compositionality in text-to-image generative models.
• It extends the GenEval framework with six tasks—ranging from multi-object relations to attribute binding and negation—to probe complex spatial arrangements.
• Empirical results demonstrate that integrating position-control mechanisms like Stitch significantly boosts model accuracy, revealing key challenges in spatial reasoning.

PosEval is a systematic benchmark for evaluating position-based spatial understanding and compositionality in contemporary Text-to-Image (T2I) generative models, especially those employing multi-modal diffusion transformer (MMDiT) architectures. Its design extends the GenEval benchmark by targeting more complex spatial arrangements and relations, systematically probing key failure modes in positional reasoning, attribute binding, negation, and relative reference. PosEval forms a critical standard for assessing spatial control advances in state-of-the-art models such as FLUX, SD3.5, and Qwen-Image, as well as for quantifying the gains yielded by position-control mechanisms such as Stitch (Bader et al., 30 Sep 2025).

1. Motivation and Design Rationale

GenEval’s “Position” task, which features two objects and one spatial relation, has been saturated by current generation T2I models. However, actual image generation scenarios present a diverse set of spatial challenges: scenes often involve more than two objects, chained or cyclic relations, simultaneous attribute binding and spatial reference, negative spatial predicates, and relations defined relative to others.

PosEval is constructed as a direct extension of GenEval, maintaining its object ($\mathcal O$; ~80 common objects) and relation ($\mathcal R$ = {left of, right of, above, below}) vocabularies, procedural analyses, and Mask2Former-based detection, but introducing five novel tasks. Each task is instantiated with 100 random prompts and four sampled images for robust statistical support (400 images/task). The focus is on controlled difficulty escalation and tight evaluation, addressing scaling, negation, attribute binding, and relative referentiality.

2. Task Suite Specification

PosEval comprises six tasks. The initial task mirrors GenEval (2Obj), and five new tasks increase object count, relation chain complexity, introduce negation, attribute-binding, and relative relations.

Task	Prompt Structure Example	Objects per prompt	Relations per prompt	Core Pass/Fail Criterion
2 Obj	“A photo of a $\langle$obj₁$\rangle$ $\langle$rel₁₂$\rangle$ a $\langle$obj₂$\rangle$.”	2	1	Both objects present; spatial relation holds
3 Obj	“A photo of a $\langle$obj₁$\rangle$, a $\mathcal R$0obj₂$\mathcal R$1, and a $\mathcal R$2obj₃$\mathcal R$3. $\mathcal R$4. $\mathcal R$5.”	3	2 (orthogonal axes)	All three detected; both relations hold (2×2 grid)
4 Obj	“A photo of $\mathcal R$6obj₁$\mathcal R$7, ...$\mathcal R$8obj₄$\mathcal R$9. $\langle$0. ... $\langle$1.”	4	4 (cyclic)	Four detected; each specified pairwise relation holds
PAB	“A $\langle$2attr₁$\langle$3 $\langle$4obj₁$\langle$5 $\langle$6rel₁₂$\langle$7 a $\langle$8attr₂$\langle$9 $\rangle$0obj₂$\rangle$1.”	2	1	Both objects/attributes present; spatial and attribute match
Neg	“A photo of a $\rangle$2obj₁$\rangle$3 and a $\rangle$4obj₂$\rangle$5, a $\rangle$6obj₁$\rangle$7 is not $\rangle$8rel₁₂$\rangle$9 a $\langle$0obj₂$\langle$1.”	2	1 (negative)	Both present; strictly inverse relation or neutral orientation
Rel	“A photo of a $\langle$2obj₁$\langle$3 $\langle$4rel₁₂$\langle$5 $\langle$6obj₂$\langle$7, and a $\langle$8obj₃$\langle$9 on the (same/opposite) side of $\rangle$0obj₂$\rangle$1 as $\rangle$2obj₁$\rangle$3.”	3	2 (1 absolute, 1 relative)	All three detected; correct relational reference resolution

Each prompt template incorporates randomization of object choice, relation, and sentence order, preventing models from exploiting index patterns.

3. Formal Evaluation Protocol

3.1 Object and Spatial Representation

Objects are detected using Mask2Former and represented via bounding boxes: $\rangle$4

3.2 Relation Metric Functions

• Basic Relations:
  • $\rangle$5 left of $\rangle$6: $\rangle$7
  • $\rangle$8 right of $\rangle$9: $\langle$0
  • $\langle$1 above $\langle$2: $\langle$3
  • $\langle$4 below $\langle$5: $\langle$6
• Negative Relations: $\langle$7, e.g. for “not left of,” either “right of” or neutral (vertically aligned) is required.
• Relative Relations:
  • “Same side” (horizontal): $\langle$8
  • “Opposite side”: $\langle$9
• Attribute Binding: After class verification, color inside $\rangle$0 is mapped to the nearest label via pre-defined HSV lookup.

3.3 Aggregate Task Scoring

For task $\rangle$1, with $\rangle$2 images,

$\rangle$3

$\rangle$4

4. Dataset Generation and Curation

All evaluations leverage the GenEval object set and a fixed set of spatial relations. Prompt generation is fully procedural, with code provided for all six tasks. Each evaluation produces 2,400 images per model (6 tasks × 100 prompts × 4 samples). No human annotation is involved—detection and analysis are performed post hoc using Mask2Former.

All resources—prompts, matching scripts, and evaluation code—are available at https://github.com/ExplainableML/Stitch/tree/main/PosEval.

5. Benchmark Execution Workflow

Evaluation on a candidate T2I model $\rangle$5 employs the following framework:

$\rangle$6

The evaluate_task procedure applies the formal checks detailed in Section 3 for the given task, including object presence, spatial tests, negation, and attribute matching as required by prompt.

6. Empirical Results and Observed Model Behavior

PosEval elucidates dramatic gaps in multi-object spatial generalization, compositionality, and complex relation resolution for leading models:

Model	2Obj	3Obj	4Obj	Neg	Rel	PAB	Avg.
FLUX (base)	0.22	0.06	0.02	0.62	0.03	0.15	0.18
+ Stitch	0.70	0.44	0.38	0.83	0.48	0.44	0.55
SD3.5 (base)	0.34	0.05	0.01	0.62	0.05	0.14	0.20
+ Stitch	0.53	0.22	0.12	0.79	0.27	0.37	0.38
Qwen-Image (base)	0.76	0.40	0.21	0.49	0.10	0.61	0.43
+ Stitch	0.87	0.67	0.61	0.93	0.43	0.77	0.71

Base models display near-chance performance for multi-object and relational reference (3Obj/4Obj/Rel), with accuracy much lower than for simple binary relation tasks. Application of Stitch confers absolute gains up to +0.48 and relative improvements of 200-400% on challenging tasks. Qwen-Image with Stitch achieves the highest PosEval mean accuracy (0.71), and leads on five of six tasks.

Significant error types include missing or mis-placed objects, attribute–relation confusion, trivial satisfaction of negative predicates by spatial neutralization (e.g., centering), and almost universal failure on relative reference tasks for baselines. Stitch recovers substantial competence on these challenging cases.

7. Tools, Reproducibility, and Benchmark Significance

PosEval employs standardized components and open-source code, including prompt generators, Mask2Former detection wrappers, and explicit metric implementations, facilitating reproducibility across research groups. Prompts, detection scripts, and evaluation logic are accessible at https://github.com/ExplainableML/Stitch/tree/main/PosEval, enabling integration with arbitrary T2I models.

The benchmark establishes a rigorous and multidimensional criterion for spatial reasoning in generative vision–language systems, and provides a granular diagnostic platform for advances in content controllability and positional accuracy (Bader et al., 30 Sep 2025).