PlanQA: Spatial Reasoning Benchmark

• PlanQA is a diagnostic benchmark that evaluates LLMs' capabilities to reason about indoor scenes using structured JSON or XML representations.
• It categorizes tasks into metric computation, topology, geometric planning, visual logic, and rule-based validation to rigorously test spatial reasoning.
• The benchmark leverages 1,800 synthetic layouts from kitchens, living rooms, and bedrooms, revealing LLM limitations in continuous-space search and geometric inference.

PlanQA is a diagnostic benchmark for evaluating geometric and spatial reasoning capabilities in LLMs using structured, symbolic representations of indoor scenes. It focuses on probing models’ abilities to reason about scenes such as kitchens, living rooms, and bedrooms, spanning both metric computations and more complex geometric planning tasks. PlanQA operationalizes its queries over input floor plans encoded as JSON or XML object-centric layouts, without recourse to visual or simulation-based spatial solvers (Rodionov et al., 10 Jul 2025).

1. Structured Scene Representations

PlanQA employs object-centric, text-based representations modeled after architectural abstractions. Each scene is described in a flattened structure—either JSON or XML—without nested hierarchies beyond a primary objects list and room-level metadata. The core schema includes:

• Room block: specifies width, depth (in meters), shape (rectangular/L-shaped/open), and optional metadata, such as shape description or intended use.
• Objects list: each object is defined by an id (e.g., "fridge_1"), type (e.g., "fridge," "sofa," "door"), and an axis-aligned bounding box: $[x_{\min}, y_{\min}, x_{\max}, y_{\max}]$ in meters (origin at bottom-left). Comments are optional.

Example JSON representation:

$x_c = (x_{\min} + x_{\max})/2$5

Corresponding XML encoding:

$x_c = (x_{\min} + x_{\max})/2$6

All layouts are presented verbatim to models, excluding any external solver, geometry engine, or simulation environment.

2. Cognitive Taxonomy of Questions

PlanQA tests ten distinct question types grouped under five cognitive targets, each with explicit formalization:

• Metric Computation (M):
  • Distance: Euclidean metric between centroids $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$ with $x_c = (x_{\min} + x_{\max})/2$
  • Area: Aggregate area, e.g., $\sum_i (x_{2,i} - x_{1,i})(y_{2,i} - y_{1,i})$
• Topology (T):
  • Free Space: Unoccupied area $$= \text{room\_area} - \text{area}(\bigcup \text{object bboxes})$$
  • Missing Object: Placement region connected (path-connected) to door and satisfying clearance constraints
• Geometric Planning (G):
  • Repositioning: Maximum translation $t^*$ before collision in a specified direction
  • Max Box: Largest empty axis-aligned rectangle avoiding all placed bboxes: maximize $(x_2-x_1)(y_2-y_1)$ s.t. no overlap
  • Fit/Placement: Boolean check for placement of a rectangle $W \times H$ in free space
• Visual Logic / Occlusion (V):
  • View Angle: $$\theta = \arccos(\mathbf{u} \cdot \mathbf{v} / (\|\mathbf{u}\|\|\mathbf{v}\|))$$ between facing vector and target
  • Obstruction: Enumerate objects whose bboxes are intersected by the centroid-to-centroid line segment
• Rule-based Validation (R):
  • Shortest Path: Graph over free-space corners with A★ search, path validity by Fréchet distance ($\leq 0.6$ m) and clearance ($$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$0 m)
  • Missing Object: Placement solution meeting $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$1 m clearance constraints

3. Dataset Construction and Distribution

The PlanQA dataset comprises 1,800 synthetic 2D room layouts generated by Gemini 2.5 Pro under explicit geometric and functional constraints, later filtered for plausibility. The layout distribution is:

• Kitchens: 600 (40% rectangular, 40% L-shaped, 20% open)
• Living rooms: 600 (mirroring kitchen proportions)
• Bedrooms: 600 (50% rectangular, 30% L-shaped, 20% open)

Each layout is instantiated with 10 question templates (one per cognitive type), resulting in 18,000 question–layout pairs. Categories are balanced across room types and question types.

4. Evaluation Protocol, Scoring, and Ablations

PlanQA uses stringent, typologically specific scoring protocols:

• Scalar (N): Acceptance if $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$2
• Boolean (B): Exact match
• Sequence (S): Path Fréchet distance $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$3 m
• List (L): Set equality
• Coord (C): Geometric predicates (non-overlap, clearance)

Responses are flagged as Correct, Incorrect, Malformed, or Missing, parsing out reasoning faults from answer formatting failures. Ablations on layout encoding (JSON/XML) and input jitter ($$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$4 m) show robust accuracy. This suggests that answer quality is insensitive to low-level serialization and minor geometric perturbations.

5. Empirical Model Performance and Failure Modes

Zero-shot evaluation of major LLMs reveals distinct strengths and weaknesses:

• Models: DeepSeek-R1, Qwen3-32B (reasoning), GPT-4.1, DeepSeek-V3, LLaMA 3-70B, phi-4, Gemma 2-27B (generalist), GPT-4o-mini, LLaMA 3-8B, phi-3.5-mini, Gemma 2-9B (compact/instruction-tuned)
• Metric Computation (M): $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$5 accuracy on top models
• Visual Logic (V): $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$6 (angle, obstruction)
• Topology (T): $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$7 for living/bedrooms, marginally higher in kitchens
• Geometric Planning (G): $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$8, with Fit/Placement best, Max Box $$d = \sqrt{(x_{c,B} - x_{c,A})^2 + (y_{c,B} - y_{c,A})^2}$$9 (GPT-4.1)
• Rule-based (R): $x_c = (x_{\min} + x_{\max})/2$0, shortest path validity often $x_c = (x_{\min} + x_{\max})/2$1

Room-type granularity shows highest mean accuracy in kitchens ($x_c = (x_{\min} + x_{\max})/2$2), with cluttered living rooms and bedrooms performing at $x_c = (x_{\min} + x_{\max})/2$3. Instruction-tuned compact models can rival large generalists in visual/occlusion queries. Scaling accuracy improves steeply up to $x_c = (x_{\min} + x_{\max})/2$4B parameters, then plateaus.

Qualitative failure modes include non-optimal rectangle selection in Max Box (e.g., LLM chooses 1.47 m² versus true 1.54 m²), and subvalid shortest paths violating clearance or cutting corners. Models tend to enumerate candidate regions but miss exhaustive, global inference.

6. Analysis of Model Limitations

Empirical analysis reveals:

• Lack of exhaustive continuous-space search: LLMs favor suboptimal regions when enumerating free space or placements.
• Collision simulation deficits: In repositioning and path finding, clearance and collision constraints are routinely ignored or misapplied.
• Poor generalization in cluttered scenes: Performance precipitously declines in living/bedrooms with irregular, dense layouts versus grid-like kitchens.
• Absence of spatial primitives: Models rely on superficial linguistic heuristics, lacking internalized geometric inference or constructive representation strategies.
• Scale plateau: Increasing parameter count and chain-of-thought tuning improve performance only up to a threshold, after which returns diminish on the most challenging geometric tasks.

A plausible implication is that architectural and dataset inductive biases, not scale alone, are critical for robust geometric and spatial reasoning.

7. Recommendations and Prospective Directions

Immediate strategies to address existing limitations include hybrid LLM+solver architectures, delegating geometric, collision-checking, or planning subroutines to specialized modules. Prompting approaches incorporating explicit enumeration of candidate regions, such as grid-based proposals, may improve coverage.

In the longer term, research directions advocated by PlanQA include spatially grounded training objectives with explicit collision, clearance, and occlusion constraints, as well as the use of negative samples to penalize constraint violations. Data augmentation with irregular, non-grid layouts is recommended to counteract inductive bias. Architectural designs embedding bounding-box or relational geometrical modules may encourage LLMs to acquire the necessary primitives for robust floor plan reasoning.

By open-sourcing PlanQA and its datasets, the benchmark is positioned to catalyze progress in developing LLMs with spatial and geometric capabilities relevant to architecture, robotics, and embodied AI (Rodionov et al., 10 Jul 2025).