Blueprint-Bench: Spatial Reasoning & Retrieval

• Blueprint-Bench is a benchmark suite designed to evaluate AI spatial reasoning and multimodal retrieval by reconstructing floorplans and retrieving complex engineering documents.
• It employs quantitative frameworks using graph-based metrics for floorplan accuracy and nDCG-based measures for document relevance, offering reproducible evaluation across various model classes.
• The suite exposes current AI deficits in spatial layout inference and retrieval, providing actionable insights and detailed scoring procedures for future research advancements.

Blueprint-Bench is a suite of rigorous, open research benchmarks designed to evaluate advanced model performance on tasks that require spatial reasoning and multimodal retrieval in real-world contexts. The benchmark suite provides quantitative, reproducible frameworks for assessing spatial intelligence and document retrieval ability in LLMs, image generation models, agent systems, and custom multimodal architectures. Two central vectors define Blueprint-Bench: spatial reconstruction from images (apartment floorplan inference) and retrieval from legacy engineering documentation (region-aware, multimodal search). The following sections detail the benchmark's design, datasets, scoring procedures, comparative results, and emerging research directions.

1. Conceptual Motivation and Core Tasks

Blueprint-Bench addresses persistent gaps in current AI systems’ capacity for spatial reasoning and complex retrieval. The first task, spatial reconstruction, requires inferring accurate two-dimensional floor plans from sets of real-world apartment photographs. This challenges models to move beyond pixel-level or pattern-based recognition and demonstrate the ability to reason about room connectivity, scale consistency, and layout constraints. The second task focuses on multimodal retrieval in the context of legacy engineering archives, demanding layout awareness and region-based information extraction from heterogeneous, poorly indexed document corpora. Blueprint-Bench exposes unique “blind spots” in current state-of-the-art models and establishes a basis for progress measurement across modalities and architectures (Petersson et al., 24 Sep 2025, Seefried et al., 12 Feb 2026).

2. Datasets and Targets

2.1 Spatial Floorplan Benchmark

• Composition: 50 apartments, each with ∼20 RGB photographs comprehensively documenting all interior rooms from multiple viewpoints.
• Ground Truth: Each apartment has a rigorously formatted 2D floor-plan image, derived from official listings and constrained by a set of nine format rules to ensure tractable, automatable evaluation. These rules specify color coding (black walls, green doors, white background, red room centers), enclosure requirements, room labeling (red dots per region), and geometric constraints (straight lines, no furniture or decor, inclusion of walk-in closets only).

2.2 Multimodal Retrieval Benchmark (Editor’s term for (Seefried et al., 12 Feb 2026)’s engineering drawing/document component)

• Corpus: 770,000+ files from a legacy document-management system; working set filtered to 558,926 files (166,125 engineering drawings, 392,801 policies/procedures).
• Formats: Vector CAD, raster images, mixed-mode PDFs, DOCX, and minor archives/videos; domain coverage spans assembly, electrical schematic, and facility plans alongside text-based procedural documents.
• Region Annotations: 750 drawings annotated for four standardized canonical regions: drawing number, data block, parts list/BOM, and revisions block.

3. Evaluation Metrics and Methodologies

3.1 Floorplan Reconstruction Scoring (Petersson et al., 24 Sep 2025)

Model outputs and ground-truth floor plans are compared through derived structures:

• Room Connectivity Graph (G): Nodes correspond to rooms (red dots); undirected edges represent doors connecting adjacent rooms.
• Size Ranking (S): Ordering of rooms by extracted pixel area.
• Score Components (with exact aggregation weights):
  • Edge overlap (S_edge): Jaccard index of door connections (weight 0.5)
  • Degree correlation (S_degree): Pearson correlation of node degrees (0.2)
  • Graph density match (S_density): 1 minus normalized density difference (0.1)
  • Room-count accuracy (S_rooms): 1 minus absolute room-count difference ratio (0.1)
  • Door-count accuracy (S_doors): 1 minus door segment count difference ratio (0.05)
  • Door-orientation similarity (S_orient): Distributional match between horizontal/vertical doors (0.05)
• Composite Score:

$$S_{\text{total}} = 0.50\, S_{\text{edge}} + 0.20\, S_{\text{degree}} + 0.10\, S_{\text{density}} + 0.10\, S_{\text{rooms}} + 0.05\, S_{\text{doors}} + 0.05\, S_{\text{orient}}$$

This yields a normalized similarity in $[0,1]$, quantifying the spatial reconstruction accuracy in terms of structural (not just pixel) similarity.

3.2 Multimodal Retrieval Scoring (Seefried et al., 12 Feb 2026)

• Task: Given 375 expert-crafted queries (vision, text, and cross-modal), return top-$k=3$ ranked items.
• Relevance Grading: Three-point scale $\ell\in\{0, 1, 2\}$ (not, partially, fully relevant), assigned by LLM committee and human raters (human–LLM consensus validated, $\kappa_w\approx 0.31$).
• Primary Metrics:
  • Success@$k$: At least one relevant item ($\ell\geq1$) in top $k$:

  $$\text{Success@}k(q) = \begin{cases} 1, & \max_{i\leq k} \ell(q, d_i) \geq 1 \ 0, & \text{otherwise} \end{cases}$$ - nDCG@$k$: Normalized discounted cumulative gain based on $[0,1]$0 rankings.

• Auxiliary Metrics: MAP@3, Precision@3, Recall@3.
• Region-Level Reranking: Final document score includes a boost for queries whose constraints are satisfied by detected region metadata, with penalties for mismatches.

4. Model Classes and Experimental Baselines

4.1 Floorplan Reconstruction (Petersson et al., 24 Sep 2025)

• LLMs: GPT-5, GPT-5-mini (OpenAI), Claude 4 Opus (Anthropic), Gemini 2.5 Pro (Google DeepMind), Grok-4 (XAI). Floorplans are produced via SVG code output, given image inputs.
• Image Generation Models: GPT-Image/4o Image Generation (OpenAI), NanoBanana/Gemini 2.5 Flash Image.
• Agent Systems: Codex CLI (GPT-5 backend), Claude Code (Claude 4 backend)—capable of multi-step SVG/code refinement.
• Baselines: Human expert (interactive drawing), random (SVG/image without image input).

4.2 Multimodal Retrieval (Seefried et al., 12 Feb 2026)

• Open Vision-LLMs (Baselines): LLaMA 3.2 Vision, LLaVA 1.6 (Mistral 7B), PaLI-Gemma 2, Pixtral 12B, Llama 4 Scout 17B; all provided only single full-page images (no explicit region cues).
• Blueprint System: Custom pipeline integrating region detection (YOLOv8-S, 89.5% [email protected]:0.95), VLM-based OCR, metadata normalization, hybrid sparse+dense retrieval, and lightweight region-level reranking.

5. Empirical Results and Model Deficits

5.1 Floorplan Task Outcomes (Petersson et al., 24 Sep 2025)

• Performance: Across 50 apartments, the random baseline achieves $[0,1]$1. Leading LLMs (GPT-5, Gemini 2.5 Pro, Grok-4) marginally outperform random ($[0,1]$2–$[0,1]$3). Claude 4 Opus, GPT-4o, NanoBanana, and GPT-Image match or underperform random.
• Human Benchmark: Human draw performance on a 12-apartment subset: $[0,1]$4.
• Error Analysis: All model families fail to infer correct room connectivity or scale; image generators exhibit frequent format violations and cannot be reliably scored. Agents with iterative refinement did not show improvements over single-pass code output. LLMs and image models regularly overlook structural constraints (e.g., adjacency, door placement), indicating that repeated image exposure in pretraining does not suffice for complex spatial inference.

5.2 Multimodal Retrieval Results (Seefried et al., 12 Feb 2026)

System	nDCG@3	MAP@3(≥1)	Succ@3
LLaMA 3.2 Vision	0.521	0.497	0.623
LLaVA 1.6 Mistral 7B	0.400	0.378	0.498
PaLI-Gemma 2	0.422	0.395	0.533
Pixtral 12B	0.502	0.486	0.592
Llama 4 Scout 17B	0.519	0.503	0.607
Blueprint	0.626	0.608	0.715

• Gains: Blueprint surpasses the strongest baseline by $[0,1]$5 absolute nDCG@3 ($[0,1]$6\% relative) and $[0,1]$7 absolute Success@3.
• Latency: Blueprint end-to-end time per file is $[0,1]$89.46 s, compared to Llama-4-Scout’s $[0,1]$964.36 s.

5.3 Oracle and Ablation Studies

• Perfect Region Detection/OCR: Oracle region boxes and high-fidelity OCR yield further improvements (nDCG@3=0.699, Success@3=0.780), indicating remaining headroom with algorithmic region and text localization.

6. Strengths, Limitations, and Future Research

• Blueprint-Bench establishes the first robust quantitative frameworks for cross-modal spatial reasoning and retrieval, enabling reproducible, fine-grained comparisons between AI models and humans.
• Spatial Reasoning Deficits: No evaluated AI system approached human-level floorplan inference; performance was insensitive to model scale for spatially compositional tasks, highlighting a fundamental architectural bottleneck.
• Retrieval Performance: Region-aware, pipeline-based systems substantially outperform monolithic VLMs, both in accuracy and efficiency, especially in real-world, poorly labeled archives.
• Instruction Following: Adherence to enforceable output formats remains challenging for image-generation models; instruction non-compliance frequently produces unscorable outputs.
• Emergent Directions: Proposed extensions include enrichment of semantic attributes (room type labeling), geometric shape matching, expansion to outdoor/multifloor layouts, and incorporation of hybrid LLM+3D reconstruction pipelines.
• This suggests that increased specialization in layout/region detection and structured output enforcement may be essential to bridge current performance gaps.

Blueprint-Bench is open source, with data, code, and annotation protocols released for both the spatial reasoning (Petersson et al., 24 Sep 2025) and retrieval (Seefried et al., 12 Feb 2026) tracks, enabling continuous community-driven evaluation and future shared-tasks on spatial intelligence and robust document retrieval in AI.