MazeBench: Spatial Reasoning Benchmark

• MazeBench is a controlled benchmark for evaluating multimodal models on spatial reasoning tasks using procedurally generated mazes.
• It offers fine-grained control over maze characteristics such as grid size, wall density, and obstacle configuration to rigorously test model performance.
• Experimental analyses reveal that high model accuracy does not equate to human-like spatial planning, highlighting limitations in current architectures.

MazeBench is a controlled benchmark for evaluating multimodal models on visual spatial reasoning and planning tasks involving procedurally generated mazes. It is designed to investigate whether high accuracy on such visual tasks reflects genuine spatial planning or merely token-level brute-force search. MazeBench provides fine-grained control over maze characteristics and difficulty, enabling rigorous, scalable assessment of LLMs and vision-LLMs from several major providers. Its quantitative and qualitative analyses reveal significant gaps between model performance and human-like spatial understanding, highlighting limitations in current multimodal model architectures and evaluation metrics (Salgado, 27 Mar 2026).

1. Procedural Generation and Difficulty Structure

MazeBench comprises 110 pixel-art mazes, each algorithmically generated to ensure control and reproducibility. Mazes are organized into nine difficulty groups, differing in grid size, wall density, trap configuration, border presence, reachability, and color palette:

• Grid Sizes: Each maze is an $r \times c$ grid with $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$ and an additional ultra-hard group with $r = c = 20$.
• Obstacle Parameters: Wall density $d \in [0, 0.55]$ specifies the fraction of internal cells as walls. Trap count $t \in \{0, \dots, 25\}$ introduces visually distinct impassable tiles. Optional border walls provide an outer impassable ring.
• Path Constraints: The start and goal are placed on opposite maze borders, separated by Manhattan distance at least $\bigl\lfloor(r + c) / 3\bigr\rfloor$. Post-placement, a BFS (Breadth-First Search) reachability check ensures connectivity, except purposely unreachable group E.
• Groups Overview:
  • A: Diagnostic (empty/near-empty, straight paths)
  • B: Grid scaling with fixed density ($25\%$), various sizes
  • C: Varying wall density with fixed size ($9\times9$)
  • D: Trap ablation (matched with/without traps)
  • E: Unreachable (no valid path)
  • F: Border wall ablation
  • G: Combined hard (high density, traps, borders, large size)
  • H: Palette stress (render variants)
  • X: Ultra-hard ($20\times20$, high density, long paths)

2. Task Definition and Evaluation Metrics

The evaluation task requires a model to parse a PNG image of a maze and then output, in JSON, both a spatial grid description and an explicit shortest path from start to goal. The required output format is: $r = c = 20$2 with $d_i \in \{\text{U, D, L, R}\}$. Criteria for a maze to be marked "solved":

1. Model “reachable” prediction matches ground truth.
2. Reported path length matches the true shortest path length ($$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$0).
3. Path is one of the annotated shortest paths.

Primary metrics:

• Solve Rate: $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$1, for $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$2 test mazes.
• Reachability Accuracy: $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$3.
• Average Path-Length Error: $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$4.
• Token Consumption per Solved Maze: $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$5, with $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$6 being the sum of “thinking” and “output” tokens.

3. Evaluation Methodology and Model Protocols

MazeBench evaluates 16 configurations from OpenAI, Anthropic, Google, and Alibaba, each varying in architecture and the allocated “reasoning budget” (prompting style/extensiveness):

• OpenAI: gpt-5.4, gpt-5.4-mini, three reasoning effort levels (“none,” “low,” “medium”).
• Anthropic: Claude Opus 4.6, Sonnet 4.6, Haiku 4.5, each with “no-thinking” and “low-effort.”
• Google: Gemini 3.1 Pro Preview, Gemini 3 Flash Preview (default).
• Alibaba: Qwen 3.5 Plus, Qwen 3.5 Flash (default).

Protocol for each maze and model:

1. Submit a fixed prompt (“no-tools, JSON-only”) with a base64 image.
2. Enforce temperature = 0.0; for Anthropic, temperature = 1.0 if “thinking” is enabled.
3. Retry up to twice for JSON parsing errors.
4. Record solution status, reachability accuracy, token counts (including “thoughtsTokenCount” where available), and latency.

4. Experimental Results and Analysis

Performance on the 100-maze core set (Groups A–H) reveals substantial variation in both efficiency and approach across models:

Model	Effort	Solve %	Reach %	Tokens / Solve
GPT-5.4	medium	91	95	2,913
GPT-5.4	low	85	92	1,710
Gemini 3.1 Pro	default	79	86	9,250
Gemini 3 Flash	default	53	82	15,171
GPT-5.4-mini	low	49	89	3,105
Claude Sonnet 4.6	none	6	70	14,313
Claude Sonnet 4.6	low	2	60	—

Notable findings:

• Reasoning Effort: For GPT-5.4, increasing “effort” boosts solve rate from $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$7 to $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$8 and $$(r, c) \in \{5, 7, 9, 11, 13\} \times \{5, 7, 9, 11, 13\}$$9 (diminishing returns for “medium”).
• Ultra-hard Mazes: On Group X ($r = c = 20$0, paths 28–42 steps), GPT-5.4 (medium) solved 3/10 (all with $r = c = 20$1); for longer paths, models hit their 8,192 token limits and failed.
• Token Efficiency: Marked disparity: GPT-5.4 (low) used ∼1.7K tokens per solved maze, Gemini 3 Flash ∼15.2K, with Anthropic’s Claude models exceeding 22K per (mostly incorrect) attempt.

All top-performing models operated via a common two-stage process: (1) translating the image into a text grid (“Row 0: W, ., ...”) and (2) performing a token-wise, serial path enumeration akin to BFS in prose. Even with “visual-only” prompts, models reverted to this text grid construction and search.

Ablation on Claude Sonnet 4.6 demonstrates the effect of supplying ground-truth grid input: Solve rate jumps from 6% (image input, no thinking) to 80% (grid input, low effort), and tokens needed per solve fall from 14,313 to 3,222—demonstrating that vision is a primary bottleneck, not search.

5. Strategic, Cognitive, and Computational Implications

MazeBench substantiates that high solve rates do not equate to human-like spatial planning. Leading models, despite high accuracy, do not employ continuous spatial abstraction but instead execute exhaustive, serial enumeration in language/token space, and do not “trace” a path holistically.

• Token Budget as a Limiting Resource: When the required enumeration path exceeds model context length, failure ensues, regardless of visual clarity. For ultra-long or complex mazes, models default to “unreachable.”
• Robustness to Prompting: Instruction compliance remains weak: Models are unable to abstain from grid construction or BFS-like search strategies, even when explicitly instructed not to do so.

A plausible implication is that current multimodal architectures are fundamentally constrained in spatial cognition by their reliance on language-based enumeration, rather than constructing or manipulating spatial representations analogous to those used by humans.

6. Limitations and Directions for Future Research

MazeBench exposes several limitations of the present generation of multimodal models and their associated benchmarks:

• Metric Limitations: Standard accuracy or solve-rate metrics fail to capture qualitative deficiencies in spatial reasoning and token efficiency.
• Benchmark Boundaries: The benchmark currently targets closed-source LLMs; open-weight models and direct human time baselines are not included, limiting generalization about spatial planning across architectures and species.
• Future Avenues:
  • Develop new benchmarks emphasizing strategy selection and token efficiency rather than correctness alone.
  • Design tasks requiring abstraction, compositional visual reasoning, or spatial inference beyond square-grid topologies.
  • Integrate human and open-model baselines to more accurately characterize human-like visual planning and its computational correlates.
  • Explore model modifications or hybrid tool use to enhance spatial abstraction and tractable long-horizon planning.

MazeBench is available at https://github.com/alrod97/LLMs_mazes and is recommended for standardizing spatial reasoning assessment in multimodal systems (Salgado, 27 Mar 2026).