MazeEval: Benchmark for Spatial Reasoning in LLMs

Updated 3 July 2026

MazeEval is a formal evaluation benchmark designed to isolate and measure the pure maze navigation and spatial reasoning abilities of large language models using minimal, coordinate-based feedback.
The benchmark employs a strict function-calling API with discrete perfect mazes, deterministic transitions, and defined move constraints to ensure objective assessment of spatial decision-making.
Experimental results reveal that while some models (e.g., O3) handle larger maze sizes effectively, most LLMs exhibit catastrophic failures due to excessive looping and diminished cross-linguistic performance.

MazeEval is a formal evaluation benchmark that isolates and quantifies the sequential spatial reasoning abilities of LLMs. It is designed to measure an LLM’s capacity for pure maze navigation—without visual input or rich environment descriptions—by presenting maze traversal tasks using strictly coordinate-based and minimal feedback, enabling explicit assessment of spatial cognition. MazeEval contrasts with earlier text-based navigation and mapping benchmarks by focusing exclusively on decision-making anchored in spatial feedback, not language parsing or map inference. It offers robust protocols for testing cross-linguistic transfer and enables principled model comparison at scale (Einarsson, 27 Jul 2025).

1. Benchmark Definition and Task Setup

MazeEval tasks require LLMs to navigate discrete $n\times n$ “perfect” mazes (single unique solution, no loops) of varying sizes, with $n$ ranging from 5 to 15 in core evaluations and up to 30 or 40 for certain models (notably O3). Each navigation episode is defined as an MDP where

State $s = (x, y)$ : current agent coordinates, with $x,y \in \{0, \ldots, n-1\}$ .
Action $A \in \{\text{north}, \text{south}, \text{east}, \text{west}\}$ .
Transition Function $T(s, a)$ : deterministic, moves to $s'$ if there is no wall in the selected direction, remains at $s$ otherwise.
Observation Model: At each step, the LLM receives

$O(s) = (x, y, d_N, d_S, d_E, d_W)$

where $d_\ast$ reports Manhattan distance to the nearest wall in each direction.

Prompting is conducted via a strict function-calling API; for each turn, the model receives the current state, distances, goal, and full trajectory history, and must produce a direction to move. The environment enforces two termination constraints: visiting any cell more than 10 times results in episode failure, and a global move budget of $n$ 0 prevents pathologically inefficient behaviors.

Maze generation uses a randomized DFS to ensure a single-solution structure, and for every size, five distinct mazes are used to average out instance-specific variance. This protocol allows precise measurement of spatial problem solving under controlled stochasticity (Einarsson, 27 Jul 2025).

2. Evaluation Metrics

MazeEval provides a suite of formal metrics for quantifying distinct aspects of navigation proficiency:

Success Rate:

$n$ 1

This is the primary outcome, measuring pure task completion.

Step Efficiency:

$n$ 2

Computed only for successful episodes, indicating planning optimality.

Loop Rate:

$n$ 3

Tracks failure modes dominated by excessive revisiting rather than naïve exploration.

Invalid Move Rate:

$n$ 4

Highlights deficiencies in local feasibility detection (Einarsson, 27 Jul 2025).

All failures reported in the principal study are due exclusively to looping behavior, not to exceeding move budgets or legal move errors, indicating that catastrophic navigation failure arises from limitations in persistent spatial memory or strategy (Einarsson, 27 Jul 2025).

3. Experimental Results and Comparative Evaluation

In systematic experiments, eight state-of-the-art LLMs were assessed via identical APIs, with all prompts and directions translated for cross-linguistic tests (English and Icelandic). Maze sizes ranged from $n$ 5 to $n$ 6 for all models, with the O3 model evaluated up to $n$ 7.

Key results include:

Models such as OpenAI O3 maintained perfect or near perfect Success Rates up to $n$ 8 (100% for all tested $n$ 9), while Claude Opus 4, Claude Sonnet 4, Gemini 2.5 Flash, Gemini 2.5 Pro, and GPT-4o all failed beyond $s = (x, y)$ 0 (dropping to 0% at $s = (x, y)$ 1 for most).
All non-O3 model failures were traceable one-to-one to excessive looping, with Loop Rates $s = (x, y)$ 2 and no recorded over-budget terminations.
Median Efficiency for successful O3 runs is $s = (x, y)$ 3 across all sizes; other models drop to $s = (x, y)$ 4 as $s = (x, y)$ 5 increases, and variance grows with maze complexity.
In Icelandic, every model (except O3) coped with mazes $s = (x, y)$ 6– $s = (x, y)$ 7 sizes smaller than in English, confirmed by Wilcoxon signed-rank ( $s = (x, y)$ 8, Cohen’s $s = (x, y)$ 9) (Einarsson, 27 Jul 2025).

The empirical pattern reveals a sharp scaling wall: for most LLMs, spatial competence collapses catastrophically beyond $x,y \in \{0, \ldots, n-1\}$ 0, regardless of the language used. Performance in low-resource languages degrades much faster, suggesting an emergent effect of training corpus frequency and composition, not an underlying universal spatial schema in current architectures.

Model	Max $x,y \in \{0, \ldots, n-1\}$ 1 (English)	Max $x,y \in \{0, \ldots, n-1\}$ 2 (Icelandic)
Claude Opus 4	11	8
Claude Sonnet 4	12	8
Gemini 2.5 Flash	8	8
Gemini 2.5 Pro	11	8
GPT-4o	8	8
GPT-4o-mini	–	–
GPT-4.1-mini	8	7
O3	$x,y \in \{0, \ldots, n-1\}$ 3	$x,y \in \{0, \ldots, n-1\}$ 4

Table: Maximum size of maze ( $x,y \in \{0, \ldots, n-1\}$ 5) solved in at least one instance per model/language (Einarsson, 27 Jul 2025).

4. Linguistic Transfer and Emergent Limitations

MazeEval critically explores cross-linguistic transfer by translating direction words and prompts into Icelandic, keeping numeric and structural feedback identical. Models universally display a pronounced drop in navigation success for Icelandic, with median maximum solvable maze sizes reduced by 3–4 levels relative to English. Only O3 shows no measurable performance loss.

This strongly supports the interpretation that spatial reasoning in LLMs is not based on modality-independent, language-agnostic mechanisms. Rather, the cognitive mapping required for navigation appears to emerge through linguistic patterns and distributional regularities in the training data. Lower-resource languages lacking sufficient corpus coverage yield substantial drops in spatial problem-solving. This finding has direct implications for robustness of LLM-powered systems in global, multilingual deployments (Einarsson, 27 Jul 2025).

5. Interpretive Analysis and Failure Modes

The dominant failure mode is catastrophic looping: the model, despite full access to coordinate histories and wall distances, revisits the same cell repeatedly (at least 10 times), indicating failure to encode or exploit visited state information over long horizons. Notably, no failures in the core set are due to move legality (all models respected local wall constraints), ruling out a basic perception or parsing deficit.

Models do not appear to construct or retain internal “cognitive maps” of the traversed structure, a finding consistent with prior diagnostic results showing a rapid collapse in sequential spatial memory beyond relatively small maze dimensions. Excessive looping in the presence of full information implies that persistent memory or trajectory abstraction is not spontaneously realized in current transformer models over these task scales (Einarsson, 27 Jul 2025).

A plausible implication is that the scaling wall—behavioral collapse by $x,y \in \{0, \ldots, n-1\}$ 6—reflects not merely the combinatorial explosion of planning but a fundamental inefficiency (or absence) of memory-binding across long trajectories. The success of O3 up to $x,y \in \{0, \ldots, n-1\}$ 7 suggests that architectural or training improvements, e.g., explicit memory modules or increased context window, can partially overcome this barrier.

6. Methodological Directions and Future Benchmarks

The MazeEval framework motivates several methodological extensions to more fully capture the space of spatial reasoning challenges:

Multimodal inputs: Introducing “visualization-of-thought” (VOT) prompting, where LLMs are asked to generate, update, or interpret diagrammatic representations, potentially augments internal map construction and explicit memory (Einarsson, 27 Jul 2025).
Memory-Augmented Architectures: New designs, inspired by hippocampal grid/place cell systems, could introduce explicit working-memory submodules, directly addressing the observed failures in persistent state tracking.
Prompt Engineering: Adopting subroutine scaffolding or partial map revelation to promote route-planning and backtracking strategies may extend the size regime in which LLMs remain effective.
Extended domains: Adding settings involving continuous or 3D environments, as well as multi-agent coordination, would generalize MazeEval’s relevance to embodied AI and autonomous robotics.
Cross-benchmark comparison: Positioning MazeEval alongside graph-based navigation QA (MANGO (Ding et al., 2024)), visually-encoded navigation (AlphaMaze (Dao et al., 20 Feb 2025)), and multi-scale diagnostic probing (Lost in Aggregation (Jiang et al., 20 Jun 2026)) situates its specificity and highlights open research directions.

The authors recommend leveraging MazeEval for principled, neuroscience-inspired, and cross-modal innovations to advance LLM spatial intelligence and ensure reliable navigation capabilities in diverse settings.

MazeEval occupies a unique intersection within the landscape of maze-based evaluation tools:

Contrast to MANGO: Where MANGO (Ding et al., 2024) focuses on QA over walkthroughs and tests mapping inference in narrative text, MazeEval centers on sequential, decision-time spatial planning with minimal environmental feedback.
Relation to AMaze: AMaze (Godin-Dubois et al., 2024) provides a pipeline for generalization analysis and curriculum design, with human-in-the-loop protocols and explicit visual sign complexity. MazeEval, in contrast, carries a function-calling API with no signs, symbols, or visual ambiguity—its focus is pure spatial reasoning disentangled from perception.
Multi-Scale Diagnostic Benchmarks: The “Lost in Aggregation” framework (Jiang et al., 20 Jun 2026) decomposes navigation into Fine/Meso/Macro cognitive subproblems and reveals that LLMs retain isolated competence in local passability and global orientation well beyond the point where full-sequence navigation collapses—a phenomenon also visible in MazeEval’s error profiles.
AlphaMaze: AlphaMaze (Dao et al., 20 Feb 2025) demonstrates that supervised fine-tuning on tokenized visual maze data, followed by reinforcement learning (GRPO), can specifically boost success rates on MazeEval-like tasks, with explicit chain-of-thought and self-correction emerging as LLMs adapt to the navigation setting.

Collectively, these findings position MazeEval as a benchmark that isolates and exposes foundational spatial deficits in LLMs, grounds promising avenues for architectural and training improvements, and motivates future research on closing the gap between linguistic and spatial intelligence in artificial agents (Einarsson, 27 Jul 2025).