GameTraversalBenchmark: Evaluating LLM Spatial Planning

• GameTraversalBenchmark (GTB) is a domain-agnostic evaluation framework that transforms planning into structured spatial traversal on 2D grid maps.
• It operationalizes navigation through alphanumeric grids, measuring optimal path efficiency, syntax accuracy, and decision robustness using concrete reward metrics.
• GTB’s comprehensive evaluation uncovers the strengths and limitations of LLMs, highlighting gaps in internal spatial reasoning and action sequencing.

GameTraversalBenchmark (GTB) is a domain-agnostic evaluation framework for assessing planning abilities in LLMs through their performance on structured 2D grid-based game maps. GTB operationalizes planning not through language alone but as the concrete traversal of unseen spatial environments specified as alphanumeric grids. The benchmark was introduced to probe the extent to which LLMs can internally perform stepwise spatial reasoning distinct from memorization or linguistic paraphrasing, positioning itself as a rigorous structural complement to natural-language-based planning benchmarks such as PlanBench (Nasir et al., 2024).

1. Benchmark Motivation and Design Principles

Modern LLMs, despite their success in language generation and understanding, face critical open questions regarding their ability to plan—especially in settings with little overlap to their training distributions. GTB addresses this by requiring agent traversal on character-encoded 2D grid environments populated with walkable/nonwalkable tiles, interactable objects, and sequential objectives unencountered during pretraining. Planning is thus cast as a purely structural, non-linguistic problem, exposing the internal spatial models and decision strategies of LLMs. The benchmark emphasizes two demands: optimality in navigation and robustness against action and syntax errors, with all inputs and outputs formatted as character sequences.

2. Dataset Construction and Task Specification

GTB comprises 150 distinct maps, each ranging from approximately $5\times10$ to about $100\times35$ grid size, generated by the Word2World pipeline (Nasir et al., 2024). Word2World is itself an LLM-driven engine employing GPT-4-Turbo, GPT-4, GPT-3.5, Claude-3-Opus, and Claude-3-Haiku to synthesize for each map:

• A narrative and tileset specification,
• A randomly initialized grid of alphanumeric characters,
• Placement of static environmental elements and interactables,
• Sequential “objectives” specified as coordinates.

For each map $m$ and each objective $i$, the LLM receives:

• The entire character-string grid,
• The agent’s current position,
• The immediate next objective,
• The symbolic legend and valid moveset (U/D/L/R).

The LLM must, in a single emission (“one-shot”), return a discrete sequence of actions traversing from the current start to the target objective.

3. Evaluation Metrics and Reward Scheme

GTB employs a multi-faceted reward structure that integrates:

• Path Optimality: The cost of the agent’s path ($-1$ per step, compared to shortest path via $A^{*}$).
• Generation Robustness: Penalties ($-1$) for each generation error, encompassing both illegal moves and syntactic failures (retries allowed, up to 10).
• Objective Proximity: A graded terminal reward indexed by the Manhattan distance $d$ at episode end:
  • $d=0$ (exact match): $+200$
  • $100\times35$0: $100\times35$1
  • $100\times35$2: $100\times35$3
  • $100\times35$4: $100\times35$5
  • $100\times35$6: $100\times35$7
  • $100\times35$8: $100\times35$9

For each map $m$0, let $m$1 denote the sum of optimal path lengths, and $m$2 the maximal possible sequence errors. The theoretical reward bounds are:

$m$3

$m$4

The realized reward $m$5 is further normalized by subtracting actual steps $m$6 and errors $m$7, yielding the per-map normalized score:

$m$8

The aggregate GTB_Score (GTBS) is the mean normalized score over all $m$9 maps:

$i$0

Key auxiliary metrics include:

• Mean Generation Errors (MGE): $i$1
• Mean Path Length (MPL): $i$2
• Mean Actions Taken (MAT): $i$3, where $i$4 counts all actions including reversals
• Top-$i$5 Proximity Accuracies (Top-0, Top-1, Top-5): Fractions of objectives achieved at varying distance thresholds

4. Model Evaluation Protocols

Benchmarked LLMs encompass GPT-4-Turbo, GPT-4-o, GPT-3.5-Turbo, Claude-3-Opus, Claude-3-Sonnet, Claude-3-Haiku, LLaMa-3-8B/70B, Gemma-7B, and Mixtral-8×7B, together with o1—a large, private reasoning model with internal Chain-of-Thought (CoT)—and the smaller o1-mini.

Prompting for all models employs a static, zero-shot format: a persistent template describing map legend, state, objective, and action grammar. The one-shot mode allows one additional try with feedback (prior actions, last distance), but only two attempts per objective are permitted.

Decoding proceeds via greedy selection, with no explicit CoT outside o1’s internal mechanism. All outputs are discrete, mapped actions; outputs with structural or semantic errors are penalized according to the metrics above.

5. Quantitative Results and Findings

GTB_Score (GTBS) performances (mean ± standard deviation, zero-shot setting) are summarized in the following table:

Model	GTBS (%)	MGE (approx)	Notable Observations
GPT-4-Turbo	44.97 ± 0.22	0.02–0.5	Best among publicly-available LLMs
GPT-4-o	30.95 ± 0.76	–	–
Claude-3-Opus	28.65 ± 0.59	–	–
Random-FP baseline	18.02 ± 0.50	–	Random moves, fixed length
Random-RP baseline	3.04 ± 0.65	–	Random moves, random length
Gemma-7B, etc.	≤16	–	Smaller LLMs perform near random
o1 (single run)	67.84	–	Uses internal CoT
o1-mini	61.36	–	Preliminary

One-shot prompting yields measurable increases, e.g., GPT-4-Turbo rising to 52.07% and LLaMa-3-70B from 11.39% to 15.17%. The highest preliminary score is by o1 at 67.84%, yet even this model falls substantially short of perfect planning. Current state-of-the-art chat-oriented LLMs remain under 50% zero-shot GTBS. Leading models demonstrate occasional illegal actions or syntax errors (0.02–0.5 MGE), while smaller models typically output nearly random action streams—scoring comparably or worse than random agent baselines, implicating a lack of internal spatial reasoning.

6. Methodological Insights and Benchmark Significance

GTB isolates planning as a spatial traversal problem divorced from natural language, challenging LLMs to synthesize spatial models solely from symbolic maps and action grammars. Success on GTB thus reflects genuine internalization of domain-agnostic planning strategies rather than shallow memorization of linguistic forms. The “one-shot” error-correction mechanism also partially simulates realistic agent-environment feedback loops, though in simplified, discrete settings.

A salient finding is that even advanced LLMs exhibit brittle planning: their occasional syntactic or semantic errors and failure to exploit map structure represent core challenges. The marked performance gap between public LLMs and the privately-trained o1 model—yet with o1 not approaching ceiling—suggests persistent limitations in chain-of-thought and environment modeling when domain knowledge is almost wholly novel.

7. Limitations and Directions for Extension

GTB is explicitly scoped to static, turn-based environments. Only four discrete actions (up, down, left, right) are present; real-time dynamics, moving NPCs, and complex interaction affordances (e.g., inventory use, temporally extended subgoal creation) are not modeled. Uniform prompting may disproportionately favor large models; adaptive prompt tuning or few-shot approaches could reveal distinct sample efficiency profiles. Further, the omission of dynamic obstacles and stochastic transitions constrains the benchmark’s ecological validity for certain planning domains.

Future developments proposed include:

• Allowing LLM-driven procedural generation of maps, closing the loop between planning and content creation,
• Adding time-based or adversarial elements (e.g., moving targets, traps),
• Expanding the action space to encompass richer, multi-agent or object-manipulation operations,
• Developing automated prompt curricula to level the playing field between LLMs of varying size/capacity,
• Fine-tuning on held-out GTB tasks to systematically probe generalization to novel map topologies.

This suggests that GTB provides a controlled, quantifiable testbed for the study of LLM spatial planning, highlighting both current model capacities and persisting open problems in structural reasoning, robust action sequencing, and generalization beyond linguistically familiar textures (Nasir et al., 2024).