EvoEmpirBench: Dynamic Spatial Benchmark

• EvoEmpirBench is a dynamic spatial-reasoning benchmark designed to test long-horizon planning, memory use, and adaptive strategy in partially observable environments.
• It features two interactive tasks—locally observable maze navigation and match-2 elimination—that require exploration, hazard management, and resource allocation under changing conditions.
• The framework integrates Agent-ExpVer, which verifies learning through experience replay and truth induction to enhance policy updates and reduce catastrophic forgetting.

Searching arXiv for the primary EvoEmpirBench paper and closely related benchmark-evolution work. I’ll retrieve the relevant arXiv records to ground the article in the primary source and nearby benchmark-evolution literature. EvoEmpirBench is a dynamic spatial-reasoning benchmark introduced to evaluate whether LLMs can reason and adapt in environments where perception is only locally observable and where every action can change the state of the world. The benchmark is paired with Agent-ExpVer, an online learning and memory framework centered on “experience + verification.” In contrast to static or globally observable spatial benchmarks, EvoEmpirBench is designed to test long-horizon planning, memory use, survival, and adaptive strategy revision under uncertainty, using two interactive tasks: locally observable maze navigation and match-2 elimination (Zhao et al., 16 Sep 2025).

1. Conceptual scope and motivation

The paper positions EvoEmpirBench, abbreviated “EEB,” as a response to a limitation in existing spatial reasoning benchmarks: many are static or globally observable, so a model can solve them by reasoning over a complete map or a fixed puzzle state without maintaining a policy over time. The benchmark is therefore designed around repeated observation, action, and belief updating under partial observability, with environment dynamics tightly coupled to decision-making (Zhao et al., 16 Sep 2025).

This motivation is also framed as a benchmark-validity issue. Static benchmarks can saturate quickly, can be vulnerable to contamination, and may fail to measure long-horizon planning, memory use, or adaptive strategy revision. The paper explicitly contrasts EEB with SmartPlay, CrossWordBench, GameArena, Agent-Pro, and AutoManual, arguing that these alternatives either lack true environment mutation, lack partial observability, focus on narrow game-theoretic or task-completion objectives, or do not require continual adaptation. EvoEmpirBench is intended to close that gap by requiring exploration, hidden-structure inference, and adaptation to changing rules, hazards, and resource constraints (Zhao et al., 16 Sep 2025).

A useful way to situate EEB is to distinguish two meanings of “benchmark evolution.” Some recent work evolves benchmarks by rewriting or refreshing test items, as in EvoEval for coding (Xia et al., 2024), AutoEvoEval for close-ended evaluation data (Wu et al., 30 Jun 2025), and ArenaBencher for automatic benchmark updates (Liu et al., 9 Oct 2025). EvoEmpirBench instead evolves the interaction state during evaluation itself: the environment changes after each action, so the benchmark’s difficulty arises from dynamic execution rather than only from curated item variation (Zhao et al., 16 Sep 2025).

2. Benchmark tasks and environment structure

EvoEmpirBench contains two games, both dynamic, partially observable, and multi-objective, but designed to stress different forms of spatial intelligence (Zhao et al., 16 Sep 2025).

Task	Environment	Primary stressors
Locally observable maze navigation	9×9 grid world	exploration, survival, hazard management, item-conditioned planning
Match-2 elimination	8×8 board	combinatorial planning, step efficiency, resource allocation

The maze game is a 9×9 grid world with local observability. The agent does not see the entire map at once and must explore to reveal new tiles. Its objectives are to collect five gold coins, explore efficiently, and reach the goal while staying alive. The action space is discrete and consists of 12 movement actions: actions 0–2 are UP by 1/2/3, 3–5 are DOWN by 1/2/3, 6–8 are LEFT by 1/2/3, and 9–11 are RIGHT by 1/2/3 (Zhao et al., 16 Sep 2025).

Difficulty is organized into three levels. Easy uses a simple 9×9 map with no monsters and five coins. Medium adds two moving monsters. Hard adds those monsters together with four special items: Pickaxe, Iron Sword, Magnet, and Key. These items change the environment dynamics. The pickaxe can break obstacles, the sword allows unlimited combat with monsters, the magnet collects coins within a neighborhood, and the key is required to exit. The paper emphasizes that planning is therefore genuinely dynamic: obtaining an item or destroying a barrier changes what is possible later, while monster movement makes the environment non-stationary (Zhao et al., 16 Sep 2025).

Maze evaluation is multi-metric rather than binary. Reported metrics are Suc.Rate, A.Score, A.steps, A.Explor, A.Gold, Rem.HP, A.kills, and A.Barr. These jointly measure episode completion, cumulative reward, step efficiency, exploration, gold collection, remaining health, defeated enemies, and destroyed barriers (Zhao et al., 16 Sep 2025).

The match-2 elimination game uses an 8×8 partially observable dynamic board. The board contains tiles of four colors, A–D. The agent eliminates connected components of the same color when the component size is at least 2, and rewards increase with match size. The prompt example gives the scoring rule as

$$\text{score} = \text{tiles} \times 5 + 3 \times \max(0,\text{tiles}-2).$$

After each elimination, remaining blocks fall downward and new random tiles spawn at the top, so the board changes continuously. The agent also has one-use props—row clear, column clear, bomb, and hammer—each with an associated point cost. The output format is JSON with an action type and either a position or row/column index: $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$ Difficulty is controlled by step budget and target requirements: easy gives 15–18 steps with per-color targets around 8–12, medium gives 12–15 steps with targets around 12–16, and hard gives 10–13 steps with targets around 16–20 (Zhao et al., 16 Sep 2025).

Match-2 is evaluated with Suc.Rate, A.Score, R/M.S, Score/Step, Clear/Step, and API Eff. The paper treats this task as a test of optimization under tight resource constraints: the agent must satisfy elimination targets before steps run out, decide when to use expensive props, and balance immediate score against long-term completion (Zhao et al., 16 Sep 2025).

3. Agent-ExpVer and the memory architecture

A central contribution of the work is Agent-ExpVer, described as an online learning and memory framework built around “experience + verification.” The framework is organized into three components: GeoLink Agent, InsightForce Agent, and TruthWeaver Agent (Zhao et al., 16 Sep 2025).

GeoLink Agent interacts with the environment and collects trajectories. At timestep $t$, the agent observes a partial state $\mathbf{s}_t$, chooses an action $a_t$ according to policy $\pi_t$, and the environment transitions according to $\mathcal{E}$: $$a_t \sim \pi_t(\mathbf{s}_t), \qquad (\mathbf{s}_{t+1}, r_t) \sim \mathcal{E}(\mathbf{s}_t, a_t).$$ Over an episode, the interaction history is stored as

$$\mathcal{H}_{0:T} = \{(\mathbf{s}_0,a_0,r_0), \ldots, (\mathbf{s}_T,a_T,r_T)\}.$$

This formalization makes clear that the benchmark is built around online interaction histories rather than single-shot question answering (Zhao et al., 16 Sep 2025).

InsightForce Agent abstracts a completed trajectory and its outcome metrics $\mathbf{m}$ into a subjective experience: $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$0 That experience is added to an experience memory $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$1. The paper’s terminology is notable: the memory object is not a neutral transcript, but a “subjective experience” derived from trajectory summarization and performance outcomes (Zhao et al., 16 Sep 2025).

TruthWeaver Agent manages the transition from provisional experience to reusable knowledge. It stores, merges, removes, and manages reusable truths, and is explicitly described as a de-duplication and consolidation mechanism intended to prevent memory bloat and conflicting beliefs. This gives Agent-ExpVer a dual-memory structure: a provisional experience store and a curated truth repository (Zhao et al., 16 Sep 2025).

The practical prompt design is part of this mechanism. For maze navigation, prompts prioritize survival over reward, present a local map with symbols such as A, G, C, #, ?, and ., and inject learned truths as “Training Knowledge.” For match-2, prompts specify the board, score, steps remaining, inventory, targets, and legal action JSON schema. The ExpVer prompt asks the model to analyze a session, list strengths and weaknesses, and organize the knowledge base by removing duplicates and merging similar entries. This indicates that the memory system operates not only through data structures but also through prompt-mediated control of reasoning (Zhao et al., 16 Sep 2025).

4. Verification, truth induction, and policy update

The defining feature of Agent-ExpVer is that memory is not accepted merely because it is generated. Instead, a candidate experience is validated behaviorally. After an episode is summarized into $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$2, the system replays the same episode with $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$3 injected into the prompt or policy context. If the replay passes the level and the score improves, the experience is promoted into the truth repository: $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$4 Here $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$5 denotes that the level was passed, and $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$6 denotes improvement over the previous score. The paper treats this as a key design choice: memory induction is tied to measurable behavioral gain rather than purely textual plausibility (Zhao et al., 16 Sep 2025).

The current policy is formed by combining the base prompt $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$7 with all stored truths: $$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$8 The framework also defines a policy improvement score

$$\{"action": \{ "type": \text{"eliminate"|"row"|"col"|"bomb"|"hammer"}, \dots \}\}.$$9

If $t$0, the most recent update is reverted and abstraction is repeated. The paper therefore presents truth induction as self-correcting: the agent keeps only memory updates that improve performance on average (Zhao et al., 16 Sep 2025).

This verification logic is conceptually close to recent work on self-evolving memory in task streams, but the emphasis is different. Evo-Memory evaluates whether agents can retrieve, integrate, update, and improve memory over sequential datasets (Wei et al., 25 Nov 2025). EvoEmpirBench instead binds memory evolution to dynamic spatial interaction with hazards, tools, and partial observability, and validates memory through replay-based performance gains rather than only through downstream task scores (Zhao et al., 16 Sep 2025).

A potential misconception is to treat Agent-ExpVer as ordinary retrieval augmentation. The paper does not describe it as passive memory lookup. Its components explicitly summarize episodes, validate candidate experiences, consolidate truths, and revert harmful updates. This suggests a stronger form of test-time adaptation than simple retrieval of past context (Zhao et al., 16 Sep 2025).

5. Experimental protocol and empirical findings

The experimental study covers a broad set of proprietary and open-source models on both tasks: GPT-4, GPT-4.1, Gemini-2.0-flash, Gemini-2.5-flash-preview, Claude-3-5-sonnet, Claude-3-7-sonnet, Grok-3, Deepseek-V3, Llama-3.1-8B/70B-instruct, Qwen2.5-7B/14B/32B/72B-instruct, and Qwen3-30B-a3B. Each task contains 90 instances total, with 30 per difficulty level. The paper reports aggregate “all levels” results and separately evaluates Agent-ExpVer variants on GPT-4.1, Gemini-2.5-flash-preview, Claude-3-7-sonnet, and Qwen2.5-32B-instruct (Zhao et al., 16 Sep 2025).

In maze navigation, the benchmark reveals a substantial gap between humans and current models. The human baseline is 90% success with mean score 2914.67. Among non-human baselines, GPT-4.1 and Claude-3-7-sonnet reach around 73–69% success, Grok-3 reaches 67.78%, and Deepseek-V3 reaches 61.11%. Smaller or weaker models are markedly worse; Llama-3.1-8B-instruct achieves 23.33% success and a negative score. The paper reports that Agent-ExpVer improves maze results consistently, with average gains of about +5.6% in success rate and +9.5% in mean score. GPT-4.1-ours reaches 78.89% success and 2805.67 score, approaching the human baseline most closely among tested agents. Qwen2.5-32B-instruct improves from 42.22% success to 54.44% and from 1122.22 to 1532.33 score. Claude-3-7-sonnet-ours reaches 72.22% success and 2858 score (Zhao et al., 16 Sep 2025).

The match-2 task is described as even more difficult. Baseline average success across models is only 33.7%, compared with 86.67% for humans. Among baselines, Deepseek-V3 reaches 37.78%, Qwen3-30B-a3b 38.89%, Claude-3-7-sonnet 41.11%, Grok-3 42.22%, and GPT-4.1 40.00%. Agent-ExpVer yields gains of roughly +13.3% success rate, +9.8% more steps remaining, +1.2 score per step, and +0.8 clears per step. GPT-4.1-ours becomes the top non-human agent at 53.33% success and 17.31 R/M.S. The paper also notes that Gemini-2.5-flash-preview-ours keeps the same success rate as its baseline but slightly improves score, and in one comparison surpasses the human benchmark score on the task (Zhao et al., 16 Sep 2025).

The reported results are interpreted as evidence that mainstream models struggle particularly with harder, partially observable, long-horizon settings. The benchmark’s design is intentionally compound: the action that maximizes immediate score is not always the one that best supports final completion. This makes EEB a test of policy quality under evolving state, not merely of static pattern recognition (Zhao et al., 16 Sep 2025).

6. Ablations, interpretation, and relation to adjacent benchmark paradigms

The ablation studies emphasize the importance of TruthWeaver. When TruthWeaver is removed, performance declines on both maze and match-2 for Qwen2.5-32B and GPT-4.1. For GPT-4.1, maze success drops from 78.89 to 77.78 without TruthWeaver and match-2 success drops from 53.33 to 48.89; the full system also performs better in score and step efficiency. For Qwen2.5-32B, the full system improves maze score from 1272.11 without TruthWeaver to 1532.33, and match-2 success from 35.56 to 41.57. The paper attributes this to reduced catastrophic forgetting and more stable belief consolidation (Zhao et al., 16 Sep 2025).

The paper also compares EEB to simplified settings such as full-vision maze and no-props match-2. Relaxing partial observability or tool constraints makes the tasks substantially easier. GPT-4.1 maze success rises from 73.33% to 93.33% under full vision, and Qwen2.5 likewise improves. In match-2, removing props changes the challenge profile and generally makes the task less demanding. These controls support the claim that partial observability, environment mutation, and constrained tool use are the mechanisms that create the benchmark’s adaptive-reasoning burden (Zhao et al., 16 Sep 2025).

Qualitatively, the paper argues that Agent-ExpVer changes strategy rather than only final scores. In maze, learned truths make agents more risk-aware, preserving more health and avoiding dangerous but potentially rewarding routes. In match-2, the learned policy becomes more deliberate and efficient, improving clearance per step and API efficiency. The paper also notes a transient failure mode: early in training, the system can overgeneralize from lucky episodes and temporarily degrade, but after 3–4 rounds of verification and truth refinement, performance recovers and improves. This suggests that memory induction is not monotone; verification and reversion are necessary stabilizers (Zhao et al., 16 Sep 2025).

Within the broader landscape of benchmark research, EvoEmpirBench occupies a distinct position. EvoEval shows that coding leaderboard rankings can change substantially when seed problems are evolved into different targeted domains (Xia et al., 2024). ArenaBencher updates test cases while preserving the original objective and selecting candidates that expose shared weaknesses across multiple models (Liu et al., 9 Oct 2025). AutoEvoEval introduces 22 interpretable atomic evolution operations and multi-round compositions for close-ended LLM evaluation (Wu et al., 30 Jun 2025). By contrast, EvoEmpirBench does not primarily evolve benchmark items; it constructs environments in which the benchmark state itself evolves during interaction. A plausible implication is that EEB extends benchmark evolution from dataset curation into environment design, where difficulty emerges from local perception, action-conditioned state change, and verified memory reuse rather than only from rewritten prompts or transformed examples (Zhao et al., 16 Sep 2025).

The main limitation acknowledged by the paper is that performance remains strongly dependent on model capacity: smaller models lag behind, and even the strongest systems remain below the human baseline on the hardest settings. The authors therefore position EvoEmpirBench as a platform for future work on lightweight adaptation, temporal reasoning, multi-agent collaboration, and richer truth-induction mechanisms (Zhao et al., 16 Sep 2025).