HeroBench LLM Planning Benchmark

• HeroBench is a benchmark that rigorously assesses LLM capabilities in planning extended, interdependent task sequences within RPG-inspired virtual worlds.
• It challenges models with tasks such as resource gathering, skill leveling, crafting, and combat, emphasizing structured reasoning and long-term dependency tracking.
• The platform uses precise simulation metrics and exhaustive validation to reveal performance gaps in current autonomous planning approaches.

HeroBench is a benchmark specifically designed to evaluate the capabilities of LLMs in long-horizon planning and structured reasoning within RPG-inspired virtual worlds. Unlike prior benchmarks that rely on abstract or low-dimensional algorithmic tasks, HeroBench embeds combinatorial complexity, layered dependencies, and end-to-end validation in a simulated environment. The platform systematically challenges models to devise extended, interdependent action sequences involving resource acquisition, skill learning, crafting dependency navigation, and combat strategy, thereby offering an exhaustive testbed for examining the robustness of LLM-based autonomous planners (Anokhin et al., 18 Aug 2025).

1. Motivation and Objectives

Existing LLM benchmarks such as Blocksworld and various abstract algorithmic puzzles are limited in their ability to assess long-term dependency tracking, strategic reasoning over extended horizons, and real-world planning analogues. Real-world planning, as instantiated in HeroBench, demands high-level strategy formulation, nuanced numerical reasoning (for example, integrating damage and resistance calculations), precise management of resource and skill dependencies, and the decomposition and verification of multi-stage plans.

HeroBench addresses these deficiencies by embedding tasks within ArtifactsMMO, an RPG-inspired virtual domain with explicitly controlled complexity, precise evaluation metrics, and exhaustive end-to-end solutions validation. Its principal objective is to rigorously quantify and analyze the long-horizon planning and structured reasoning proficiency of LLMs. The benchmark’s design encompasses both crafting-intensive and combat-oriented tasks, each characterized by multiple interdependent subtasks and compositional difficulty (Anokhin et al., 18 Aug 2025).

2. Benchmark Structure and Task Suite

HeroBench defines a comprehensive suite of tasks, categorized by the dominant reasoning and execution demands:

• Resource Gathering: Actions such as mining, woodcutting, and hunting to collect primary resources (ore, wood, monster drops).
• Skill Acquisition (optional): Leveling up professions (e.g., mining, woodcutting) from level 1 to a task-dependent tier.
• Crafting: Sequential creation of items via traversal of a directed acyclic graph (DAG) reflecting compositional item dependencies.
• Combat: Equipping minimal viable gear and defeating a target monster in a turn-based simulation.

Task Generation Pipeline: Tasks are initialized by sampling a target monster with given difficulty, simulating combat to determine the minimal set of gear required for victory, decomposing missing gear into crafting and resource-gathering chains, and validating auxiliary item distractors and noise recipes. Extended tasks require sequencing profession leveling steps from base skill levels and infuse plausible but uncraftable distractor items ("noise"), heightening demand for structured plan construction.

Complexity Metrics:

• Planning Horizon ($H$): Approximate length of the required action sequence, scaling with the total crafting difficulty score $D_{total}$.
• State–Action Graph Size ($|S|$): For HeroBench, $|locations|=70$, $|items|=208$, $|monsters|=25$, supporting exponential complexity.
• Action Dependency Depth ($D$): Maximum depth of the crafting DAG or required chain of thought in combat.

3. Simulated Environment and Dynamics

HeroBench models a rigorously specified RPG environment:

• State Representation: $$s = (\text{loc}, c_L\ \text{stats}, \text{inventory}, \text{equipped gear}, \text{skill levels})$$ with $loc \in \{1,\ldots,70\}$.
• Action Space:
  • move_to($i$)
  • gather($D_{total}$0)
  • craft($D_{total}$1)
  • equip($D_{total}$2)
  • attack($D_{total}$3)
• Transition Dynamics: Deterministic transitions update the player’s inventory, progress through crafting, and resolve combat via HP/stat adjustments and loot drops. Crafting and combat dynamics enforce prerequisites and track resource consumption with a high degree of verifiability.
• Reward Structure:
  • Terminal reward $D_{total}$4 for goal achievement (item crafted or monster defeated), $D_{total}$5 otherwise.
  • Progress Score: Provides normalized (0–100) partial credit for incremental subtask completion (e.g., gathering, crafting intermediates, equipping, defeating secondary monsters).

4. Dataset Construction and Validation

• Task Dataset: The complete dataset contains 844 base tasks spanning nine difficulty brackets (base difficulty $D_{total}$6 ranging from 2 to 97). The primary experimental set comprises 180 tasks, evenly distributed across brackets. Optional extended tasks incorporate skill leveling and noise items.
• Diversity and Validation: The dataset is validated via simulate($D_{total}$7) calls, ensuring correctness of minimal gear selection and crafting chains. It exhibits substantial compositional diversity: 208 unique items, 17 resource types, 25 monsters, and spatial coverage over 70 locations. Crafting DAGs vary in depth between 2–8 steps.

5. Evaluation Metrics

HeroBench utilizes multiple quantitative metrics:

Metric	Definition	Notes
Success Rate (SR)	$D_{total}$8	Primary endpoint
Average Plan Length ($D_{total}$9)	$|S|$0 where $|S|$1: plan for task $|S|$2	Plan complexity
Progress Score	Subtask completion (0–100), mean ± SD across tasks	Sensitive to intermediate progress
Token Efficiency	Average output token count per task	Compactness of model outputs
Composite Metrics	SR/Tokens, Score/Tokens	Efficiency-adjusted scores

6. Empirical Findings and Error Taxonomy

Evaluation of 25 state-of-the-art LLMs, including major open-source and proprietary models (GPT-4.1, GPT-5, Gemini-2.5-pro, Grok-4), revealed unusually wide performance disparities:

• Success Rate Curve: For base tasks (N=180), GPT-4.1 yielded SR of 31.7%, while Grok-4 and GPT-5 achieved 91.7% and 83.9% respectively. Progress scores and token usage scale with planning complexity.
• Extended Tasks (Skill and Noise): Only top-tier models (Grok-4, GPT-5) demonstrated any substantial proficiency when profession leveling and distractor items were introduced, indicating significant gaps in structured multi-stage planning.

Error Analysis: Failure modes were classified by:

• High-Level Plan Errors: Omissions in identifying required intermediate items or monsters.
• Execution Errors: Quantitative miscalculations (incorrect resource counts, misallocation of gathering, improper loops).
• Taxonomy Table (Excerpt):

Model	HL-Errors ±SD	Exec-Err ±SD	Only Gear	Gear+Exec	Only Exec	Invalid Code
GPT-4.1	1.50 ± 1.39	0.78 ± 0.89	26.1%	32.2%	3.3%	2.2%
Grok-4	0.11 ± 0.16	0.02 ± 0.04	2.8%	1.7%	0.6%	2.8%

A representative failure example included a model omitting the crafting of "Reinforced Wood" for a "Dragon-Scale Shield," causing plan failure at the terminal crafting step (Anokhin et al., 18 Aug 2025).

7. Impact, Limitations, and Future Directions

HeroBench has established that standard chain-of-thought (CoT) prompting protocols are insufficient for solving extended, structured action sequences in open domains. RL-based “thinking” training improves planning pass@k, but this benefit materializes primarily on tasks with layered dependencies; multi-agent decomposer–critic frameworks can outperform single models at low complexity but underperform at scale. Proprietary reasoning-augmented models (GPT-5, Grok-4) sustain performance with minimal drop-off at high difficulty, indicating systematic advances in the field.

Proposed Extensions: Future iterations are planned to introduce multi-agent collaboration and competition, visual-language modalities (e.g., integration of grid map imagery), stochasticity (e.g., variable drop rates), branching recipes, and opportunities for emergent behaviors. Persistent challenges include scaling scenarios to open-ended quests, devising automated curricula for smooth difficulty transitions, and constructing reward structures that faithfully represent incremental learning and achievement.

HeroBench thus provides a methodologically transparent, flexible, and scalable framework for research into robust, long-horizon autonomous planning and structured reasoning using LLMs, with immediate relevance for both the evaluation and development of advanced generative agents (Anokhin et al., 18 Aug 2025).