MineExplorer Benchmark Overview

• MineExplorer is a benchmark for evaluating long-horizon open-world reasoning in Minecraft, focusing on composite task synthesis and milestone-based evaluation.
• It employs a structured multi-agent workflow with a ReAct protocol to filter and compose tasks from atomic actions, ensuring robust performance analysis.
• Empirical results show a significant drop in multi-hop task success, highlighting the challenges of transitioning from single-step to complex, long-horizon scenarios.

MineExplorer is a benchmark designed for evaluating the long-horizon open-world exploration capacities of multimodal LLM (MLLM) agents in the Minecraft Java Edition environment. Unlike previous embodied or game-based benchmarks, which typically emphasize short-horizon or game-mechanical tasks, MineExplorer centers on general open-world reasoning while minimizing reliance on Minecraft-specific priors. It accomplishes this by systematically filtering atomic tasks, leveraging a ReAct-based action/reasoning protocol, and introducing a rigorous multi-agent workflow for robust composite task synthesis and milestone-based evaluation (Ju et al., 29 May 2026).

1. Formal Structure and Environment Specification

The benchmark operates in the Minecraft Java Edition, exposing both a rich visual and structured symbolic state space:

• State Space ($\mathcal S$): Each agent step receives an RGB first-person view and a symbolic "info" dictionary, which includes:
  • Player position and orientation $\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$
  • Inventory (36 slots: $\{\mathrm{item\_type}, \mathrm{quantity}\}$)
  • Cumulative counters for core activities (craft, mine, kill, use, pickup)
  • Optional voxel and mob lists within user-defined bounding boxes
• Action Space ($\mathcal A$): Agents output at each step one discrete action from the standard Minecraft API, controlling locomotion, camera direction, item usage, inventory, and chat commands.
• ReAct-Style Protocol: At each time step $t$, the agent receives the full trajectory

$$\bigl\{(\mathrm{frame}_0, \mathrm{thought}_0, \mathrm{action}_0), \dots, (\mathrm{frame}_{t-1}, \mathrm{thought}_{t-1}, \mathrm{action}_{t-1})\bigr\}$$

and outputs a "thought" (natural-language reasoning) and an "action" (one-hot over $\mathcal A$). Task episodes terminate on milestone satisfaction or after 300 environment steps.

2. Task Taxonomy and Capability Decomposition

MineExplorer defines its task space through filtration and hierarchical capability annotation:

• Atomic Tasks: Derived from 3,382 Minecraft MCU single-step atomic tasks, with an LLM-based filter ($f_{\rm MC}(t)=0$) retaining only those solvable by general open-world or commonsense reasoning, resulting in a pool $\mathcal T^*$ of 1,497 atomic tasks.
• Capability Annotation: Each retained atomic task is mapped into a binary property vector $\varphi(t) \in \{0,1\}^{|\mathcal C|}$, decomposed into:
  • $\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$0: Perceptual demands (spatial, temporal, entity, state, inventory)
  • $\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$1: Reasoning types (common, causal, relational)
  • $\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$2: Action primitives (move, jump, collect, place, craft, attack)
• Composite Tasks (Multi-Hop): Multi-hop tasks $\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$3 are formed by composing instructions ($\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$4), initial state ($\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$5), a DAG over atomic tasks ($\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$6), and rule-based milestone checkers ($\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$7). The normalized difficulty of a composite task is defined as:

$\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$8

where $\{x, y, z, \mathrm{pitch}, \mathrm{yaw}\}$9 is the transitive closure of $\{\mathrm{item\_type}, \mathrm{quantity}\}$0, and $\{\mathrm{item\_type}, \mathrm{quantity}\}$1 the capability matrix over nodes, with $\{\mathrm{item\_type}, \mathrm{quantity}\}$2 increasing for longer chains and more diverse requirements.

3. Multi-Agent Synthesis Pipeline

Robust instance construction is achieved through a multi-agent design-and-validation loop:

• Agent Roles:

1. TaskSelectorAgent—assembles atomic task sets and implicit instruction $\{\mathrm{item\_type}, \mathrm{quantity}\}$3, outputs the candidate DAG. 2. SceneDesignerAgent—generates Minecraft commands to instantiate scenes, previews and refines layouts. 3. MilestoneAgent—assigns and tests rule-based milestones for each DAG node. 4. MinecraftExpertAgent—audits for dependence on Minecraft-specific conventions. 5. ValidatorAgent—checks for graph soundness, initial milestone falsity, and semantic match.

• Workflow Phases:
  • Initialization: Sequential handoff among the first three agents, with initial scene preview.
  • Debate/Renovation: Iterative critique and revision among all agents, coordinated by an orchestrating agent.
• Human Validation: Compared to a single-agent prompt baseline, the multi-agent workflow substantially increases validity and quality, as summarized below.

Pipeline	1-hop Valid	2-hop Valid	3-hop Valid	4-hop Valid	Overall Valid
Single-Agent	64.2%	56.3%	27.1%	27.0%	44.3%
Multi-Agent	89.0%	84.4%	68.7%	65.5%	78.3%

Scores reflect human screening post-synthesis. Composite task pool after screening: 813 instances.

4. Evaluation Metrics and Methodology

Assessment centers on milestone-tracked success and capability decoupling:

• Milestone Success Rate (MSR):

$\{\mathrm{item\_type}, \mathrm{quantity}\}$4

• Task Success Rate (TSR):

$\{\mathrm{item\_type}, \mathrm{quantity}\}$5

• Capability-Level Decomposition: Perception (P), reasoning (R), and action (A) performance are tracked by milestone satisfaction rates.
• Difficulty Binning: Instances are binned by normalized task difficulty $\{\mathrm{item\_type}, \mathrm{quantity}\}$6 (width 0.1). TSR drops sharply (≈50 percentage points) from easy ($\{\mathrm{item\_type}, \mathrm{quantity}\}$7) to hard ($\{\mathrm{item\_type}, \mathrm{quantity}\}$8) bins, establishing a strong coupling between graph-structured complexity and agent performance.

5. Empirical Results and Model Behavior

• Main Results: Sixteen SOTA MLLMs were evaluated, with leading model performance as follows:

Model	1-hop TSR	Multi-hop TSR	Overall TSR
Claude-Opus-4.6	77.7%	23.9%	41.1%
Gemini-3.1-Pro	74.2%	19.5%	37.0%

Sharp "multi-hop collapse": both models incur >50 percentage point TSR reduction from 1- to multi-hop.

• Perception, Action, Reasoning: Across all models, perception rates exceed action, with reasoning lowest—especially pronounced on multi-hop tasks.
• Scaling and Prompting: Larger model variants (e.g., Qwen, Gemini) exhibit no consistent advantage over smaller ones. Chain-of-thought-style (thinking-mode) prompting sometimes underperforms direct "Instruct" mode, indicating the limits of superficial reasoning augmentation for open-world tracking.
• Milestone Reliability and Human Alignment: Human ratings for Claude-Opus-4.6 runs align strongly (agreement ≈86.8%) with automated MSR-based passes; mean human rating ≈4.0/5 if all milestones pass, ≈2.5/5 if none do.
• Interaction Efficiency: Successful multi-hop completions rarely exceed 120 steps; failed episodes typically reach the 300-step cap.

6. Dataset Release and Artifact Structure

• Atomic Pool: 1,497 filtered, capability-annotated single-hop tasks.
• Composite Tasks: 813 human-validated multi-hop instances adjusted for validity and diversity, distributed among 1-hop (≈260), 2-hop (254), 3-hop (145), and 4-hop (154).
• Release Contents: Each composite instance is provided with:
  • JSON manifest $\{\mathrm{item\_type}, \mathrm{quantity}\}$9
  • Scene construction script
  • Rule-based milestone definitions
  • Fine-grained capability annotations
• Code and Dataset: Publicly available at https://github.com/Jometeorie/MineExplorer.

7. Implications, Limitations, and Future Directions

MineExplorer rigorously isolates and probes the fundamental challenges faced by MLLM agents in visually rich, dynamic 3D environments with hidden prerequisites and long-horizon dependencies. The benchmark demonstrates that, while state-of-the-art models succeed on many single-hop tasks, performance collapses as multi-step coordination and latent state-tracking become necessary.

Future research directions suggested include:

• Development of architectures with improved memory and state-tracking for 3D exploration;
• Intrinsic motivation or other specialized exploration heuristics to bootstrap discovery of latent task structure;
• Curricula to interpolate between short- and long-horizon tasks;
• Transferring these methods to real-world embodied agent domains.

MineExplorer thus constitutes a rigorous and extensible testbed for advancing generalizable open-world exploration in multimodal agent research (Ju et al., 29 May 2026).