MineNPC-Task: Task Suite for Memory-Aware Minecraft Agents

Abstract: We present \textsc{MineNPC-Task}, a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world \emph{Minecraft}. Rather than relying on synthetic prompts, tasks are elicited from formative and summative co-play with expert players, normalized into parametric templates with explicit preconditions and dependency structure, and paired with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. The harness captures plan/act/memory events-including plan previews, targeted clarifications, memory reads and writes, precondition checks, and repair attempts and reports outcomes relative to the total number of attempted subtasks, derived from in-world evidence. As an initial snapshot, we instantiate the framework with GPT-4o and evaluate \textbf{216} subtasks across \textbf{8} experienced players. We observe recurring breakdown patterns in code execution, inventory/tool handling, referencing, and navigation, alongside recoveries supported by mixed-initiative clarifications and lightweight memory. Participants rated interaction quality and interface usability positively, while highlighting the need for stronger memory persistence across tasks. We release the complete task suite, validators, logs, and harness to support transparent, reproducible evaluation of future memory-aware embodied agents.

• The paper establishes a novel benchmark that evaluates memory-aware, mixed-initiative Minecraft agents by strictly using public Mineflayer APIs.
• It employs user-authored tasks with structured templates and explicit clarifications to assess planning, execution, and repair strategies in dynamic gameplay.
• Empirical results with GPT-4o and expert players show a roughly 33% subtask failure rate, underscoring challenges in memory retention and context recovery.

MineNPC-Task: A Benchmark for Memory-Aware, Mixed-Initiative Agents in Minecraft

Motivation and Benchmark Design

MineNPC-Task addresses a key gap in embodied LLM evaluation by introducing a reproducible, user-authored benchmark focused on memory-aware, mixed-initiative agents in Minecraft. Unlike prior synthetic or privilege-enhanced benchmarks, MineNPC-Task grounds agent assessment in live co-play with expert players, producing a task suite based on authentic player behavior. All agent perception and action are bounded strictly to public Mineflayer APIs, disallowing privileged operations such as /give or out-of-world map introspection. Task templates feature explicit preconditions, slot-parameterization, and normalized plan structures, and each attempt is evaluated with machine-checkable validators using strictly in-world evidence for robust, fair comparison.

A core design principle is mixed-initiative interaction: the agent is required to preview a plan, issue targeted one-shot clarifications for underspecified slots, and rely on persistent but lightweight memory (landmarks, user preferences, breakdowns) with explicit provenance. The evaluation harness is designed to be model-agnostic, enabling fair measurement of downstream memory systems, planning, clarification ability, and repair/recovery strategies.

Architecture and Interaction Loop

The framework is encapsulated in a model-agnostic Mineflayer envelope. Upon receiving a user request through chat or control events, the router classifies intent (task, memory update, control), then passes task requests to the planner. The planner compiles a compact, dependency-aware plan of 3–5 subtasks, each tied to parameters, dependencies, a clarifying question (when necessary), and success criteria.

Figure 2: Player interface for memory-grounded agent interaction, displaying primary in-game view and a chat side-rail for plan previews, clarifications, validator outputs, and progress notifications.

The loop proceeds as follows:

• Clarification: If a slot parameter is missing, a single contextual clarification is issued (e.g., “Which pickaxe, iron or diamond?”).
• Code Generation and Execution: Approved JavaScript is generated and reviewed for API conformance and safety (with capped retries). Code is then executed to perform the subtask.
• Validation and Repair: Completion is determined solely from in-game state deltas and local chat evidence. On failure, a minimal repair suggestion or replan is invoked, and all major decisions and plan deltas are logged for transparency and auditability.

Memory entries (e.g., named player landmarks or contextual preferences) are stored with scope and provenance, and can be queried for slot defaulting in future interactions. The system logs timing, clarifications, plan structure, retries, validator rationale, memory events, and token usage at all stages.

Task Suite and Planning Surface

The MineNPC-Task suite comprises 44 expert-derived high-level tasks with 216 subtasks representing the major functional dimensions of Minecraft gameplay: mining, building, agriculture, construction, crafting, inventory/tool management, and navigation. Each user-authored task is normalized into a template specifying preconditions and dependencies, and subtasks are designed for explicit—and reproducible—completion checking.

Figure 4: Planning surfaces for “harvest oak logs,” illustrating conversion of a naturalistic chat request into a structured, legible plan and parameterized subtask with defaults and clarifications.

The framework emphasizes that plan preview, clarification, and bounded repair are intrinsic to robust co-play. Memory assists not just with slot-filling but also in surfacing breakdowns and supporting recoverability when the environment changes or ambiguity arises.

Empirical Evaluation: GPT-4o with Expert Players

The initial evaluation of MineNPC-Task uses GPT-4o as the agent model, co-playing with 8 Minecraft experts. Across 44 tasks (216 subtasks), the system logs all planning, recall, clarification, execution, and repair events.

Performance and Failure Analysis

The benchmark reports 71 subtask failures out of 216 (∼33%). Failure types are rigorously catalogued:

• Code/execution (24/71): Invalid parameters, API misfires, numerical errors.
• Inventory/tools (10/71): Missing, improper, or context-inappropriate tools/items.
• Referencing (6/71): Deictic/egocentric failures—ambiguity in what objects or locations are referenced.
• Context misunderstanding (7/71): Misparsed intent or world state.
• Navigation (5/71): Poor spatial targeting or drift from intended waypoints.

  Figure 6: Cross-tabulation of failure categories by Minecraft task type—code and inventory failures cluster in mining/construction, while referencing/context breakdowns dominate navigation and retrieval.

Notably, failures are often recoverable. In a majority of cases, participants completed goals after one or more attempts by refining commands, specifying slots, or collaborating through the repair prompt.

User Experience and Design Implications

User surveys indicate strong positive impressions of interaction and usability. Memory-based recall was considered helpful, though several participants highlighted the need for deeper persistence and visibility into the agent's memory contents to mitigate misunderstandings. Transparent, contextual feedback—such as error explanations and validator rationales—promoted perseverance after failures.

Players expected the agent to seamlessly ground references (e.g., "what I'm looking at") to shared spatial context, revealing opportunities for integrating egocentric representation and situational awareness. Design findings suggest that social presence (through unobtrusive camera/NPC cues and responsive agent feedback) enhances perceived companionship.

Implications and Future Directions

MineNPC-Task establishes a methodological baseline for evaluating mixed-initiative LLM agents in embodied, memory-intensive environments. By grounding all evaluation in in-world state and constraining agent action to public APIs, the benchmark supports reproducible, transparent measurement free from privileged model shortcuts or artificial advantage.

The strong assertion—supported by empirical results—is that general-purpose LLMs (e.g., GPT-4o) under this rigorous evaluation regime still exhibit substantial brittleness in plan execution, slot clarification, and memory retention, with ∼1 in 3 atomic actions failing in expert-authored tasks. This challenges narratives based solely on superficially high-level task completion statistics commonly reported in prior Minecraft benchmarks that permit privileged access or synthetic prompts.

Practically, the public release of MineNPC-Task—including templates, validators, and agent logs—enables controlled multi-model evaluation, opens the path for leaderboard-based progress, and provides a foundation for research on robust, generalizable memory and interaction strategies. The framework's modularity facilitates extensions to more complex dependencies, richer memory, and alternative planning and communication protocols.

Future research will benefit from:

• Multi-agent and multi-model comparison under identical knowledge and perception constraints.
• Expanded, diverse user-authored task pools with more complex dependencies.
• Fine-grained partial credit scoring (cost, efficiency, number of clarifications, repair rate).
• UI and agent improvements for memory transparency, egocentric reference resolution, and user correction generalization.

Conclusion

MineNPC-Task (2601.05215) represents a substantive advance in the reproducible evaluation of memory-aware, mixed-initiative agents in open-world settings. It demonstrates that, when privileged shortcuts are disallowed, mixed-initiative and transparent reasoning mechanisms become mandatory for robust success—even for frontier LLM agents. The release of this benchmark, along with detailed breakdown analysis and instrumentation, lays critical groundwork for principled progress in embodied LLM research, and provides a robust baseline for future systems aspiring to human-compatible co-play and long-horizon task completion.