MineNPC-Task: Benchmark for Minecraft LLM Agents

• MineNPC-Task is a benchmark that evaluates memory-aware and mixed-initiative LLM agents operating within an open-world Minecraft environment.
• It uses user-authored task templates with explicit preconditions, structured subtasks, and machine-checkable validators to ensure realistic execution.
• Initial evaluations with GPT-4o on 216 subtasks reveal recurring issues in planning, navigation, and resource management under bounded-knowledge constraints.

MineNPC-Task is a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world Minecraft. Rather than relying on synthetic prompts, it elicits tasks from formative and summative co-play with expert players, normalizes them into parametric templates with explicit preconditions and dependency structure, and pairs them with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. In its initial released snapshot, the framework is instantiated with GPT-4o and evaluated on 216 subtasks across 8 experienced players, exposing recurring breakdown patterns in code execution, inventory and tool handling, referencing, and navigation (Doss et al., 8 Jan 2026).

1. Definition, research target, and design rationale

MineNPC-Task targets a specific evaluation problem: how to assess LLM agents that must plan, clarify, act, and remember while operating inside an open-world Minecraft environment. Its central claim is not that open-world execution alone is sufficient, but that realistic evaluation requires a coupled benchmark-and-harness design in which the task specification, action interface, memory interface, and success criteria are all externally inspectable. In this sense, MineNPC-Task is simultaneously a task suite and a model-agnostic runtime framework (Doss et al., 8 Jan 2026).

The benchmark is explicitly aimed at memory-aware and mixed-initiative agents. “Memory-aware” denotes the use of a simple typed memory store containing landmarks, artifacts or resources, user preferences, commitments, and breakdowns, with provenance labels seen, told, and inferred. “Mixed-initiative” denotes a collaborative interaction pattern in which the agent proposes a short plan, asks targeted clarifying questions when required parameters cannot be safely bound, and participates in bounded repair after failures rather than behaving as a passive instruction follower. This design shifts evaluation away from single-shot command execution toward longitudinal coordination over partially specified tasks (Doss et al., 8 Jan 2026).

A further design constraint is bounded knowledge. The harness is built so that both action and judgment must remain inside the public in-game interface. The agent cannot use admin commands, global map or seed introspection, or scans beyond loaded chunks; judging is likewise restricted to in-world evidence such as inventory, equipment, position, nearby blocks and entities, and recent chat. A plausible implication is that MineNPC-Task treats realism of access as part of the benchmark definition rather than as an implementation detail (Doss et al., 8 Jan 2026).

2. Task suite, parametric templates, and formal subtask structure

The released suite comprises 44 user-authored Minecraft tasks decomposed into 216 subtasks. Tasks were collected from expert players during co-play and then normalized into compact templates with explicit preconditions, minimal dependency structure, and machine-checkable success criteria. The examples given in the release span requests such as “collect 20 oak logs,” “remember this as weapon_storage,” “build a pyramid from sand (5×5 base),” and “harvest wheat and craft 3 pieces of bread” (Doss et al., 8 Jan 2026).

At the subtask level, MineNPC-Task uses a structured TaskModel with fields for index, subtask, dependencies, parameters, status, clarifying_questions, and success_criteria. A plan is represented as

$\text{Plan} = \{ T_1, T_2, \dots, T_n \},$

with each subtask formalized as

$$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$

where $D_i$ is the dependency set, $P_i$ the parameter bundle, $$s_i \in \{\text{pending}, \text{in-progress}, \text{completed}\}$$, $Q_i$ the outstanding clarifying questions, and $C_i$ the success criterion. Dependencies therefore define a small DAG over subtasks, and a subtask can execute only after its prerequisites complete successfully (Doss et al., 8 Jan 2026).

The suite’s complexity analysis reports an average of $M = 4.9$ subtasks per task, with range 1–12. Subtask domains are distributed as follows.

Domain	Share
Resource collection	32%
Navigation & movement	28%
Item management	18%
Construction	12%
Tool management	6%
Interaction	4%

The task families listed in the appendix include resource collection and mining, tool and equipment management, agriculture and food, construction and building, crafting and processing, and storage and inventory management. This organization makes MineNPC-Task neither a pure dialogue dataset nor a pure control suite: it is a benchmark for embodied task execution under explicit dependency structure (Doss et al., 8 Jan 2026).

3. Evaluation harness, execution loop, and bounded-knowledge policy

The harness integrates Minecraft through Mineflayer, exposing perception channels such as recent_chat, position, movement, inventory, equipment, nearby_blocks, nearby_entities, and environmental context including biome, weather, and time. It exposes action capabilities through high-level skills layered over Mineflayer, including navigate, mine, craft, place, interact, and item transfer and drop. At each tick, the current world and agent snapshot is represented as BotState, which becomes the substrate both for execution and for post hoc validation (Doss et al., 8 Jan 2026).

Execution follows a Plan–Clarify–Act–Judge pipeline. Incoming chat is first routed as one of task(request), memory_update, conversation, or control. For task(request), the planner instantiates a task template into a short plan of 3–5 subtasks and emits a one-line plan preview. If a required slot is missing, the system asks exactly one targeted question drawn from clarifying_questions; the answer fills the corresponding parameter and is logged as a plan delta such as search_radius: 100 → 10. The framework then generates JavaScript code targeting Mineflayer APIs, passes it through a lightweight reviewer, and caps retries at $K \le 3$ to avoid runaway loops (Doss et al., 8 Jan 2026).

Post-execution judgment is validator-backed. Validators consume pre/post BotState, recent chat, and optional template context, and return a ValidationOutput containing success or failure, state_changes, chat_insights, optional suggestions and question_for_player, is_task_completed, final_result, and confidence_score. Because validation is tied to observable state deltas rather than hidden simulator access, the harness makes success auditing external and reproducible. Formally, the bounded-knowledge contract is expressed as

$$\forall a_t \in \text{actions},\quad a_t \in \mathcal{A}_{\text{public}} \quad \land \quad a_t \notin \mathcal{A}_{\text{priv}},$$

and any detected call from $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$0 invalidates the run (Doss et al., 8 Jan 2026).

4. Memory, clarification, and mixed-initiative interaction

The memory subsystem is intentionally lightweight. It stores landmarks with coordinates and radius, preferences such as tool choices or search radius, and task-relevant world knowledge such as chest_locations, safe_zones, and resource_hotspots. Entries carry provenance in $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$1, can be retrieved with nearest-$$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$2 queries scoped to the current task, and may be marked stale if the underlying world state changes. The design does not attempt complex long-term consolidation or reflective summarization; it is a structured memory interface meant to support slot binding, landmark recall, and limited reuse across subtasks (Doss et al., 8 Jan 2026).

Mixed-initiative behavior is implemented at plan time and at repair time. At plan time, the agent emits a plan preview and asks one contextual question when a necessary parameter is missing and cannot be safely inferred from state or memory. The canonical examples are illustrative: for “collect 20 oak logs,” the planner can ask whether known oak trees are nearby and, after the answer “within 10 blocks,” update search_radius: 100 → 10 and persist that preference; for “get me a pick_axe,” after a prior memory update “remember this as weapon_storage,” the system can retrieve the landmark, propose “go to weapon_storage → select a pickaxe → return,” and ask “Which pickaxe, iron or diamond?” before execution (Doss et al., 8 Jan 2026).

At repair time, failed subtasks trigger bounded alternatives rather than unconstrained replanning. The validator may suggest retrying with modified parameters, backtracking, or re-specifying the task; the player selects among these options, and the system may partially replan from the failing step but never rewrites completed steps. This suggests a benchmark emphasis on recoverability and coordination cost, not only on raw completion. The framework’s event logs therefore include plan previews, plan deltas, clarifications, memory reads and writes, reviewer iterations, validator evidence, and repair choices, making mixed-initiative interaction a first-class evaluation object rather than an incidental UX layer (Doss et al., 8 Jan 2026).

5. Empirical snapshot, failure taxonomy, and user study

The initial release evaluates GPT-4o with 8 experienced Minecraft players over 44 high-level tasks and 216 subtasks. Outcome is measured at the subtask level. If $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$3 is the number of subtasks that reached execution and $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$4 is the number with is_task_completed = False, then

$$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$5

For the released GPT-4o snapshot,

$$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$6

The benchmark release characterizes this as an initial snapshot rather than a final leaderboard number (Doss et al., 8 Jan 2026).

Failure analysis identifies several recurring breakdown types. Figures in the release report code/execution failures $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$7, inventory/tool issues $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$8, context misunderstandings $$T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),$$9, referencing failures $D_i$0, and navigation failures $D_i$1. Mining and construction tasks show more code and inventory problems, while retrieval and navigation tasks show more referencing and context issues. The qualitative examples are consistent with these patterns: invalid Vec3 parameters and asynchronous-loop bugs appear in generated code; tool handling fails around mining and building; deictic references such as “this block” and “the block I’m looking at” remain difficult under the bounded-knowledge Mineflayer interface (Doss et al., 8 Jan 2026).

The accompanying user study reports positive but qualified reception. 7/8 users agreed that interaction was effective; usability and overall experience both had >75% agreement; and 6/8 participants found memory-based recall helpful. At the same time, users emphasized the need for stronger persistence across tasks, more visible memory state, and better handling of shared perception assumptions. A plausible implication is that MineNPC-Task exposes an asymmetry common in embodied LLM systems: interaction quality can be rated positively even when subtask failure remains substantial, provided explanations, plan previews, and repair mechanisms remain legible (Doss et al., 8 Jan 2026).

6. Position within Minecraft-agent research

MineNPC-Task occupies a specific niche within Minecraft agent research. Earlier systems such as CraftAssist focus on “agents that can complete tasks specified by dialogue, and eventually, to learn from dialogue interactions,” using symbolic memory, task stacks, and dialogue objects inside Minecraft (Gray et al., 2019). MCPDial contributes a persona-driven dialogue dataset with canonical function calls such as Call locate a resource on diamonds and Call craft an item on <item>, emphasizing long, task-oriented player–NPC conversations with explicit tool-use annotations (Alavi et al., 2024). VillagerAgent addresses multi-agent dependency management through a DAG task decomposer, agent controller, and state manager on VillagerBench, targeting spatial, causal, and temporal coordination in Minecraft (Dong et al., 2024). By contrast, MineNPC-Task is centered on single-agent, memory-aware, mixed-initiative evaluation with validator-backed subtasks and bounded public-API access (Doss et al., 8 Jan 2026).

It also differs from synthetic or modular sub-benchmarks that isolate narrower competencies. “A LLM Benchmark based on the Minecraft Builder Dialog Agent Task” reduces the builder setting to synthetic tasks in Absolute Addressing, Relative Addressing, and Primitive Shapes, emphasizing spatial reasoning and vector-based math rather than open-world execution (Madge et al., 2024). BAP v2 formalizes the Builder Action Prediction problem in the Minecraft Collaborative Building Task and contributes cleaned evaluation and synthetic training data for multimodal dialogue-to-action prediction (Jayannavar et al., 18 Jan 2025). MineNPC-Task instead evaluates full plan–clarify–act–judge loops in an open-world setting with real player-authored requests and machine-checkable validators (Doss et al., 8 Jan 2026).

Finally, MineNPC-Task is adjacent to broader open-world planning and control work but not reducible to it. MP5 studies process-dependent and context-dependent tasks with goal-conditioned active perception and reports a 22% success rate on difficult process-dependent tasks and 91% on tasks that heavily depend on context (Qin et al., 2023). BAR studies backward reasoning for long-horizon Minecraft tasks with recursive goal decomposition, state consistency maintenance, and stage memory (Du et al., 20 May 2025). Task Scoping studies task-specific abstraction in open-scope planning and reports a 75x reduction in search time in a Minecraft domain (Fishman et al., 2020). MineNPC-Task does not propose a new control policy of that kind; rather, it provides a controlled harness, a user-authored task suite, and dense event logging that can be used to compare such agents under a shared bounded-knowledge regime (Doss et al., 8 Jan 2026).