---
title: 'MineNPC-Task: Benchmark for Minecraft LLM Agents'
url: https://www.emergentmind.com/topics/minenpc-task
type: topic
---

# MineNPC-Task: Benchmark for Minecraft LLM Agents

MineNPC-Task is a user-authored benchmark and evaluation harness for testing memory-aware, mixed-initiative LLM agents in open-world *Minecraft*. Rather than relying on synthetic prompts, it elicits tasks from formative and summative co-play with expert players, normalizes them into parametric templates with explicit preconditions and dependency structure, and pairs them with machine-checkable validators under a bounded-knowledge policy that forbids out-of-world shortcuts. In its initial released snapshot, the framework is instantiated with GPT-4o and evaluated on **216** subtasks across **8** experienced players, exposing recurring breakdown patterns in code execution, inventory and tool handling, referencing, and navigation [2601.05215].

## 1. Definition, research target, and design rationale

MineNPC-Task targets a specific evaluation problem: how to assess LLM agents that must **plan, clarify, act, and remember** while operating inside an open-world *Minecraft* environment. Its central claim is not that open-world execution alone is sufficient, but that realistic evaluation requires a coupled benchmark-and-harness design in which the task specification, action interface, memory interface, and success criteria are all externally inspectable. In this sense, MineNPC-Task is simultaneously a task suite and a model-agnostic runtime framework [2601.05215].

The benchmark is explicitly aimed at **memory-aware** and **mixed-initiative** agents. “Memory-aware” denotes the use of a simple typed memory store containing landmarks, artifacts or resources, user preferences, commitments, and breakdowns, with provenance labels `seen`, `told`, and `inferred`. “Mixed-initiative” denotes a collaborative interaction pattern in which the agent proposes a short plan, asks targeted clarifying questions when required parameters cannot be safely bound, and participates in bounded repair after failures rather than behaving as a passive instruction follower. This design shifts evaluation away from single-shot command execution toward longitudinal coordination over partially specified tasks [2601.05215].

A further design constraint is **bounded knowledge**. The harness is built so that both action and judgment must remain inside the public in-game interface. The agent cannot use admin commands, global map or seed introspection, or scans beyond loaded chunks; judging is likewise restricted to in-world evidence such as inventory, equipment, position, nearby blocks and entities, and recent chat. A plausible implication is that MineNPC-Task treats realism of access as part of the benchmark definition rather than as an implementation detail [2601.05215].

## 2. Task suite, parametric templates, and formal subtask structure

The released suite comprises **44** user-authored *Minecraft* tasks decomposed into **216** subtasks. Tasks were collected from expert players during co-play and then normalized into compact templates with explicit preconditions, minimal dependency structure, and machine-checkable success criteria. The examples given in the release span requests such as “collect 20 oak logs,” “remember this as weapon_storage,” “build a pyramid from sand (5×5 base),” and “harvest wheat and craft 3 pieces of bread” [2601.05215].

At the subtask level, MineNPC-Task uses a structured `TaskModel` with fields for `index`, `subtask`, `dependencies`, `parameters`, `status`, `clarifying_questions`, and `success_criteria`. A plan is represented as
$$
\text{Plan} = \{ T_1, T_2, \dots, T_n \},
$$
with each subtask formalized as
$$
T_i = (i, \text{subtask}_i, D_i, P_i, s_i, Q_i, C_i),
$$
where \(D_i\) is the dependency set, \(P_i\) the parameter bundle, \(s_i \in \{\text{pending}, \text{in-progress}, \text{completed}\}\), \(Q_i\) the outstanding clarifying questions, and \(C_i\) the success criterion. Dependencies therefore define a small DAG over subtasks, and a subtask can execute only after its prerequisites complete successfully [2601.05215].

The suite’s complexity analysis reports an average of \(M = 4.9\) subtasks per task, with range 1–12. Subtask domains are distributed as follows.

| Domain | Share |
|---|---:|
| Resource collection | 32% |
| Navigation & movement | 28% |
| Item management | 18% |
| Construction | 12% |
| Tool management | 6% |
| Interaction | 4% |

The task families listed in the appendix include resource collection and mining, tool and equipment management, agriculture and food, construction and building, crafting and processing, and storage and inventory management. This organization makes MineNPC-Task neither a pure dialogue dataset nor a pure control suite: it is a benchmark for embodied task execution under explicit dependency structure [2601.05215].

## 3. Evaluation harness, execution loop, and bounded-knowledge policy

The harness integrates *Minecraft* through **Mineflayer**, exposing perception channels such as `recent_chat`, `position`, `movement`, `inventory`, `equipment`, `nearby_blocks`, `nearby_entities`, and environmental context including biome, weather, and time. It exposes action capabilities through high-level skills layered over Mineflayer, including `navigate`, `mine`, `craft`, `place`, `interact`, and item transfer and drop. At each tick, the current world and agent snapshot is represented as `BotState`, which becomes the substrate both for execution and for post hoc validation [2601.05215].

Execution follows a **Plan–Clarify–Act–Judge** pipeline. Incoming chat is first routed as one of `task(request)`, `memory_update`, `conversation`, or `control`. For `task(request)`, the planner instantiates a task template into a short plan of 3–5 subtasks and emits a one-line plan preview. If a required slot is missing, the system asks exactly one targeted question drawn from `clarifying_questions`; the answer fills the corresponding parameter and is logged as a plan delta such as `search_radius: 100 → 10`. The framework then generates JavaScript code targeting Mineflayer APIs, passes it through a lightweight reviewer, and caps retries at \(K \le 3\) to avoid runaway loops [2601.05215].

Post-execution judgment is validator-backed. Validators consume pre/post `BotState`, recent chat, and optional template context, and return a `ValidationOutput` containing `success` or `failure`, `state_changes`, `chat_insights`, optional `suggestions` and `question_for_player`, `is_task_completed`, `final_result`, and `confidence_score`. Because validation is tied to observable state deltas rather than hidden simulator access, the harness makes success auditing external and reproducible. Formally, the bounded-knowledge contract is expressed as
$$
\forall a_t \in \text{actions},\quad a_t \in \mathcal{A}_{\text{public}} \quad \land \quad a_t \notin \mathcal{A}_{\text{priv}},
$$
and any detected call from \(\mathcal{A}_{\text{priv}}\) invalidates the run [2601.05215].

## 4. Memory, clarification, and mixed-initiative interaction

The memory subsystem is intentionally lightweight. It stores landmarks with coordinates and radius, preferences such as tool choices or search radius, and task-relevant world knowledge such as `chest_locations`, `safe_zones`, and `resource_hotspots`. Entries carry provenance in \(\{\text{seen}, \text{told}, \text{inferred}\}\), can be retrieved with nearest-\(k\) queries scoped to the current task, and may be marked `stale` if the underlying world state changes. The design does not attempt complex long-term consolidation or reflective summarization; it is a structured memory interface meant to support slot binding, landmark recall, and limited reuse across subtasks [2601.05215].

Mixed-initiative behavior is implemented at plan time and at repair time. At plan time, the agent emits a plan preview and asks one contextual question when a necessary parameter is missing and cannot be safely inferred from state or memory. The canonical examples are illustrative: for “collect 20 oak logs,” the planner can ask whether known oak trees are nearby and, after the answer “within 10 blocks,” update `search_radius: 100 → 10` and persist that preference; for “get me a pick_axe,” after a prior memory update “remember this as weapon_storage,” the system can retrieve the landmark, propose “go to weapon_storage → select a pickaxe → return,” and ask “Which pickaxe, iron or diamond?” before execution [2601.05215].

At repair time, failed subtasks trigger bounded alternatives rather than unconstrained replanning. The validator may suggest retrying with modified parameters, backtracking, or re-specifying the task; the player selects among these options, and the system may partially replan from the failing step but never rewrites completed steps. This suggests a benchmark emphasis on recoverability and coordination cost, not only on raw completion. The framework’s event logs therefore include plan previews, plan deltas, clarifications, memory reads and writes, reviewer iterations, validator evidence, and repair choices, making mixed-initiative interaction a first-class evaluation object rather than an incidental UX layer [2601.05215].

## 5. Empirical snapshot, failure taxonomy, and user study

The initial release evaluates **GPT-4o** with **8** experienced *Minecraft* players over **44** high-level tasks and **216** subtasks. Outcome is measured at the subtask level. If \(N_{\text{attempted}}\) is the number of subtasks that reached execution and \(N_{\text{failed}}\) is the number with `is_task_completed = False`, then
$$
\text{FailureRate} = \frac{N_{\text{failed}}}{N_{\text{attempted}}}.
$$
For the released GPT-4o snapshot,
$$
N_{\text{attempted}} = 216,\quad N_{\text{failed}} = 71,\quad \text{FailureRate} \approx \frac{71}{216} \approx 0.33.
$$
The benchmark release characterizes this as an initial snapshot rather than a final leaderboard number [2601.05215].

Failure analysis identifies several recurring breakdown types. Figures in the release report **code/execution failures** \((n=24)\), **inventory/tool issues** \((n=10)\), **context misunderstandings** \((n=7)\), **referencing failures** \((n=6)\), and **navigation failures** \((n=5)\). Mining and construction tasks show more code and inventory problems, while retrieval and navigation tasks show more referencing and context issues. The qualitative examples are consistent with these patterns: invalid Vec3 parameters and asynchronous-loop bugs appear in generated code; tool handling fails around mining and building; deictic references such as “this block” and “the block I’m looking at” remain difficult under the bounded-knowledge Mineflayer interface [2601.05215].

The accompanying user study reports positive but qualified reception. **7/8** users agreed that interaction was effective; usability and overall experience both had **>75% agreement**; and **6/8** participants found memory-based recall helpful. At the same time, users emphasized the need for stronger persistence across tasks, more visible memory state, and better handling of shared perception assumptions. A plausible implication is that MineNPC-Task exposes an asymmetry common in embodied LLM systems: interaction quality can be rated positively even when subtask failure remains substantial, provided explanations, plan previews, and repair mechanisms remain legible [2601.05215].

## 6. Position within Minecraft-agent research

MineNPC-Task occupies a specific niche within *Minecraft* agent research. Earlier systems such as **CraftAssist** focus on “agents that can complete tasks specified by dialogue, and eventually, to learn from dialogue interactions,” using symbolic memory, task stacks, and dialogue objects inside *Minecraft* [1907.08584]. **MCPDial** contributes a persona-driven dialogue dataset with canonical function calls such as `Call locate a resource on diamonds` and `Call craft an item on <item>`, emphasizing long, task-oriented player–NPC conversations with explicit tool-use annotations [2410.21627]. **VillagerAgent** addresses multi-agent dependency management through a DAG task decomposer, agent controller, and state manager on VillagerBench, targeting spatial, causal, and temporal coordination in *Minecraft* [2406.05720]. By contrast, MineNPC-Task is centered on **single-agent, memory-aware, mixed-initiative evaluation with validator-backed subtasks and bounded public-API access** [2601.05215].

It also differs from synthetic or modular sub-benchmarks that isolate narrower competencies. “A LLM Benchmark based on the Minecraft Builder Dialog Agent Task” reduces the builder setting to synthetic tasks in **Absolute Addressing**, **Relative Addressing**, and **Primitive Shapes**, emphasizing spatial reasoning and vector-based math rather than open-world execution [2407.12734]. **BAP v2** formalizes the Builder Action Prediction problem in the Minecraft Collaborative Building Task and contributes cleaned evaluation and synthetic training data for multimodal dialogue-to-action prediction [2501.10836]. MineNPC-Task instead evaluates full plan–clarify–act–judge loops in an open-world setting with real player-authored requests and machine-checkable validators [2601.05215].

Finally, MineNPC-Task is adjacent to broader open-world planning and control work but not reducible to it. **MP5** studies process-dependent and context-dependent tasks with goal-conditioned active perception and reports a **22%** success rate on difficult process-dependent tasks and **91%** on tasks that heavily depend on context [2312.07472]. **BAR** studies backward reasoning for long-horizon *Minecraft* tasks with recursive goal decomposition, state consistency maintenance, and stage memory [2505.14079]. **Task Scoping** studies task-specific abstraction in open-scope planning and reports a **75x reduction in search time** in a *Minecraft* domain [2010.08869]. MineNPC-Task does not propose a new control policy of that kind; rather, it provides a controlled harness, a user-authored task suite, and dense event logging that can be used to compare such agents under a shared bounded-knowledge regime [2601.05215].

Source: https://www.emergentmind.com/topics/minenpc-task