VistaWise: Minecraft Agent Framework

• VistaWise is a Minecraft agent framework that combines a finetuned YOLOv10-L detector with a cross-modal knowledge graph to enhance visual and symbolic reasoning.
• It dramatically reduces domain-specific training from millions of samples to a few hundred, achieving a 33% diamond success rate compared to heavier baselines.
• The system’s modular design leverages a retrieval-based pooling strategy and desktop-level skill library to execute precise, real-world Minecraft actions via mouse-and-keyboard inputs.

VistaWise is a Minecraft agent framework for embodied decision-making in a virtual open-world environment. It is designed to achieve strong performance on long-horizon tasks while sharply reducing domain-specific training cost by combining a finetuned object detection model for visual analysis, a cross-modal knowledge graph that unifies Minecraft domain knowledge with current visual state, a retrieval-based pooling strategy for task-relevant graph extraction, and a desktop-level skill library that directly operates the vanilla Minecraft client through mouse-and-keyboard inputs (Fu et al., 26 Aug 2025).

1. Motivation and problem setting

VistaWise addresses a central limitation of LLM-based agents in open-world environments: LLMs are effective at symbolic reasoning and compositional instruction following, but their performance is hindered by the absence of domain-specific knowledge in Minecraft. Competent play requires explicit knowledge of crafting prerequisites, mining constraints, and tool-resource dependencies such as logs $\rightarrow$ planks $\rightarrow$ sticks $\rightarrow$ wooden pickaxe $\rightarrow$ cobblestone $\rightarrow$ stone pickaxe $\rightarrow$ iron ore $\rightarrow$ iron ingot $\rightarrow$ iron pickaxe $\rightarrow$ diamond. Without that structure, agents hallucinate recipes, mis-handle prerequisites, or choose infeasible actions (Fu et al., 26 Aug 2025).

The framework is also a response to two practical constraints in prior systems. First, many earlier Minecraft agents rely on environment APIs such as MineFlayer, which expose privileged textual world state and high-level actions; this simplifies control but limits autonomy and weakens transfer to environments that do not provide such APIs. Second, large-scale finetuning approaches are expensive in both data and compute. VistaWise explicitly asks whether comparable or better performance can be obtained by finetuning only a small visual component and injecting domain knowledge through structured retrieval rather than through full-model domain adaptation (Fu et al., 26 Aug 2025).

The cost argument is central. VistaWise reduces the requirement for domain-specific training data from millions of samples to a few hundred by finetuning only a YOLOv10-L detector on 471 frames, while leaving the LLM unfine-tuned. The comparative scale reported for representative baselines is shown below.

Method	Data scale / VRAM	Diamond success
STEVE-1	160M frames; 192G	0%
GROOT	1.6B frames; 384G	0%
OmniJARVIS	990M tokens; 640G	8%
VPT	140B frames; 23,040G	15%
ROCKET-1	1.6B frames; *	25%
VistaWise	471 frames; 24G	33%

This comparison suggests that VistaWise is not merely a lower-cost approximation of large-scale policy training. A plausible implication is that explicit domain structure, when coupled to direct low-level control, can substitute for a substantial fraction of brute-force domain-specific pretraining in open-world tasks (Fu et al., 26 Aug 2025).

2. System architecture and control formulation

VistaWise centers the LLM as the action policy and places three graph processes, a desktop interaction layer, and a bounded history mechanism around it. The architecture comprises text-modal knowledge graph construction, cross-modal knowledge graph construction, retrieval-based graph pooling, a desktop-level skill library, and a memory stack (Fu et al., 26 Aug 2025).

The policy is formalized as

$$\mathcal{A}(t)=\pi_\theta\big(\mathcal{I}(s), \mathcal{G'}(s,t), \mathcal{H}(t), \mathcal{L}\big),$$

where $\rightarrow$0 is the synthetic prompt for task $\rightarrow$1, $\rightarrow$2 is the pooled cross-modal knowledge graph at time $\rightarrow$3, $\rightarrow$4 is the recalled decision history, and $\rightarrow$5 is the skill library. The environment then evolves according to

$\rightarrow$6

This decomposition is significant because it separates three roles that are often entangled in prior embodied LLM systems. Visual grounding is handled by a dedicated detector rather than by the LLM’s own multimodal perception. Domain knowledge is externalized into a graph rather than stored in finetuned parameters. Action execution is mediated by a desktop-level function library rather than by a simulator API. The result is a policy loop in which the LLM reasons over symbolic structure, dynamic visual attributes, and recent decisions, then outputs parameterized low-level skills (Fu et al., 26 Aug 2025).

VistaWise uses GPT-4o as the policy model in the reported system. No finetuning is performed on the LLM itself. This design is explicitly intended to preserve general reasoning ability while supplying Minecraft-specific competence through retrieval and structured action interfaces rather than through large-scale supervised gameplay corpora (Fu et al., 26 Aug 2025).

3. Visual analysis and observable state

VistaWise defines the observable state at timestep $\rightarrow$7 as

$\rightarrow$8

where $\rightarrow$9 is the environment view and $\rightarrow$0 is the inventory view. The system partitions visual entities into environmental entities

$\rightarrow$1

and conditional entities

$\rightarrow$2

Environmental entities are world objects such as logs, trunks, ores, lava, and water. Conditional entities are inventory icons such as log_icon, plank_icon, stick_icon, or iron_pickaxe_icon, which encode the agent’s current crafting and mining conditions (Fu et al., 26 Aug 2025).

Visual grounding is provided by a dedicated YOLOv10-L object detector. The model is finetuned on 471 frames containing 3,304 annotated instances spanning 23 Minecraft visual entity types, using a 9:1 train/test split, 150 epochs, batch size 16, and input image size 768 on an NVIDIA L4 GPU with 24G VRAM. This is the only finetuned component in the framework (Fu et al., 26 Aug 2025).

For each environmental entity $\rightarrow$3, the detector returns

$\rightarrow$4

and the full environmental attribute set is

$\rightarrow$5

For each conditional entity $\rightarrow$6 in the inventory,

$\rightarrow$7

with

$\rightarrow$8

These attributes serve two roles. First, they encode the current resource state needed for symbolic reasoning about feasibility. Second, they support coarse spatial control. VistaWise uses thresholds $\rightarrow$9 and $\rightarrow$0 on bounding-box width and height to determine whether an environmental target is within interaction range, effectively using apparent size as a distance proxy in the fixed-FOV Minecraft view (Fu et al., 26 Aug 2025).

An important empirical result is that dedicated object detection is more reliable than relying on multimodal LLM vision alone. Reported experiments with GPT-4o, Gemini-1.5-pro, and Qwen-VL-Max show that configurations using object detection succeed on complex multi-step goals, whereas visual-only configurations fail more often in later tasks. This suggests that the framework’s perception policy is deliberately conservative: it trades broad multimodal understanding for narrow, task-aligned visual grounding (Fu et al., 26 Aug 2025).

4. Cross-modal knowledge graph and retrieval-based pooling

The static backbone of VistaWise is a text-modal knowledge graph

$\rightarrow$1

constructed from online Minecraft knowledge, including wiki-like sources, with GPT-4o used to summarize entities and relations into partial NetworkX graphs. These partial graphs are merged and refined, and the edge vocabulary is restricted to nine relation types: “includes,” “can use,” “can mine,” “can be used to mine,” “is used to craft,” “is used to produce,” “can be put in/on,” “is the fuel of,” and “outputs.” A light manual cleanup removes redundant nodes and edges. Node attributes are intentionally lightweight: the representation keeps entity names and graph structure rather than long textual descriptions (Fu et al., 26 Aug 2025).

At runtime, the detector outputs $\rightarrow$2 and $\rightarrow$3 are injected as dynamic attributes on the corresponding graph nodes, producing the cross-modal graph

$\rightarrow$4

This graph fuses static domain dependencies with the current visual state. An “iron ore” node, for example, carries both symbolic relations to furnaces and pickaxes and dynamic screen-space attributes indicating whether the ore is presently visible and where it is located (Fu et al., 26 Aug 2025).

Because the full graph is too large and too irrelevant in aggregate to be passed directly to the LLM, VistaWise applies a two-stage retrieval-based pooling mechanism. The first stage, Path Searching Pooling (PSP), extracts global task dependencies. For a task $\rightarrow$5, let $\rightarrow$6 denote the “Player” node and $\rightarrow$7 the task target node. PSP finds the path set $\rightarrow$8 from $\rightarrow$9 to $\rightarrow$0 and forms

$\rightarrow$1

This preserves the structural chain of prerequisite entities and actions required for the task (Fu et al., 26 Aug 2025).

The second stage, Entity Matching Pooling (EMP), performs local filtering using the synthetic task prompt $\rightarrow$2 and the current dynamic attributes. It retains

$\rightarrow$3

forming

$\rightarrow$4

This subgraph is then textualized and inserted into the action prompt (Fu et al., 26 Aug 2025).

The reported ablation favors PSP followed by EMP over pure semantic retrieval and over reversing the order of the two pooling stages. Similarity-based retrieval using BERT embeddings and cosine similarity exhibits higher false positive rate and false negative rate, whereas PSP-EMP preserves task structure while still filtering by state relevance. This indicates that graph topology, not only node semantics, is a primary source of competence in VistaWise (Fu et al., 26 Aug 2025).

5. Skill library, prompting, and decision loop

VistaWise executes actions through a desktop-level skill library $\rightarrow$5 built on PyAutoGUI and designed for the vanilla Minecraft 1.11.2 desktop client on Windows 10. This avoids simulator APIs and allows the agent to operate via actual mouse-and-keyboard control, including camera motion, hotbar selection, inventory interaction, crafting, smelting, and mining (Fu et al., 26 Aug 2025).

The skills range from low-level primitives to composite procedures. Examples reported in the framework include turn(x, y), turn_and_move_forward(d, x, y), mine_log(d), mine_diamond_ore(k, d), dig_vertical_mine_tunnels(k, d), craft_plank(l_x, l_y), craft_stick(p_x, p_y), craft_wood_pickaxe(p_x, p_y, s_x, s_y), and smelt_iron_ore(i_o_x, i_o_y, p_x, p_y). Skill code is synthesized with GPT-4o mini from human-designed templates and then lightly refined. The resulting library is parameterized: the LLM must choose both the function and its arguments from visual and graph context (Fu et al., 26 Aug 2025).

Task specification is also structured. For each task $\rightarrow$6, VistaWise defines a task description $\rightarrow$7 and a Chain-of-Thought prompt $\rightarrow$8, yielding

$\rightarrow$9

Typical Chain-of-Thought questions include whether the crosshair is close enough to the target, whether further alignment is needed, and whether the agent must move closer before interacting. The prompt supplied to the LLM also includes the textualized pooled knowledge graph, current inventory attributes, environment attributes, the available skill list, and recent decisions from the memory stack (Fu et al., 26 Aug 2025).

History is maintained as a Last-In-First-Out memory stack $\rightarrow$0. At timestep $\rightarrow$1, a query $\rightarrow$2 with a desired recall depth returns

$\rightarrow$3

This mechanism is meant to preserve recent causal structure without bloating the prompt with long transcripts. The decision loop is therefore: detect entities, update the graph, pool a task-relevant subgraph, build the prompt, call the LLM, execute the emitted skill, and push the result into memory (Fu et al., 26 Aug 2025).

This loop is the operational core of VistaWise. A plausible interpretation is that the framework treats the LLM not as a perceptual model or a motor policy but as a symbolic controller over explicit state abstractions and low-level actuation routines. That division of labor is consistent with the paper’s broader emphasis on cost-effective specialization (Fu et al., 26 Aug 2025).

6. Evaluation, cost-effectiveness, and reported significance

VistaWise is evaluated in vanilla Minecraft 1.11.2 on Windows 10, in Survival mode, in a Plains biome with structures enabled. The benchmark is the classic “obtain diamond” challenge, decomposed into nine ordered sub-goals: obtain log, obtain wooden pickaxe, obtain cobblestone, obtain stone pickaxe, obtain iron ore, obtain furnace, obtain iron ingot, obtain iron pickaxe, and obtain diamond. Experiments are run for 15 repeated trials per configuration (Fu et al., 26 Aug 2025).

The framework reports full success in early and mid-stage sub-goals, 0.73 success rate in iron ingot and iron pickaxe, and 0.33 success rate in the final diamond goal. This 33% “obtain diamond” success rate exceeds the cited baselines ROCKET-1 at 25%, VPT at 15%, JARVIS-1 at 9%, OmniJARVIS at 8%, and STEVE-1 and GROOT at 0%, establishing the paper’s claim of state-of-the-art performance across various open-world tasks (Fu et al., 26 Aug 2025).

Ablations localize the gain to the interaction of all major components. Removing the memory stack causes failure in later stages such as iron pickaxe and diamond. Removing Chain-of-Thought prompting causes failure in earlier stages. Removing the knowledge graph permits some partial progress but fails at diamond. Reported comparisons of visual backbones show that dedicated object detection sharply improves task completion over multimodal LLM vision alone. Retrieval ablations show that PSP-EMP has lower false positive rate and false negative rate than similarity-based retrieval and than applying EMP before PSP (Fu et al., 26 Aug 2025).

Cost-effectiveness is quantified at both training and inference time. Training requires only the 471-frame detector dataset and 24G VRAM. At inference time, VistaWise using GPT-4o over 160 iterations costs about $\rightarrow$425 for Voyager under comparable conditions, a 94.9% reduction. The framework therefore combines a three-order-of-magnitude reduction in domain-specific data scale with a higher reported diamond success rate than several heavily trained predecessors (Fu et al., 26 Aug 2025).

In the context of embodied LLM agents, VistaWise is notable for advancing a specific thesis: strong open-world performance need not require large-scale finetuning of the policy model if domain dependencies are externalized into a cross-modal knowledge graph, perception is narrowed to a dedicated detector, and action is grounded in a parameterized desktop-level skill library. The paper presents this as a cost-effective path to state-of-the-art Minecraft performance, and its broader implication is that explicit structure can substitute for a substantial amount of domain-specific end-to-end training in virtual embodied control (Fu et al., 26 Aug 2025).