MineWorld: Real-Time Minecraft World Model

Updated 29 March 2026

MineWorld is a real-time, open-source interactive world model for Minecraft that unifies perceptual and action dynamics in a single autoregressive framework.
It employs a unified visual–action Transformer with specialized tokenizers for image and action inputs, achieving efficient diagonal-parallel decoding.
The model provides strong action control metrics and serves as a baseline in research on spatial consistency and controllable video generation despite limitations in long-term memory.

MineWorld is a real-time, open-source interactive world model for Minecraft, providing a unified framework for world simulation and policy learning. Unlike classical simulation environments or pure video prediction models, MineWorld models both perceptual and action dynamics as a single, token-based autoregressive process, achieving high frame rates and strong action-controllability in an open-ended, sandbox context. It is widely referenced as a baseline in research on world models, spatial consistency, controllable video generation, foundation models, and embodied agent evaluation within Minecraft-like environments (Guo et al., 11 Apr 2025).

1. Visual–Action Autoregressive Transformer Architecture

MineWorld employs a single, unified visual–action autoregressive Transformer to simulate forward dynamics. At time step $i$ , the system receives an RGB Minecraft frame $x_i$ and a user action $a_i$ (mouse/keyboard inputs). Both modalities are converted to sequences of discrete token IDs using specialized tokenizers—a fine-tuned VQ-VAE (for vision) and quantization/discretization for actions (mouse angles, keypresses).

The input to the Transformer is a concatenated sequence $t = [t_1, t_2, \dots, t_N]$ , with interleaved vision and action tokens across successive steps: $f_\theta(t) = \prod_{j=1}^N p_\theta(t_j \mid t_{<j}),$ where some $t_j$ are image patches, and others are quantized actions. The Transformer is trained using a next-token cross-entropy loss: $\mathcal{L}(\theta) = -\sum_{j=1}^N \log p_\theta(t_j \mid t_{<j}).$ This formulation couples the modeling of state transitions ( $x_{i+1}$ given $(x_{\leq i}, a_i)$ ) and action prediction (policy modeling of $a_{i+1} \mid x_{\leq i}$ ) (Guo et al., 11 Apr 2025).

2. Tokenization Schemes and Input Structure

The image tokenizer employs a VQ-VAE (initialized from Amused), compressing $224\times384$ RGB frames into $14\times24=336$ tokens per frame (16 $\times$ spatial compression). The action tokenizer quantizes camera rotations (11 bins per axis) and encodes keypresses into 7 classes, with boundary tokens ([aBOS], [aEOS])—resulting in 11 tokens per action.

At each timestep, image and action tokens are interleaved: $(t_{i\cdot c + 1},\dots,t_{(i+1)\cdot c},\,\text{[aBOS]},\,t^a_{1},\dots,t^a_{9},\,\text{[aEOS]})$ For 16 time steps, the total input is approximately $5{,}552$ tokens. This discrete, densely interleaved sequence enables the Transformer to learn complex dependencies between perceptual and control states within a fixed-length context ( $5{,}376$ maximum).

3. Inference: Diagonal Parallel Decoding Algorithm

Autoregressive decoding of high-resolution video tokens is computationally expensive, leading to low practical frame rates. MineWorld introduces a diagonal-parallel decoding algorithm: each frame’s token grid (of size $h\times w$ ) is filled diagonal-by-diagonal, predicting all tokens along each diagonal in parallel. Pseudocode:

initialize s = previous frames’ tokens, action tokens
for diag = 1 to (h + w – 1):
    positions = {(i,j) | i + j = diag}
    logits = Transformer(s)
    for each (i,j) in positions:
        s.append(argmax logits at position (i,j))
end

The speedup relative to raster scan is

r = \frac{h w}{h + w - 1}

; MineWorld’s

14\times24

grid yields

r \approx 3.6

versus sequential. Empirical frame rates range from $5.91$ FPS (300M params) to $3.01$ FPS (1.2B params), substantially exceeding standard autoregressive baselines (Guo et al., 11 Apr 2025).

4. Evaluation Protocols and Controllability Metrics

Beyond classical video-generation criteria (FVD, PSNR, SSIM, LPIPS), MineWorld introduces metrics for action fidelity:

Inverse Dynamics Model (IDM): Given a frame pair $(\hat{x}_i, \hat{x}_{i+1})$ , a pretrained IDM predicts the underlying action $\hat{a}_i$ . Discrete action classification F1, precision, recall (grouped into seven and four binary action tasks) are macro-averaged.
Camera Movement L1 Loss: For camera controls, the L1 distance is computed between predicted and true quantization bins:

$\mathcal{L}_{\mathrm{cam}} = \mathbb{E}[|\hat{c} - c|].$

Empirical results show MineWorld (1.2B) improving FVD from $377 \to 227$ , F1 from $0.41 \to 0.73$ , and L1 camera loss from $2.60 \to 1.02$ compared to the diffusion-based Oasis baseline (Guo et al., 11 Apr 2025).

5. Memory and Spatial Consistency: Current Limitations

MineWorld, while effective for short sequences and local controllability, retains only a sliding window (15 frames) of context. Benchmarking on LoopNav (Lian et al., 29 May 2025) shows sharp degradation in loop-closure spatial consistency: even at minimal navigation ranges ( $r=5$ ), SSIM drops to $0.31 \pm 0.09$ , LPIPS is $0.73 \pm 0.05$ , and FVD for return segments reaches $2\,089 \pm 1\,007$ . Without explicit long-horizon memory (such as key–value storage, map memory, or learned room embeddings), the model “hallucinates” scene elements rather than returning to consistent renderings upon revisiting locations.

Recommendations include augmenting with coordinate-keyed external memory, hybrid egocentric map-latent memory, or clustering and attending to “room” embeddings, thereby supporting true loop closure and planning-ready simulation (Lian et al., 29 May 2025).

6. Position Relative to Contemporary Foundation and Multiplayer Models

Subsequent models such as Matrix-Game (17B parameters, 2.7K hours data) employ two-stage training (unlabeled pretraining, action-labeled controllable generation), latent diffusion, and advanced action-conditional DiTs, significantly surpassing MineWorld in both controllability (Keyboard Acc $+10.5\%$ , Mouse Acc $+48.4\%$ ) and physical consistency (Obj. Consist. $+49.0\%$ ) under the GameWorld Score suite (Zhang et al., 23 Jun 2025). Multiplayer extensions (Solaris) add cross-agent memory, grounding, and view coherence via multiplayer self-attention, staged causal/bidirectional/self-forcing training, and a data collection framework for coordinated episodes (Savva et al., 25 Feb 2026). These advances highlight MineWorld’s strengths in real-time single-agent modeling, while exposing its limitations in long-horizon memory and multi-agent environments.

7. Impact, Open Source Contributions, and Future Directions

MineWorld’s key contributions to the field are:

The first open-sourced, real-time, visual–action world model for Minecraft, unifying video and control streams in a single autoregressive Transformer
A diagonal parallel decoding strategy enabling frame rates of 4–7 FPS, with practical support for real-time interactive use
Introduction of action-following controllability metrics alongside perceptual ones, establishing a new evaluation protocol in open-ended simulated world tasks

All code, three pretrained checkpoints (300M, 700M, 1.2B), and data preparation are released publicly (Guo et al., 11 Apr 2025).

Given the current absence of explicit memory, research converges on the necessity of spatially-grounded memory modules for robust planning, spatial consistency, and reliable open-world simulation. This has set the roadmap for successor models (including MineWorld 2.0), aiming at multi-agent, infinite-horizon, and semantically structured world modeling (Lian et al., 29 May 2025, Zhang et al., 23 Jun 2025, Savva et al., 25 Feb 2026).