MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft (2504.08388v1)

Abstract: World modeling is a crucial task for enabling intelligent agents to effectively interact with humans and operate in dynamic environments. In this work, we propose MineWorld, a real-time interactive world model on Minecraft, an open-ended sandbox game which has been utilized as a common testbed for world modeling. MineWorld is driven by a visual-action autoregressive Transformer, which takes paired game scenes and corresponding actions as input, and generates consequent new scenes following the actions. Specifically, by transforming visual game scenes and actions into discrete token ids with an image tokenizer and an action tokenizer correspondingly, we consist the model input with the concatenation of the two kinds of ids interleaved. The model is then trained with next token prediction to learn rich representations of game states as well as the conditions between states and actions simultaneously. In inference, we develop a novel parallel decoding algorithm that predicts the spatial redundant tokens in each frame at the same time, letting models in different scales generate $4$ to $7$ frames per second and enabling real-time interactions with game players. In evaluation, we propose new metrics to assess not only visual quality but also the action following capacity when generating new scenes, which is crucial for a world model. Our comprehensive evaluation shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion based world models significantly. The code and model have been released.

• The paper introduces MineWorld, a real-time, open-source interactive world model for Minecraft that uses a visual-action autoregressive Transformer.
• A novel parallel decoding algorithm boosts generation speed to 4-7 FPS, enabling real-time interaction, and achieves state-of-the-art results on visual quality and action following.
• MineWorld provides a robust open-source platform for developing AI agents in gaming and simulation, with potential applications in virtual training and robotics.

MineWorld: A Comprehensive Study of an Interactive World Model in Minecraft

The paper "MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft" introduces MineWorld, a transformative advance in the domain of world modeling. This paper presents both theoretical and practical advancements in creating real-time interactive models within the popular sandbox environment of Minecraft. MineWorld leverages a visual-action autoregressive Transformer framework, driving innovation in the interaction between intelligent agents and digital environments.

Framework and Methodology

MineWorld employs a visual-action autoregressive Transformer to model the dynamic environments within Minecraft. It represents both game scenes and corresponding actions as discrete token IDs using an image tokenizer and an action tokenizer. These tokenized inputs are interleaved and processed sequentially by a Transformer model, trained to predict the next token in the sequence. The use of tokenization and autoregressive methods ensures the model learns rich, accurate representations of game states and the relationship between states and actions.

A significant technical contribution outlined in the paper is the development of a novel parallel decoding algorithm. This algorithm boosts the generation speed to 4 to 7 frames per second, facilitating real-time interaction capabilities—a feature of paramount importance in interactive applications.

Evaluation Metrics and Results

The authors propose new metrics for evaluating the visual quality and action following capacity of generated scenes. These metrics are pivotal for assessing how well the model simulates and adheres to the dynamics of the game world. The comprehensive evaluation demonstrates that MineWorld significantly exceeds state-of-the-art (SoTA) performance metrics compared to open-sourced diffusion-based world models. The reported results highlight the model's proficiency not only in visual fidelity but also in maintaining control accuracy during gameplay simulations.

Implications and Future Prospects

Practically, MineWorld's open-source nature can significantly impact gaming and simulation technologies by providing a robust platform for developing AI-driven game agents and simulations. Theoretically, its architecture and mechanisms can extend to other domains requiring real-time large-scale interactive modeling, such as virtual training environments and robotic simulations.

The paper speculates on the avenues for future research, including potential enhancements in tokenization strategies, finer granularity in video modeling, and scalability to broader virtual environments beyond Minecraft.

Conclusion

"MineWorld" represents a strategic intersection of artificial intelligence, interactive modeling, and gaming applications. With its innovative methodology and real-time capabilities, it sets a new benchmark in world modeling. This work not only addresses immediate challenges in creating efficient, controllable virtual models but also opens up broader discussions on AI's role in fostering dynamic virtual environments. Future work could focus on expanding its scope and application, further enriching the landscape of AI-driven virtual interactivity.