WORLDMEM: Long-term Consistent World Simulation with Memory (2504.12369v1)

Abstract: World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.

Long-term Consistent World Simulation with Memory

This paper presents "WorldMem," a novel framework designed to address the challenge of maintaining long-term consistency in simulated world environments through an innovative memory storage mechanism. In the field of world simulation, preserving consistency—especially over extended periods—is crucial yet challenging, as existing methods often suffer from inconsistencies stemming from limited temporal context windows. WorldMem offers a compelling solution by introducing a memory bank that stores sequences of previously generated visual scenes paired with their respective states, including poses and timestamps. By leveraging a memory attention mechanism, the model effectively retrieves pertinent past information, enabling the accurate reconstruction of scenes even when facing significant temporal and spatial gaps.

Methodology and Implementation:

The key contribution of this work lies in the integration of an external memory bank within a Conditional Diffusion Transformer (CDiT) framework, which operates beyond the traditional temporal constraints. This memory bank is populated with memory units that store visual and state information from past frames. In addition, action signals are utilized to condition the framework, further enhancing the simulation of dynamic and interactive environments.

Training involves the application of Diffusion Forcing (DF), a noise-level specific denoising approach that facilitates an autoregressive generation process. During inference, the model utilizes these memory frames, deemed as “clear” latents within the memory blocks, to guide the generation of new frames without distorting the context of previously consistent environments. Importantly, these memory frames are not bound by typical time horizon constraints, allowing for long-term spatial and temporal coherence.

The memory retrieval process is optimized using a similarity-based strategy, ensuring that only the most relevant memory units are accessed during generation. This strategy thoughtfully considers both field-of-view overlap and temporal proximity to evaluate the relevance of stored frames.

Experiments and Evaluation:

Extensive experiments, conducted in the superficial environments of Minecraft and the real-world settings of RealEstate10K, demonstrate the framework's efficacy. In Minecraft, WorldMem achieves superior spatial consistency and robust scene reconstruction, significantly outperforming baseline methods in both short-term and long-term scenarios. Quantitative assessments using metrics such as PSNR, LPIPS, and rFID further affirm the framework’s capacity to deliver high fidelity and visually coherent simulations across thousands of generated frames. Notably, the performance is maintained over sequences extending beyond typical context windows of baseline models.

The practical implications of WorldMem are underscored in dynamic environments, as showcased by its ability to track temporal changes, like weather variations and object movements, with high accuracy. By integrating timestamps, the framework extends its capability to model the time-evolving aspects of environments, which has promising applications in areas requiring persistent and immersive virtual simulations, such as autonomous navigation and gaming.

Future Directions:

While the introduction of a memory-enhanced world simulator represents a substantial stride forward, the authors acknowledge limitations requiring future exploration. Potential developments include further optimizing memory utilization to address increasing memory demands and refining retrieval algorithms to improve precision. Expanding the diversity and realism of interactions in simulated environments will also be a priority. Crucially, while this work primarily tackles memory-enhanced consistency, understanding its applicability in broader contexts such as real-time strategy simulation and complex interaction modeling will be essential in future explorations.

In summary, this paper introduces a significant advancement in world simulation models, providing a framework that proficiently simulates consistent and interactive worlds. WorldMem sets the stage for continued research into memory-based approaches, which are pivotal for enhancing the robustness and reliability of virtual world simulations.