minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models

Abstract: Recent video diffusion foundation models have achieved remarkable progress in high-quality video generation, yet turning them into real-time interactive video world models remains challenging. Interactive world models require controllable, causal, and low-latency rollout, which in practice demands a full pipeline spanning data construction, controllable fine-tuning, autoregressive training, few-step distillation, and streaming inference. In this work, we present minWM, a full-stack open-source framework for building real-time interactive video world models. minWM provides an end-to-end pipeline that converts existing bidirectional T2V/TI2V video foundation models into camera-controllable few-step autoregressive world models. Specifically, minWM first fine-tunes a bidirectional video diffusion model with camera control, and then applies the Causal Forcing / Causal Forcing++ pipeline, including AR diffusion training, causal ODE or causal consistency distillation, and asymmetric DMD, to distill it into a few-step autoregressive generator for low-latency rollout. The framework is modular and architecture-extensible: we instantiate it on representative open backbones, including Wan2.1-T2V-1.3B and HY1.5-TI2V-8B, covering both cross-attention-based condition injection and MMDiT-style architectures. minWM also supports adapting existing video world models, such as HY-WorldPlay, to new data distributions, training recipes, and latency targets. Beyond releasing runnable scripts, checkpoints, documentation, and inference code, we provide practical ablations on camera trajectory quality, controllability training steps, and minimal batch-size requirements. We hope minWM serves as a reproducible and extensible recipe for building and adapting real-time interactive video world models. Project Page: https://github.com/shengshu-ai/minWM

• The paper introduces minWM, a framework that transforms bidirectional T2V/TI2V models into camera-controllable autoregressive world models with significantly reduced latency.
• It employs innovative techniques like PRoPE injection and causal forcing distillation to incorporate camera parameters, ensuring high visual fidelity during interactive generation.
• Experimental results demonstrate over 220× speedup in first-frame latency, validating minWM’s effectiveness for real-time video simulation and interactive applications.

minWM: A Comprehensive Pipeline for Real-Time Interactive Video World Models

Framework Overview

The minWM framework presents an integrated, open-source solution for constructing real-time interactive video world models by transforming powerful bidirectional T2V (text-to-video) and TI2V (text-and-image-to-video) foundation models into camera-controllable, low-latency few-step autoregressive world models. The need for interactive video world models arises from practical requirements for causal rollout, instantaneous response to user-driven camera actions, and high visual fidelity during continued generation. The minWM pipeline comprehensively addresses these demands through modular data construction, controllable fine-tuning, autoregressive training, efficient distillation algorithms, and optimized streaming inference.

Figure 1: minWM full-stack pipeline converting T2V/TI2V foundation models into camera-controllable few-step autoregressive world models.

Methodology

Camera-Controllable Bidirectional Diffusion Training

The first phase of the pipeline fine-tunes bidirectional video diffusion backbones to be camera-controllable. The PRoPE injection mechanism encodes camera parameters via projective matrices, seamlessly introducing both intrinsic and extrinsic camera information into the model's self-attention layers. This allows explicit conditioning on relative camera poses and intrinsics within the generative process, enabling the model to follow complex camera trajectories while preserving generation quality.

Autoregressive Diffusion Distillation

The transformation into a few-step AR model proceeds via the Causal Forcing/Causal Forcing++ pipeline, comprising:

1. Teacher-forced AR diffusion training: Fine-tunes the model under a causal attention mask to enable autoregressive generation. However, this stage inherits latency and exposure bias limitations from conventional multi-step AR models.
2. Causal ODE/Consistency Distillation Initialization: Optionally, the pipeline leverages causal ODE distillation by regressing intermediate noisy states to clean target frames using trajectories from the AR teacher, or causal consistency distillation (causal CD) as a storage-efficient, data-efficient alternative. Both approaches yield a few-step AR generator suitable for low-latency rollout.
3. Asymmetric Distribution Matching Distillation (DMD): Aligns the autoregressive student to the high-quality distribution of the bidirectional teacher through self-rollout and DMD optimization, substantially mitigating the AR teacher's quality gap.

Camera conditions are consistently injected throughout all stages to preserve and strengthen controllability. This modular distillation design enables adaptation to various backbone architectures, including cross-attention and MMDiT-style models.

Experimental Results

Latency and Controllability

minWM demonstrably accelerates interactive generation. Empirical results establish a 223.75× speedup for the few-step AR HY1.5 model and a 236.64× speedup for the Wan2.1 model in first-frame latency compared to their multi-step bidirectional counterparts. This performance translates to practical deployment advantages, permitting on-the-fly streaming initiation for end users.

Figure 2: Camera-controllable generation, demonstrating preservation of base model controllability after distillation.

Ablation Studies

Training Data

Models trained from datasets with perception-estimated camera poses (e.g., SpatialVid) exhibit inferior controllability, likely due to pose noise and inconsistency. Substituting ground-truth trajectories via 3D reconstruction (DL3DV) or WorldPlay-driven synthetic trajectories proves critical for learning robust camera control.

Figure 3: Training on SpatialVid data without ground-truth camera trajectories fails to yield reliable camera-controllable generations.

Training Steps

Bidirectional diffusion models gradually acquire camera controllability; uncontrollable at $<2000$ steps, partial control emerges at $~5000$ steps, and reliable controllability is reached by $~8000$ steps.

Figure 4: HY1.5-based model remains uncontrollable after early-stage training (<2000 steps).

Batch Size

The minimal batch size for effective camera-controllable learning is found to be $\geq 4$ (Wan2.1 example), with increased stability and robust controllability at batch sizes of $16$.

Figure 5: Wan2.1-based model is unable to learn camera-controllable behavior with batch sizes below 4.

Implications and Future Directions

minWM offers an extensible recipe for reproducibility and adaptation, supporting both the conversion and fine-tuning of diverse video foundation models and world models for interactive, camera-driven tasks. Its rigorous ablation studies provide actionable guidance for dataset construction, training strategy selection, and hardware efficiency optimization. The unified approach bridges previously fragmented pipelines, enabling low-latency, causal, and controllable rollout suitable for emerging real-time applications in embodied simulation, virtual environments, and generative human-computer interaction.

Theoretically, minWM's distillation strategy, particularly the Causal Forcing/Causal Forcing++, delineates a reliable path from high-quality bidirectional diffusion to efficient autoregressive generation without compromising on controllability. Practically, the framework catalyzes rapid experimentation across backbones and datasets, and offers a scalable foundation for further research.

Looking forward, incorporating additional control modalities (e.g., pose, action, memory, embodiment) and expanding architectural compatibility will further enhance world modeling fidelity and interactivity. Integration with memory architectures, reinforcement learning, and policy-driven generative agents may amplify the scope of real-time world simulation.

Conclusion

minWM establishes a structured, reproducible foundation for transforming bidirectional video diffusion models into interactive, causal, and low-latency camera-controllable AR world models. Its modular pipeline, strong numerical speedup, and practical ablation findings set a benchmark for future developments in real-time video generative modeling and world simulation (2605.30263).