WorldEngine Runtime: Real-Time Simulation

• WorldEngine Runtime is a real-time interactive video inference system that repurposes a causal autoregressive prior for low-compute simulation on consumer GPUs.
• It integrates LongLive 1’s video model with Matrix-Game 2.0’s action module, enabling live keyboard/mouse control, mid-stream reprompting, and dual-view operation.
• Performance tests report 14–15 FPS on an RTX 4090 with low memory usage, demonstrating a cost-effective approach for interactive world-model previews.

Searching arXiv for the cited paper and named precursor systems to ground the article with current identifiers. WorldEngine Runtime is the streaming inference runtime described for DreamForge-World 0.1 Preview, a preview foundational world model for real-time interactive world simulation that targets a complementary axis to frontier-scale world simulators: low-compute adaptation, consumer-GPU runtime, and broad interactive capability coverage (Ayupov et al., 29 Jun 2026). In this formulation, the runtime repurposes a causal autoregressive video prior into a closed-loop interactive system with live keyboard and mouse control, multimodal initialization, mid-stream reprompting, dual-view operation, and minute-scale interactive rollouts at native 480p resolution. The reported implementation reaches up to 14 to 15 FPS on a single RTX 4090 with a low memory footprint, while the paper explicitly states that DF-World 0.1 Preview is not yet a memory-complete or frontier-quality world simulator.

1. Definition and system scope

WorldEngine is defined around a streaming, causal autoregressive video generator whose two main pillars are the LongLive 1 causal AR video prior and a residual action module transplanted from Matrix-Game 2.0 (Ayupov et al., 29 Jun 2026). The resulting runtime accepts multimodal context, injects live keyboard and mouse actions, and emits 480×832 frames in real time. The paper situates this design as a practical low-compute route toward real-time controllable world-model previews on consumer GPUs rather than as an attempt to match frontier-scale simulators on memory completeness or overall quality.

This positioning is central to interpreting the runtime. The paper does not present WorldEngine as a general solution to long-horizon coherent simulation; instead, it emphasizes cost-efficiency, targeted adaptation runs, and deployment viability on a single consumer GPU. A plausible implication is that the runtime is best understood as an engineering substrate for interactive world-model previewing, where controllability and throughput are prioritized alongside broad interface coverage.

2. Architectural composition

At the model level, WorldEngine retains the entire LongLive 1 runtime, including prompt-switching and cache-reuse logic, and layers all DreamForge modifications on top via lightweight adapters and quantized cache tweaks (Ayupov et al., 29 Jun 2026). LongLive 1 provides frame-level autoregressive diffusion on top of Wan2.1-T2V-1.3B’s DiT backbone, cacheable KV layers with short-window attention and frame-sink context, and streaming-tuned inference through asynchronous kernel launches and overlapping VAE decode.

To add interaction, the runtime borrows the Matrix-Game 2.0 action-conditioning architecture. The described procedure has three steps: transplant the pretrained Matrix-Game 2.0 action-module weights into the LongLive 1 DiT; fine-tune the combined model on 64 hours of gameplay videos with synchronized keyboard/mouse labels; and split the result into two view-specific control checkpoints via high-rank LoRA, namely a first-person LoRA for egocentric navigation and a third-person LoRA for external camera plus character locomotion. At inference time, the chosen view simply loads the corresponding LoRA-fused weights into the action path.

The architecture therefore combines a pretrained causal video prior with a residual control interface rather than redesigning the backbone. This suggests that WorldEngine’s contribution is concentrated in adaptation strategy and runtime orchestration: the LongLive 1 stack supplies temporal autoregression and cache semantics, while the Matrix-Game–derived pathway supplies action-conditioned controllability.

3. Per-frame inference pipeline

The runtime’s data flow is specified as a multimodal-to-action-conditioned frame loop (Ayupov et al., 29 Jun 2026). Inputs originate in a user input layer that may contain a text prompt, an image or video initialization, and live keyboard/mouse input. These form a multimodal prefix that enters a latent history buffer containing up to $N$ past frames. Prompt tokens and an action embedding supplied through the residual path are then fed into the diffusion transformer, which uses short-window attention and a quantized KV-cache. The transformer produces a new latent frame, which is decoded by a VAE decode module identified as either the Wan2.1 VAE or LightTAEW2.1, yielding an RGB frame that is displayed and appended to history.

A defining property of the implementation is asynchronous execution where possible. The paper states that steps such as DiT compute and VAE decode are overlapped and that the runtime shares a single GPU memory pool. This organization is directly tied to the real-time objective: the system is not merely autoregressive in a modeling sense, but streamed as a closed-loop inference service in which display output immediately becomes part of the conditioning history for subsequent generation.

The same section also specifies how user control enters the loop. At each diffusion step, the runtime polls OS-level keyboard/mouse buffers asynchronously, embeds continuous mouse motion and discrete button presses into a small action vector, and injects this vector through the residual action pathway into the DiT’s intermediate layers. The reported synchronized labels include move, jump, sprint, crouch, RMB/L2, and LMB/R2. In operational terms, this means that each generated frame is conditioned on the most recent controls rather than on a coarser control schedule.

4. Low-compute adaptation and runtime optimization

The low-compute profile is attributed to a cluster of runtime and adaptation techniques rather than to pruning (Ayupov et al., 29 Jun 2026). The paper lists parameter-efficient adapters using rank-64 and high-rank LoRAs for both backbone and action paths, low-precision execution in bf16 and 8-bit weight modes, KV-cache quantization to 4-bit, Deep-Forcing style cache management, and asynchronous pipelining that overlaps frame generation and decoding while prioritizing diffusion transformer kernels for minimal idle time.

The cache strategy is especially important in the stated design. KV-cache quantization compresses past key/value tensors to 4-bit in order to reduce activation memory, while the Deep-Forcing style cache mechanism is described as a training-free “frame sink” that periodically refreshes cache to mitigate drift without extra forward passes. Combined with prompt-switching via KV recache inherited from LongLive 1, these methods indicate that cache management functions both as a memory optimization and as a control mechanism for long-running interactive sessions.

The paper explicitly states that no model pruning was applied and that the small footprint arises from adapter tuning, quantization, and compute-efficient runtime kernels. This narrows the interpretation of “low-compute”: it does not denote a smaller backbone, but rather an execution regime that makes an existing open video backbone tractable for real-time interactive use on consumer hardware.

5. Control interfaces and mode switching

WorldEngine exposes three interactive control surfaces within the same autoregressive loop: live keyboard and mouse integration, mid-stream reprompting, and dual-view operation (Ayupov et al., 29 Jun 2026). The first is implemented through the residual action pathway, allowing continuous mouse motion and discrete button presses to modulate intermediate DiT layers on a per-frame basis. The second relies on LongLive’s KV-cache semantics: when a new text prompt is issued, the runtime triggers a KV recache with the updated prompt tokens, retains the latent frame history unchanged, and resumes streaming inference under the new semantic condition.

Dual-view operation is implemented by swapping between two separate action-module checkpoints at session start. First-person mode fuses the FPV LoRA into the action path, whereas third-person mode fuses the TPV LoRA. The paper states that no architectural switch is needed mid-rollout; the desired LoRA is loaded and autoregression continues.

These interface choices clarify what “controllable” means in the runtime. Control is not restricted to initial prompting or post hoc editing; rather, it is interleaved with frame synthesis through prompt recaching, action-vector injection, and view-specific LoRA fusion. At the same time, the paper notes a practical asymmetry: stability is strongest in FPV, and excessively aggressive camera rotations in TPV mode should be avoided during minute-scale rollouts.

6. Performance characteristics, formalization, and deployment

All reported numbers are measured end-to-end in the DreamForge runtime at 480×832 resolution and include DiT, action conditioning, VAE decoding, and streaming overhead (Ayupov et al., 29 Jun 2026). The paper reports the following empirical throughput and memory figures on an RTX 4090:

Path and precision	Throughput	VRAM
Default Wan2.1 VAE, bf16	~10 FPS	~9 GB
Default Wan2.1 VAE, fp8	~12 FPS	~5 GB
LightTAEW 2.1, bf16	~12 FPS	~8 GB
LightTAEW 2.1, fp8	~14–15 FPS	~4 GB

The corresponding latency figures are given as approximately 100 ms/frame for bf16/Default and 66–70 ms/frame for fp8/LightTAEW. The paper also formalizes throughput as inversely related to per-frame inference time, $\mathrm{FPS} = 1 / T_{\mathrm{inf}}$, and specifies memory footprint as

$$M_{\mathrm{total}} = M_{\mathrm{model}} + M_{\mathrm{activation}} .$$

Compute cost in FLOPs is expressed as

$$C_{\mathrm{FLOPs}} = \sum_{\ell}\Bigl(2\,n_{\mathrm{MACs}}^{(\ell)}\Bigr) .$$

The deployment guidance is correspondingly concrete. The recommended hardware is an NVIDIA RTX 4090 with at least 24 GB VRAM or comparable hardware; the recommended precision/runtime path for hitting 14–15 FPS is fp8 plus LightTAEW, with approximately 4 GB VRAM; and the software stack is PyTorch with NVIDIA’s 8-bit kernels and custom C++/CUDA ops for KV quantization. For minute-scale rollouts, the paper advises beginning with a multimodal seed such as text plus image/video, using mid-stream reprompting every 15–30 s to re-inject semantic coherence and reduce drift, monitoring VRAM headroom, clearing Python caches or restarting the runtime every few minutes if memory fragmentation appears, and, for the longest qualitative consistency, fusing LoRAs ahead of time, enabling Deep-Forcing periodic cache refresh, and preferring fp8 quantization on consumer GPUs.

A common misconception would be to equate these deployment figures with a claim of frontier-quality simulation. The paper makes the opposite point: the runtime demonstrates a practical low-compute route to real-time controllable world-model previews on consumer GPUs, but it is not yet a memory-complete or frontier-quality world simulator. In that sense, its significance lies in showing that live keyboard/mouse control, mid-stream reprompting, dual-view modes, and minute-scale real-time rollouts can be integrated into a single-GPU causal video runtime with a modest adaptation budget and memory envelope.