- The paper introduces a 5B-parameter action-conditioned video world model that streams 720P interactive environments at up to 16 FPS with 1.2-second action-to-first-frame latency and about 19 GiB of VRAM on one RTX 5090.
- The paper combines feedback-driven data collection, keyboard-action conditioning, progressive bidirectional-to-causal distillation, and LongForcing to reduce autoregressive drift during extended rollouts.
- The paper shows competitive WorldRoamBench results against larger models, including a 0.6752 trajectory score, while hour- and day-scale tests demonstrate persistent scene coherence but reveal remaining weaknesses in long-term memory and quantitative evaluation.
ABot-World-0 is an action-conditioned video world model from the AMAP CV Lab designed for real-time, long-horizon closed-loop interaction on consumer hardware. The system streams 720P video at up to 16 FPS with a 1.2 s action-to-first-frame latency and approximately 19 GiB peak VRAM on a single NVIDIA RTX 5090 (2607.19191). The paper frames interactive world modeling as a full-stack problem spanning data collection, model training, and deployment co-design, and contributes a unified keyboard-action control interface, a multi-source data infrastructure with training-feedback-driven collection, a progressive bidirectional-to-causal distillation pipeline with a novel LongForcing stage, and an optimized streaming inference stack.
Data infrastructure
The training corpus combines three complementary sources: AAA game recordings (the largest source, providing ground-truth synchronized controls via runtime API access), simulation-engine data rendered through Unreal Engine and a 3D Gaussian Splatting backend built on ABot-3DGS over proprietary street aerial photography and scanning assets, and internet videos annotated with pose-estimated pseudo-labels. The paper is explicit that these sources are complementary rather than interchangeable: games give exact supervision but narrow style; simulations give geometric precision; internet video gives natural camera dynamics and domain generalization at the cost of label noise.
A central component is WorldExplorer, an agent-driven collection system applicable to both live games and reconstructed 3DGS scenes. Its navigation agent uses a multi-phase goal-selection strategy (unexplored regions first, then nearby search, then forward movement with collision detection) to maximize coverage. Capture pipelines synchronize video, camera parameters, controls, and environment state to millisecond-precision timestamps with cross-modal alignment error below 33 ms at 30 FPS. The distinguishing feature is the training-feedback-driven closed loop: per-category performance monitoring identifies weak scene–action combinations, and collection ratios adapt automatically while maintaining minimum coverage floors. This converts dataset construction into an active servo coupled to model evaluation rather than a one-time batch process.
Quality filtering applies 14 deterministic checks across six dimensions (file integrity, visual validity, geometric consistency, game-state correctness, action-label alignment, metadata quality), supplemented by VLM-based semantic screening for UI overlays, loading screens, death sequences, and rendering anomalies. Notably, third-stage metadata issues produce soft weighting signals rather than hard rejection, allowing imperfect but informative samples into training. Annotation yields packed frame-level actions, VLM-generated scene descriptions that deliberately omit camera motion (to decouple text conditioning from trajectory signals), semantic tags for stratified sampling, and directional person thumbnails synthesized into canonical frontal portraits for identity conditioning in third-person rollouts.
Model architecture and control interface
ABot-World-0 builds on the Wan2.2 video DiT backbone and conditions generation on raw keyboard inputs represented as an 8-dimensional multi-hot vector per frame (W/A/S/D for movement, I/J/K/L for rotation). Four consecutive frame-level actions are channel-packed into 32-dimensional tokens aligned with the VAE temporal patch size of 4, then injected additively at the patch-embedding stage via an Action Control Adapter (PixelUnshuffle followed by convolutional residual blocks). The paper argues this discrete, bounded action space avoids two failure modes of alternative interfaces: calibrated global camera trajectories drifting out of distribution over long rollouts, and periodic re-anchoring inconsistencies.
For third-person character consistency, a reference-character memory module encodes canonical reference images as identity-memory tokens prepended to the video-token sequence. These tokens receive fixed negative temporal RoPE indices and participate in an asymmetric attention pattern—video tokens attend to memory tokens, but memory tokens remain isolated—providing persistent appearance retrieval throughout autoregressive rollout without contaminating the generated trajectory's positional structure.
Progressive bidirectional-to-causal training
Training proceeds in three stages. First, a bidirectional teacher is fine-tuned on the full multi-source corpus to generate complete clips conditioned on an initial frame, the full action sequence, and multimodal conditions, exploiting full-horizon attention for visual consistency and action alignment. Second, teacher forcing adapts this teacher into a causal student that predicts future chunks from clean ground-truth history under a causal attention mask. Third, causal ODE distillation compresses iterative denoising into few-step inference by learning the probability-flow ODE endpoint map of the frozen Stage-1 causal model under identical causal conditioning, ensuring the distilled flow map depends only on information available at deployment.
The key methodological contribution is LongForcing, the final stage addressing accumulated autoregressive drift. The paper's diagnosis is that local distillation cannot solve closed-loop generation: each student prediction alters the visual context for subsequent predictions, producing long-horizon student-rollout contexts that short-horizon objectives cover only weakly. LongForcing trains on long student self-rollouts and applies Distribution Matching Distillation against an extended-horizon bidirectional teacher, providing distribution-level corrective supervision precisely where errors have had the most opportunity to accumulate. In a 60-second rollout ablation against a Causal-Forcing-style baseline trained identically except for shorter-horizon DMD supervision, differences emerge primarily in the second half of the rollout: the baseline shows declining HPSv3 alongside rising high-saturation pixel ratio, perceptual blur, and patch repetition, while LongForcing retains higher HPSv3 and lower artifact metrics. The authors also note a trade-off in the broader design space: sink-based stabilization and fixed reference anchoring can delay drift but risk restricting scene evolution—an assumption worth noting when interpreting the stability results.
Deployment co-design
The systems analysis supports one of the paper's stronger claims: few-step sampling alone does not yield real-time interaction. Both the base configuration and the SageAttention2-only variant run out of memory on the RTX 5090 at 1280×704 resolution; memory feasibility requires joint optimization across decoding, precision, attention, positional encoding, and scheduling.
| Configuration |
DiT (ms/chunk) |
VAE (ms/chunk) |
FPS |
VRAM (GiB) |
| Base |
— |
— |
OOM |
OOM |
| + SageAttention2 |
— |
— |
OOM |
OOM |
| + LightVAE |
1191.1 |
78.3 |
9.117 |
20.491 |
| + FP8 |
845.2 |
76.0 |
12.405 |
15.925 |
| + Fast-RoPE |
786.9 |
71.7 |
13.269 |
19.281 |
| + MXFP6 |
718.3 |
86.0 |
14.098 |
18.287 |
| + MXFP4 |
638.8 |
73.0 |
15.831 |
17.148 |
Each chunk contains 3 latent frames decoded into 12 video frames. The lightweight TAEHV-inspired decoder (LightVAE) is what makes the pipeline feasible at all, indicating the original VAE was a primary memory bottleneck. FP8 DiT quantization cuts DiT time by roughly 29% and peak memory by about 4.6 GiB. Fast-RoPE re-anchors temporal RoPE within the local attention window and uses a Triton kernel, further reducing overhead—though its configuration raises measured peak VRAM to 19.281 GiB, illustrating that peak memory is determined by the whole runtime configuration rather than any single operator. More aggressive MXFP4/MXFP6 formats on DiT linear layers (with VAE and text encoder kept in higher precision) extend throughput toward 16 FPS. Bounded local-context KV caching with rolling eviction keeps cache footprint independent of rollout duration, and TurboQuant-style online vector KV quantization is explored as a complement. The reported 1.2 s latency covers the full path from keypress to first decoded response frame, not sampling speed in isolation—a more conservative metric than many prior reports. The paper concedes that quantization-aware training could improve the speed–memory–quality trade-off but leaves it unexplored.
Evaluation
On WorldRoamBench, ABot-World-0 (5B parameters) is compared against Genie 3, HappyOyster, LingBot-World (14B), and HY-World 1.5 (8.3B). It achieves second-best scores on strict action accuracy (0.5266 vs. HappyOyster's 0.5317), partial accuracy (0.7290 vs. 0.7631), trajectory score (0.6752 vs. Genie 3's 0.6719—here ABot trails only marginally), aesthetics (0.5039), and imaging (0.4651), while trailing on mechanics (0.5223 vs. Genie 3's 0.5454) and memory (0.5041 vs. HappyOyster's 0.6309). The competitive showing at less than half the parameter count of LingBot-World is notable, though the memory deficit relative to HappyOyster indicates room for improvement in long-range state retention.
Qualitative stress tests extend well beyond the benchmark horizon: hour-scale rollouts across five independent runs preserve coherent environments and responsive control, and day-scale rollouts retain recognizable scene structure and active motion at sampled checkpoints under severe accumulated-error conditions. Out-of-domain evaluations show the unified action interface generalizing to unseen scene–character combinations, and physical-interaction cases (object collision, water disturbances, persistent snow footprints, wall blocking, railing collision without interpenetration) demonstrate emergent physical plausibility despite no explicit symbolic physics or collision annotations—the paper attributes these behaviors to large-scale interactive video experience rather than engineered constraints.
Limitations and open questions
Several limitations are stated or evident. Internet-video pseudo-labels carry pose-estimation noise absent from game and simulation ground truth, and the quality of the unified action space depends on thresholding choices during displacement binarization. The additive patch-level injection is justified specifically for explicit discrete actions; the paper acknowledges that ambiguous signals such as latent actions, continuous trajectories, or semantic instructions may require richer conditioning mechanisms. The day-scale results are supported by sampled checkpoints rather than continuous quantitative evaluation, so degradation behavior between checkpoints remains uncharacterized. Quantitatively, the memory sub-dimension lags the leading competitor, suggesting that reference-character memory and bounded context do not yet fully close the long-horizon retention gap. Open questions include whether multi-scale LongForcing and persistent scene memory can further improve consistency, whether quantization-aware training improves the deployment trade-off, and how the stack extends to lower-capability consumer GPUs.
Conclusion
ABot-World-0 demonstrates that unified keyboard-based interactive control, stable long-horizon autoregressive rollout, and real-time 720P streaming can be combined in a single 5B-parameter video world model running on one desktop GPU. Its principal technical contributions—training-feedback-driven data collection, LongForcing's extended-horizon distribution matching over student self-rollouts, and full-stack inference co-design—are individually grounded in ablation and system measurements, and together they support the paper's central thesis that interactive world modeling is a systems capability rather than a single generative objective.