RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Abstract: Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.

• The paper presents a physically-grounded 4D world model, RynnWorld-4D, that synchronizes RGB, depth, and optical flow to enhance robotic manipulation.
• It employs a tri-branch diffusion model with joint cross-modal attention and phased training to ensure spatial and temporal consistency.
• It demonstrates state-of-the-art performance with robust geometric accuracy (δ1 of 0.610) and effective policy integration achieving real-time manipulation at ~9 Hz.

RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation

Introduction and Motivation

Recent advances in world models for robotics have focused on pixel-based generative models, but the lack of explicit 3D reasoning has led to substantial limitations in spatial understanding and action synthesis. Conventional 2D video generation models are fundamentally constrained by their projective nature, limiting their applicability for precise manipulation tasks that require 6-DoF pose reasoning and temporally consistent geometry. RynnWorld-4D addresses these challenges by introducing a physically-grounded, scalable 4D world model framework composed of synchronized RGB, depth, and optical flow modalities (RGB-DF). This architecture is designed to explicitly encode appearance, geometry, and motion, simultaneously narrowing the sim-to-real gap and providing a representational substrate that is more congruent with end-effector action spaces.

Figure 1: Given an input RGB-D image and description, RynnWorld-4D generates RGB, depth, and optical flow videos synchronously, which can be further lifted into 3D scene flow.

Dataset Curation: Rynn4DDataset 1.0

A major contribution is the curation of Rynn4DDataset 1.0, a large-scale multimodal dataset containing over 254.4 million annotated frames. This hybrid collection features both egocentric human and multi-robot manipulation video sequences, richly annotated with high-quality pseudo-labels for depth (via DA3) and optical flow (via DPFlow), as well as structured natural-language descriptions using advanced video-LLMs such as Qwen3-VL.

Figure 2: Composition of Rynn4DDataset 1.0: balance between human egocentric and robotic data promotes robust geometric and kinetic priors.

The semi-automatic curation pipeline systematically segments and annotates short video clips, providing dense, temporally aligned RGB-D, optical flow, and text instruction triplets.

Figure 3: Data curation pipeline: each video is annotated for captions, depth, and optical flow to produce high-quality, multi-modal training data.

Model Architecture: Tri-Branch 4D Diffusion Model

The core model extends scalable video diffusion (Wan-2.2) into a tri-branch transformer, assigning modality-specific architectures to RGB, depth, and optical flow. This decoupled backbone enables preservation of modality-specific distributions and facilitates cross-modal consistency via Joint Cross-Modal Attention (JA) modules equipped with 3D RoPE. This is critical for maintaining spatial correspondence and geometric alignment across modalities over the temporal sequence.

Figure 4: RynnWorld-4D pipeline: a tri-branch architecture synchronously predicts RGB, depth, and optical flow, with cross-modal aggregation for downstream policy control.

A phased training schedule with three stages (modality adaptation, joint attention, full-parameter SFT) is adopted to mitigate the risk of cross-modal interference and to promote gradual alignment across heterogeneous modalities. Branch dropout during fusion training encourages robustness by forcing JA modules to reconstruct dropped modalities from visible branches.

The overall architecture enables direct projection of the generated depth and flow into 3D point clouds and scene flow, which can be exploited for explicit metric reasoning.

Policy Integration: RynnWorld-4D-Policy

Leveraging the latent representations of RynnWorld-4D, a policy head (RynnWorld-4D-Policy) extracts temporally compressed, multi-modal features and maps them directly to closed-loop robotic actions. By bypassing per-step video denoising, this policy achieves high-frequency action output suitable for real-time manipulation, with effective operational rates up to $\sim$9 Hz on modern hardware.

The action generation module utilizes flow matching in action space, allowing efficient ODE-based parallel action chunking without necessitating per-step diffusion inference. The deployment platform includes high-DOF TIANJI M6 robots and dual WUJI HANDs.

Figure 5: Experimental hardware platform: TIANJI M6 arm and WUJI dexterous hand for complex manipulation tasks.

Experimental Evaluation

A comprehensive real-world manipulation benchmark encompassing six open-world bimanual and precision-oriented tasks is established. These tasks test dual-arm coordination, hand-over, heavy object lifting, and spatially constrained manipulations.

Figure 6: Real-world manipulation benchmark suite used for evaluating policy performance in diverse tasks.

Quantitatively, RynnWorld-4D achieves a $\delta_1$ geometric accuracy of 0.610—substantially superior to TesserAct or 4DNeX—and an AEPE of 0.170 for flow, outperforming all baselines in both spatial and temporal metrics. In end-to-end policy learning, RynnWorld-4D-Policy surpasses strong RGB-only and 2D foundation model baselines, especially in scenarios demanding fine spatial or kinetic reasoning (e.g., Lid Placement, Hand-over).

Qualitative results reveal high cross-modal and temporal consistency, with generated depth and flow maps aligning tightly with visible RGB transitions, even in complex manipulation scenarios with object occlusions and contact dynamics.

Figure 7: Qualitative synthesis: RynnWorld-4D preserves geometric structure and motion consistency across modalities.

Figure 8: Extended qualitative results: robust coherence across spatially and temporally diverse manipulation scenarios.

Ablation Studies

Extensive ablations confirm:

• Removing cross-modal interaction (independent branches) notably degrades geometric and motion accuracy.
• Omitting modality adaptation results in diminished depth accuracy and temporal coherence.
• Exclusion of large-scale 4D pretraining leads to catastrophic collapse in generalization; the diversity and scale of Rynn4DDataset 1.0 are essential for robust 4D representation learning.
• Architecture variants without 3D RoPE or using shared FFNs are empirically suboptimal, validating the modular, spatially-aware design.

Limitations and Future Directions

RynnWorld-4D imposes significant compute demands due to the diffusion-based 4D generation, resulting in a bottlenecked planning loop ($\sim$1.1 s for a full pass), which is mitigated, though not entirely resolved, by action chunking. The method is currently tailored for egocentric, single-robot viewpoints; generalization to multi-view/multi-agent systems and further acceleration of tri-branch diffusion inference remain open research challenges.

Conclusion

RynnWorld-4D introduces a physically-grounded, scalable, and multi-modal foundation for 4D world modeling, establishing a clear advancement in robotic manipulation by aligning generative capabilities with the intrinsic geometric and kinematic reasoning demands of complex tasks. This work demonstrates that tightly coupled, predictive RGB-DF representations, supported by large-scale 4D supervision, not only bridge the representational gap between perception and control but also offer tangible improvements in generalization and real-world policy precision. Future efforts could involve architectural optimizations for real-time inference, multi-agent generalization, and compositional scene reasoning, further enhancing the prospects for robust embodied AI.

---

Reference: "RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation" (2607.06559).