RynnWorld-4D: 4D Robotic World Model

• RynnWorld-4D is a language-conditioned 4D embodied model that predicts synchronized future RGB frames, depth maps, and optical flow using a unified diffusion process.
• It fuses appearance, metric geometry, and temporal motion into an RGB-DF representation, bridging the gap between 2D video prediction and 3D robotic control.
• A tri-branch transformer with joint cross-modal attention and staged training on the extensive Rynn4DDataset 1.0 drives robust action generation for complex robotic manipulation tasks.

RynnWorld-4D is a language-conditioned 4D embodied world model for robotic manipulation that predicts future RGB frames, depth maps, and optical flow from a single RGB-D observation within one unified diffusion process. Its central premise is that synchronized RGB, depth, and optical flow—denoted RGB-DF—form a physically grounded representation of scene dynamics that is materially closer to the low-level end-effector actions required by robots than RGB-only video, because appearance, metric structure, and temporal motion are modeled together rather than inferred indirectly from 2D pixels alone (Zhao et al., 7 Jul 2026).

1. Representational premise and problem formulation

RynnWorld-4D is framed against a specific limitation of earlier robotic world models: many are effectively 2D video predictors, whereas robotic manipulation depends on 3D structure and 3D motion. The method therefore recasts embodied prediction as 4D scene evolution. In the paper’s formulation, RGB provides appearance and semantics, depth supplies metric geometry, and optical flow provides dense temporal displacement; depth and flow can then be combined into a 3D scene-flow interpretation. This is the paper’s main argument for why RGB-DF narrows the gap between world prediction and policy learning (Zhao et al., 7 Jul 2026).

The geometric interpretation is explicit. Given depth map $D_t$, pixel $\mathbf{p}_t = [u,v,1]^\top$, and camera intrinsics $\mathbf{K}$, the corresponding 3D point is

$$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$

With optical flow $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$, the tracked 3D point at the next frame is

$$\mathbf{P}_{t+1} = D_{t+1}(u+\Delta u, v+\Delta v) \cdot \mathbf{K}^{-1}\bigl(\mathbf{p}_t + [\Delta u,\Delta v,0]^\top\bigr),$$

and the resulting metric scene flow is

$$\mathbf{f}_{3D} = \mathbf{P}_{t+1} - \mathbf{P}_t .$$

This formulation makes the model’s “4D” designation concrete: it is not 4D in the sense of explicit volumetric world reconstruction, but in the sense of jointly representing appearance, geometry, and motion over time in a form directly compatible with robotic interaction (Zhao et al., 7 Jul 2026).

The conditioning interface is deliberately minimal for an embodied predictor of this scale. RynnWorld-4D takes a single RGB-D image and a language instruction, then generates synchronized future RGB, depth, and flow sequences. The paper treats this as a projective 4D representation: it remains anchored to image-space priors and large-scale video pretraining, but carries sufficient geometric and temporal structure to support downstream control (Zhao et al., 7 Jul 2026).

2. Tri-branch architecture and unified flow matching

Architecturally, RynnWorld-4D extends the pretrained Wan 2.2-TI2V-5B video diffusion transformer into a tri-branch transformer with one branch each for RGB, depth, and optical flow. Each modality maintains its own latent stream

$$\mathbf{z}_t^m \in \mathbb{R}^{T \times C \times H \times W}, \qquad m \in \{\text{rgb},\text{depth},\text{flow}\},$$

so the model is not a simple channel-concatenation design. The backbone retains 30 DiT layers with hidden dimension $3072$ and FFN dimension $14336$, while text cross-attention key/value projections are shared across branches because language is treated as a modality-agnostic semantic condition (Zhao et al., 7 Jul 2026).

Cross-modal fusion is performed by Joint Cross-Modal Attention modules inserted every 3 layers, at layers $\mathbf{p}_t = [u,v,1]^\top$0, for a total of 10 modules. Before attention, each branch receives a learnable modality embedding $\mathbf{p}_t = [u,v,1]^\top$1, initialized to zero, followed by branch-specific normalization: $\mathbf{p}_t = [u,v,1]^\top$2 Queries, keys, and values are then formed as

$\mathbf{p}_t = [u,v,1]^\top$3

$\mathbf{p}_t = [u,v,1]^\top$4

and each branch attends only to the other two modalities through

$\mathbf{p}_t = [u,v,1]^\top$5

$\mathbf{p}_t = [u,v,1]^\top$6

The cross-modal output is injected by a gated residual path,

$\mathbf{p}_t = [u,v,1]^\top$7

with zero-initialized output projections and gates initialized to $\mathbf{p}_t = [u,v,1]^\top$8. The paper emphasizes this design because “double zero-initialization” led to a saddle-point deadlock, whereas the adopted scheme preserves a smooth warm start from the pretrained checkpoint while retaining nonzero gradient flow (Zhao et al., 7 Jul 2026).

The model is trained with rectified-flow-style conditional flow matching rather than a discrete DDPM schedule. For each modality,

$\mathbf{p}_t = [u,v,1]^\top$9

and the full objective is

$\mathbf{K}$0

where $\mathbf{K}$1. A notable training choice is shared Gaussian noise across branches,

$\mathbf{K}$2

which is intended to keep denoising trajectories aligned across modalities. The first RGB and depth frames are clamped to the conditioning observation, the first flow frame is zero, and only future frames are supervised (Zhao et al., 7 Jul 2026).

3. Rynn4DDataset 1.0 and staged optimization

To train the model at scale, the paper introduces Rynn4DDataset 1.0, comprising over 254.4 million frames from egocentric human and robotic manipulation videos. The human sources are Epic-Kitchens and EgoVid. The robotic sources are RoboMIND, RDT-1B, Galaxea, RoboCoin, and AgiBot. The dataset is intended to combine broad interaction priors from human egocentric footage with robot-specific execution traces (Zhao et al., 7 Jul 2026).

Each clip is enriched with language, depth, and optical flow pseudo-labels. Language descriptions are generated with Qwen3-VL by sampling video at 1 FPS, splitting it into 5-second segments, and prompting for the main subject or action, environment or background, objects or interactions, and overall context or atmosphere, with maximum output length 512 tokens and temperature 0.7. Optical flow pseudo-labels are produced with DPFlow and stored as color-encoded MP4 videos at 25 FPS. Depth pseudo-labels are produced with Depth Anything 3 using checkpoint DA3NESTED-GIANT-LARGE-1.1; videos are processed at 30 FPS with short-side resolution 392 px, then upsampled to original resolution by bilinear interpolation and clipped to $\mathbf{K}$3 meters before 8-bit quantization (Zhao et al., 7 Jul 2026).

Training proceeds in three phases. Stage 1, modality adaptation, disables joint attention and trains branches independently so that inherited RGB priors can adapt to depth and flow distributions; it uses learning rate $\mathbf{K}$4, 500-step warmup, and $\mathbf{K}$5. Stage 2 freezes the backbone, inserts the 10 joint-attention modules, and trains only the cross-modal projections, RMSNorms, per-modality LayerNorms, tanh gates, and modality embeddings, using learning rate $\mathbf{K}$6, 200-step warmup, and Branch Dropout $\mathbf{K}$7. Stage 3 unfreezes the entire model for full-parameter joint SFT at learning rate $\mathbf{K}$8, 500-step warmup, and Branch Dropout $\mathbf{K}$9 (Zhao et al., 7 Jul 2026).

Branch Dropout is a specific regularization mechanism: during Stages 2 and 3, the model randomly selects either depth or flow and replaces its future noisy latent frames with pure Gaussian noise, while RGB is never dropped. This forces the joint-attention mechanism to reconstruct missing geometry or motion from the remaining modalities. The paper’s ablations indicate that this training curriculum is not incidental. Skipping modality adaptation or large-scale 4D pretraining causes substantial collapse in depth and flow quality, which suggests that the method depends on both scale and staged specialization rather than only on backbone capacity (Zhao et al., 7 Jul 2026).

4. RynnWorld-4D-Policy and closed-loop action generation

RynnWorld-4D-Policy turns the generative world model into a control system by treating it as a frozen predictive 4D encoder. Given current RGB-D observation and instruction, the policy runs a single forward pass through the world model, extracts intermediate hidden states from all three branches, and concatenates them into

$$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$0

In implementation, the extracted features come from block 15 at diffusion timestep $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$1, with per-branch feature size 3072 channels. The central design goal is to avoid online multi-step visual denoising during control (Zhao et al., 7 Jul 2026).

The high-dimensional predictive tensor is compressed by a Flow Former. Using learnable queries $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$2, it first performs frame-wise spatial cross-attention into each frame’s spatial map, then temporal self-attention across the resulting query states: $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$3 The resulting $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$4 is a compact predictive dynamics representation used by a flow-matching inverse-dynamics head conditioned on $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$5, a text embedding, and proprioception. The paper states that action generation follows the same flow-matching setup in action space and uses an ODE solver with $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$6 steps at inference (Zhao et al., 7 Jul 2026).

The action space is 54-dimensional, matching two 7-DoF arms plus two 20-DoF dexterous hands, and the policy predicts action chunks of length 10. One full inference cycle takes about 1.106 s, giving a planning refresh rate of about 0.9 Hz and an effective control frequency of about 9 Hz because chunked actions are executed while the next plan is computed. The appendix further notes a 50 Hz cached policy interface and a 500 Hz low-level robot interface. This architecture is significant because it directly links the internal 4D predictive state of the world model to action generation, bypassing the expense of video rollout at test time (Zhao et al., 7 Jul 2026).

5. Evaluation, ablations, and real-world manipulation performance

World-model evaluation uses a held-out test set of 50 video sequences sampled from RoboMIND, RDT-1B, and Galaxea. RGB quality is measured by IQ, Motion Smoothness, Subject Consistency, I2V-Subject, SSIM, PSNR, and LPIPS; geometry is measured by AbsRel and $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$7; motion is measured by AEPE. RynnWorld-4D reports IQ 0.635, MS 0.995, SC 0.957, Subject 0.992, SSIM 0.754, PSNR 17.85, LPIPS 0.269, AbsRel 0.310, $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$8 0.610, and AEPE 0.170. The paper positions these results as competitive with RGB video baselines and stronger than 4D baselines such as Free4D, TesserAct, and 4DNeX on geometry and motion fidelity, while also being the only listed model to report explicit flow with AEPE 0.170 (Zhao et al., 7 Jul 2026).

The real-world policy evaluation uses a TIANJI M6 robot with WUJI HAND and a RealSense D435i first-person camera. Success is measured over 35 consecutive real-world trials per task with a 120-second timeout. The six tasks are Dual Picking, Block Pushing, Hand-over, Bimanual Lifting, Lid Placement, and Bowl Stacking. RynnWorld-4D-Policy achieves success rates of 94.29, 97.14, 28.57, 97.14, 65.71, and 65.71, respectively. The corresponding Diffusion Policy results are 77.14, 85.71, 17.14, 88.57, 57.14, and 57.14; $$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t .$$9 gives 88.57, 94.29, 2.86, 91.43, 34.29, and 51.43; and $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$0 gives 94.29, 100.00, 0.00, 94.29, 37.14, and 42.86. The paper emphasizes gains on Hand-over, Lid Placement, and Bowl Stacking, where spatial precision and temporal coordination are most demanding (Zhao et al., 7 Jul 2026).

Ablations isolate the technical choices. Removing joint fusion and using Independent Branches worsens AbsRel from 0.310 to 0.737, $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$1 from 0.610 to 0.245, and AEPE from 0.170 to 0.247. Removing modality adaptation yields $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$2 0.479 and AEPE 0.231. Training without large-scale 4D pretraining degrades to AbsRel 0.797, $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$3 0.263, and AEPE 0.729. Removing RoPE inside joint attention worsens AbsRel to 0.420, $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$4 to 0.450, and AEPE to 0.210. Replacing branch-specific FFNs with a shared FFN degrades to AbsRel 0.580, $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$5 0.380, and AEPE 0.280. On the policy side, replacing the predictive 4D encoder with a ResNet-18 image encoder lowers Dual Picking from 94.29 to 71.43, Hand-over from 28.57 to 11.43, and Lid Placement from 65.71 to 51.43. Modality-combination ablations further show that RGB+Depth tends to help spatial-precision tasks, RGB+Optical Flow helps motion-sensitive tasks, and full RGB-DF performs best overall (Zhao et al., 7 Jul 2026).

6. Position within the Rynn family, adjacent 4D research, and limitations

Within the broader Rynn line, RynnWorld-4D can be read as a move from video-centric world modeling toward explicitly embodied RGB-DF prediction. RynnWorld-Teleop introduced digital teleoperation through an action-conditioned robot-centric latent video model that synthesized egocentric execution video from a reference image and a hand-pose stream, while RynnBrain defined a spatiotemporal embodied foundation model centered on egocentric understanding, spatiotemporal localization, grounded reasoning, and planning (Zhao et al., 7 Jul 2026, Dang et al., 13 Feb 2026). This suggests a broader program in which prediction, embodiment, and structured spatiotemporal reasoning are progressively integrated, although only RynnWorld-4D explicitly formalizes RGB, depth, and flow as a joint manipulation-oriented world representation (Zhao et al., 7 Jul 2026).

In the wider 4D literature, adjacent emphases differ. LLaVA-4D targets coordinate-grounded language understanding in dynamic scenes rather than control (Zhou et al., 18 May 2025). Full-4D focuses on full-scene 4D generation from single-view video through synchronized $\mathbf{f}_{opt} = [\Delta u,\Delta v]^\top$6 multi-view synthesis and 4DGS lifting (Chen et al., 25 May 2026). WorldReel generates RGB together with pointmaps, cameras, scene flow, and dynamic masks for 4D-consistent video generation (Fang et al., 8 Dec 2025). ST-Gen4D inserts an explicit graph-based spatiotemporal cognition state between perception and 4D generation (Wang et al., 8 May 2026). SA4D addresses temporally consistent object segmentation in 4D Gaussian worlds (Ji et al., 2024). Against that backdrop, RynnWorld-4D is distinctive in centering robotic manipulation and in coupling a generative RGB-DF world model to an inverse-dynamics policy head (Zhao et al., 7 Jul 2026).

The paper states several limitations directly. Inference remains expensive: effective control runs at about 9 Hz on an RTX 5090. The model is primarily optimized for egocentric perspectives. Extending the framework to multi-view consistency and collaborative multi-robot settings remains open. A plausible implication is that the reliance on pseudo-labeled depth and flow also couples performance to the quality of monocular depth and optical-flow estimators, although the paper does not provide a separate failure taxonomy for that dependency. More broadly, RynnWorld-4D is not an explicit 4D geometric simulator in the sense of multi-view-consistent dynamic scene reconstruction; it is a unified RGB-DF predictor whose value lies in making appearance, metric geometry, and motion jointly available to a downstream manipulation policy (Zhao et al., 7 Jul 2026).