RynnWorld-4D-Policy: Inverse-Dynamics Controller

• RynnWorld-4D-Policy is an inverse-dynamics controller that extracts predictive 4D features by fusing RGB, depth, and optical flow to enable precise robot manipulation.
• It employs a Flow Former module to compress high-dimensional spatiotemporal data, followed by a flow matching inverse dynamics head for fast action decoding.
• Empirical evaluations showcase state-of-the-art spatial precision and temporal coordination across multiple dexterous tasks, outperforming pure RGB-based methods.

Searching arXiv for the specified paper and closely related work. RynnWorld-4D-Policy is an inverse-dynamics controller introduced as the control-side companion to the 4D embodied world model RynnWorld-4D. It is designed for robotic manipulation settings in which control depends not only on visual appearance but also on explicit 3D structure and temporal motion. The policy operates on synchronized RGB, depth, and optical flow representations learned by the frozen world model, treating the model’s internal predictive latents as a compact, geometrically grounded state for action selection rather than using the world model only for visualization or planning (Zhao et al., 7 Jul 2026). In this formulation, policy inference is separated from expensive multi-step video denoising: a single forward pass through the frozen backbone yields predictive 4D features, which are then compressed and decoded into robot action chunks in closed loop (Zhao et al., 7 Jul 2026).

1. Conceptual role within the RynnWorld-4D framework

The problem setting is robotic manipulation in the open world, where a robot must react to both appearance and 3D dynamics. The underlying paper argues that pure RGB video prediction is limited because pixels are fundamentally 2D and do not explicitly encode geometry or motion in metric space. This makes precise object pose estimation, depth-aware interaction, and temporal coordination difficult for dexterous manipulation (Zhao et al., 7 Jul 2026).

RynnWorld-4D addresses this limitation by predicting future synchronized RGB, depth, and optical flow videos from a single RGB-D observation and a language instruction. RynnWorld-4D-Policy exists because a world model alone does not directly control a robot. The control objective is not to generate videos for their own sake, but to produce actions fast enough for real-world manipulation. Accordingly, the policy is introduced to bridge the gap between predictive 4D representation and executable robot control (Zhao et al., 7 Jul 2026).

The base distinction is explicit. RynnWorld-4D is a generative tri-branch diffusion transformer that predicts future RGB, depth, and flow sequences through a denoising process. RynnWorld-4D-Policy is not a new world model; it is a lightweight inverse-dynamics or action head built on top of the frozen world model. Its function is to consume internal features from RynnWorld-4D, compress them, and output actions. This suggests that the main design hypothesis is representational: the internal 4D latents are treated as closer to the robot’s action space than raw 2D pixels because they jointly encode appearance, geometry, and motion (Zhao et al., 7 Jul 2026).

2. Predictive 4D representation as the policy state

At inference time, the policy uses RynnWorld-4D as a frozen 4D vision encoder. It performs a single forward pass through the world model and extracts intermediate hidden states across all three branches: RGB, depth, and flow. These hidden states are concatenated along the channel dimension to form

$$F_p \in \mathbb{R}^{B \times T \times 3C \times H \times W},$$

where $C$ is the per-branch latent channel dimension (Zhao et al., 7 Jul 2026).

The fused tensor $F_p$ is the policy’s predictive 4D representation. Its semantics are explicitly partitioned: RGB features provide appearance and object context, depth features provide explicit geometry, and flow features provide explicit motion and kinetic cues. In the implementation details, the policy extracts features at diffusion timestep $t=500$ from block 15 of the transformer, using the frozen RynnWorld-4D at $480 \times 640$ resolution, producing $T=21$ latent frames after temporal compression (Zhao et al., 7 Jul 2026).

The geometric motivation for this representation is grounded in the paper’s definition of 4D structure. Depth is unprojected into 3D points by

$$\mathbf{P}_t = D_t(u, v) \cdot \mathbf{K}^{-1} \mathbf{p}_t,$$

while optical flow and depth are back-projected to obtain the next-frame 3D point

$$\mathbf{P}_{t+1} = D_{t+1}(u+\Delta u, v+\Delta v) \cdot \mathbf{K}^{-1} (\mathbf{p}_t + [\Delta u, \Delta v, 0]^\top),$$

yielding 3D scene flow

$\mathbf{f}_{3D} = \mathbf{P}_{t+1} - \mathbf{P}_t.$

These definitions are not policy equations in the narrow sense, but they explain why the internal representation is presented as action-friendly: it contains explicit geometric and motion cues relevant to manipulation (Zhao et al., 7 Jul 2026).

A plausible implication is that the policy is not merely consuming a rich visual embedding; it is consuming a latent representation whose inductive bias is tied to scene kinematics and contact-relevant geometry.

3. Architecture: Flow Former and flow-matching action head

The policy architecture consists of two principal components beyond the frozen backbone: a compression module called Flow Former and an inverse-dynamics action decoder implemented as a flow matching policy head (Zhao et al., 7 Jul 2026).

Flow Former

Because $F_p$ is high-dimensional, the policy compresses it with Flow Former. The module uses learnable queries and processes the 4D features in two stages: frame-wise spatial cross-attention followed by temporal self-attention. The formulation is

$C$0

for $C$1 (Zhao et al., 7 Jul 2026).

Operationally, each frame’s latent features are first attended to spatially by learnable queries; the resulting per-frame summaries are then fused over time. The output $C$2 is a compact spatiotemporal token set used for action decoding (Zhao et al., 7 Jul 2026).

Inverse dynamics head

The action decoder is a flow matching policy head operating in action space $C$3, conditioned on predictive 4D tokens $C$4, text embedding $C$5, and proprioception $C$6 (Zhao et al., 7 Jul 2026). The paper states that the policy head follows the objective defined for the world model in action space rather than introducing a wholly separate generative formalism.

The world-model preliminaries define rectified flow interpolation as

$C$7

with conditional flow-matching loss

$C$8

The main world-model loss is

$C$9

For the policy, the paper states that the same flow-matching principle is reused in action space: the head learns a velocity field over actions and samples actions by integrating that field (Zhao et al., 7 Jul 2026).

During inference, action generation is performed via an ODE solver with 4 Euler steps. The paper does not provide a separate closed-form action equation beyond that description (Zhao et al., 7 Jul 2026).

4. From latent prediction to closed-loop robot action

RynnWorld-4D-Policy is specifically designed to bypass expensive multi-step denoising as the source of action inference. The backbone is run once, internal hidden features are extracted, and a lightweight Flow Former plus action head produces actions. The appendix emphasizes that the 4D visual features are extracted from the frozen world model in one forward pass, $F_p$0, while only the action head performs 4-step ODE sampling (Zhao et al., 7 Jul 2026).

The mapping from latent representation to control output is described as a sequence of stages:

1. RGB-D observation and instruction.
2. Frozen RynnWorld-4D forward pass.
3. Extraction of internal RGB, depth, and flow hidden states.
4. Concatenation into $F_p$1.
5. Flow Former compression into $F_p$2.
6. Flow matching head producing a 10-step action chunk.
7. Robot execution of the chunk in closed loop (Zhao et al., 7 Jul 2026).

Each action is 54-dimensional, and the policy predicts a chunk of 10 actions at a time. During deployment, the robot executes these actions open-loop before the next vision refresh (Zhao et al., 7 Jul 2026). This arrangement underlies the system’s timing profile. A full forward cycle takes about 1.1 s; the world model refreshes at about 0.9 Hz; because each planning cycle outputs a chunk of 10 actions, the effective control frequency is about 9 Hz. The appendix further notes that actions are executed sequentially while the next planning cycle runs in parallel (Zhao et al., 7 Jul 2026).

The robot-system interface includes lower-level timing details. The low-level interface operates at 500 Hz, commands are sent with a delay of 18–30 ms, and the policy itself is said to infer at 50 Hz in the robot setup description, although the higher-level world-model refresh remains approximately 9 Hz effective control after chunking (Zhao et al., 7 Jul 2026). This suggests a two-timescale control hierarchy: a slower predictive planning loop wrapped around a faster execution interface.

5. Training regime and implementation specifics

The policy is trained separately from the world model. RynnWorld-4D is frozen during policy learning, and only the Flow Former and the flow matching policy head are optimized (Zhao et al., 7 Jul 2026). Supervision is direct imitation-style supervision from real robot demonstrations; the paper does not describe a separate reward signal or reinforcement-learning objective. The action head is trained on recorded action sequences, that is, behavior cloning or imitation learning with flow matching as the generative parameterization (Zhao et al., 7 Jul 2026).

Policy learning uses demonstrations from six real-world robot manipulation tasks: Dual Picking, Block Pushing, Hand-over, Bimanual Lifting, Lid Placement, and Bowl Stacking. The paper states that training uses a fixed budget of 200 episodes per task, totaling 1,200 episodes (Zhao et al., 7 Jul 2026).

The optimization configuration is specified as follows.

Component	Setting
Optimizer	AdamW
Learning rate	$F_p$3
$F_p$4	$F_p$5
Weight decay	$F_p$6
Schedule	2% linear warmup, 8% constant hold, 90% cosine decay to $F_p$7 of peak
Batch size	1 per GPU
Epochs	100
Precision	Mixed precision

Several implementation details are also fixed in the reported system: frozen RynnWorld-4D backbone; resolution $F_p$8; $F_p$9 latent frames after 4x temporal compression; features extracted at diffusion timestep $t=500$0; intermediate state from transformer block 15; concatenated branch features of width 3072 per branch; Flow Former compression; action chunk length 10; action dimension 54; and 4-step Euler ODE sampling for action generation (Zhao et al., 7 Jul 2026).

The latency profile is dominated by the world-model backbone. In the reported timing breakdown, RynnWorld-4D accounts for 89.5% of the total forward latency, while the Flow Former and action head add only a few milliseconds (Zhao et al., 7 Jul 2026). This is central to the policy’s characterization as lightweight: the computational cost lies in the encoder that produces predictive 4D latents, not in the action decoder.

6. Empirical evaluation on real-world dexterous manipulation

The policy is evaluated on six real-world dexterous manipulation tasks: Dual Picking, Block Pushing, Hand-over, Bimanual Lifting, Lid Placement, and Bowl Stacking. These tasks are described as testing dual-arm coordination, temporal sequencing, contact-rich interaction, precision placement, and long-horizon manipulation (Zhao et al., 7 Jul 2026).

The primary metric is success rate, measured over 35 consecutive real-world trials per task. A trial is successful if the task is completed within 120 seconds (Zhao et al., 7 Jul 2026). Baselines are Diffusion Policy (DP), $t=500$1, and $t=500$2 (Zhao et al., 7 Jul 2026).

The reported success rates for RynnWorld-4D-Policy are as follows.

Task	Success rate
Dual Picking	94.29%
Block Pushing	97.14%
Hand-over	28.57%
Bimanual Lifting	97.14%
Lid Placement	65.71%
Bowl Stacking	65.71%

The paper states that the policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks and particularly excels in tasks demanding spatial precision and temporal coordination (Zhao et al., 7 Jul 2026). It specifically highlights Lid Placement and Bowl Stacking, where the policy reaches 65.71%, beating the next best baseline by 8.57% (Zhao et al., 7 Jul 2026).

The Hand-over task is identified as especially difficult. The paper notes that other foundation models struggle there because of bias toward parallel-jaw grippers in pretraining and difficulty reasoning about 3D distance and self-occlusion for dexterous hands (Zhao et al., 7 Jul 2026). This contextualizes the comparatively lower absolute score of 28.57% while also framing it as a challenging benchmark rather than an isolated failure mode.

A plausible implication is that the policy’s advantages are strongest when manipulation depends on metric geometry, object-relative alignment, and short-horizon anticipation rather than purely reactive image-to-action mapping.

7. Ablations, interpretation, and relation to adjacent systems

The policy ablation on input modalities shows a structured pattern. The variants reported are: w/o RynnWorld-4D, RGB only, RGB + Depth, RGB + Optical Flow, and full RGB + Depth + Optical Flow. The paper states that removing RynnWorld-4D causes a noticeable drop across tasks; RGB only is worse than the full model; RGB + Depth yields strong gains on spatial precision tasks; RGB + Optical Flow helps motion-sensitive tasks; and the full RGB + Depth + Optical Flow representation is best overall (Zhao et al., 7 Jul 2026).

This ablation supports the paper’s central claim that the combined 4D representation improves both geometry-sensitive and motion-sensitive control. The most policy-relevant representation ablation replaces RynnWorld-4D with a standard ResNet-18 image encoder and causes substantial performance drops, especially on manipulation tasks requiring anticipation (Zhao et al., 7 Jul 2026). The paper presents this as the strongest evidence that the controller benefits from internal 4D world-model latents rather than from a static visual embedding.

Additional ablations are performed primarily on the world model, but they bear directly on policy design because policy quality depends on latent quality. The paper reports that independent branches hurt depth and flow quality, no modality adaptation reduces geometric accuracy and motion precision, no 4D pretraining causes a major collapse especially in motion, removing 3D RoPE from joint attention hurts geometry and flow alignment, and using a shared FFN degrades all modalities, indicating that modality-specific specialization matters (Zhao et al., 7 Jul 2026).

These observations matter because RynnWorld-4D-Policy does not learn its own perceptual state from scratch. Its effectiveness is contingent on the representational structure induced by the tri-branch architecture, cross-modal fusion, modality adaptation, 4D pretraining, and frame-wise 3D RoPE in the frozen backbone (Zhao et al., 7 Jul 2026). In that sense, the policy can be understood as an exploitation layer over a pretrained predictive latent space rather than a standalone visuomotor learner.

A related system, RynnWorld-Teleop, is described as a digital teleoperation data engine that generates action-aligned robot trajectories from hand-pose streams and a single reference image (Zhao et al., 7 Jul 2026). The available description states that RynnWorld-Teleop generates synthetic but action-aligned robot trajectories that could become training data for a policy such as RynnWorld-4D-Policy or other imitation learners (Zhao et al., 7 Jul 2026). This suggests a broader research program in which world models serve two distinct but compatible roles: as control-state encoders for closed-loop policy inference and as generative engines for scalable trajectory collection.