Rynn4DDataset 1.0: 4D Multimodal Video Dataset

• Rynn4DDataset 1.0 is a large-scale multimodal video dataset offering synchronized RGB, depth, optical flow, and language captions to model 4D scene dynamics.
• It uses a pseudo-labeling pipeline with advanced estimators to produce high-quality depth and optical flow annotations across over 254.4 million frames.
• The dataset underpins a tri-branch diffusion model that fuses appearance, spatial geometry, and temporal motion for enhanced robotic manipulation and action planning.

Rynn4DDataset 1.0 is a large-scale multimodal video dataset introduced alongside the RynnWorld-4D framework for 4D embodied world modeling in robotic manipulation. It is designed to provide synchronized supervision in RGB, monocular depth, and dense optical flow—denoted RGB-DF—together with language descriptions, with the stated goal of aligning visual appearance, spatial geometry, and temporal motion in a representation that is closer to the low-level end-effector actions required by robotic systems than pure 2D pixel observations (Zhao et al., 7 Jul 2026). The dataset contains over 254.4 million frames collected from a hybrid corpus of egocentric human activity and robotic manipulation videos, and its annotations are generated through a pseudo-labeling pipeline based on Qwen3-VL, Depth Anything 3, and DPFlow. Within the RynnWorld-4D study, the dataset serves as the supervisory substrate for training a tri-branch diffusion world model and, indirectly, for learning policy-relevant 4D latent representations.

1. Conceptual role and motivation

Rynn4DDataset 1.0 is motivated by the claim that synchronized RGB, depth, and optical flow form a physically grounded representation of scene dynamics. In the formulation used by the RynnWorld-4D work, RGB encodes appearance, depth encodes spatial geometry, and optical flow encodes temporal motion. The intended benefit of this multimodal combination is that it aligns perceptual state with underlying 3D structure and inter-frame dynamics, making it possible—under standard pinhole-camera assumptions—to back-project observations into 3D scene flow (Zhao et al., 7 Jul 2026).

The dataset is therefore not described merely as a repository of videos with auxiliary labels. Its function is to support a specific class of 4D embodied world models that jointly predict future appearance, geometry, and motion. In that framing, Rynn4DDataset 1.0 is the training corpus that supplies the RGB-DF supervision required for a unified diffusion process and for the downstream extraction of 4D internal representations used by RynnWorld-4D-Policy. A plausible implication is that the dataset is intended to reduce the representational gap between passive visual prediction and action-oriented robotic control, although the paper attributes the resulting control improvements to the combined model-and-dataset system rather than to the dataset in isolation.

2. Corpus composition and scale

The reported scale of Rynn4DDataset 1.0 is over 254.4 million frames. Its source material is a hybrid corpus that combines human egocentric activity videos with robotic manipulation videos. The paper emphasizes that the dataset “balances human egocentric videos with diverse robotic manipulation data,” but it does not specify exact ratios, hours of footage, counts of sequences, or counts per source dataset (Zhao et al., 7 Jul 2026).

The sources named in the paper are as follows.

Category	Source datasets	Notes stated in the paper
Human-centric	Epic-Kitchens; EgoVid-5M	Egocentric human activity data
Robotic manipulation	RoboMIND; RDT-1B; Galaxea; RoboCOIN; AgiBot	Robot manipulation video data

This composition is presented as distinctive because it combines scale, tri-modal pseudo-labels, and domain diversity. The human-centric portion is intended to contribute general interaction priors, while the robotic manipulation portion is intended to contribute robot-specific execution traces. The paper does not enumerate object categories or environment taxonomies, so any stronger claim about semantic coverage would exceed the reported evidence.

A common misunderstanding would be to treat Rynn4DDataset 1.0 as a conventional robot learning dataset with action and state supervision. The dataset description does not report robot action labels, proprioception, or other control signals within Rynn4DDataset 1.0 itself. Its primary supervisory content is multimodal perceptual rather than action-conditioned.

3. Modalities and annotation scheme

Rynn4DDataset 1.0 contains four principal annotation streams: RGB frames from the original sources, monocular depth maps generated per frame by Depth Anything 3, dense optical flow estimated per consecutive frame pair by DPFlow, and language instructions or captions generated with Qwen3-VL for each 5-second segment (Zhao et al., 7 Jul 2026).

The language stream is generated by sampling videos at 1 FPS and splitting them into 5-second segments. Qwen3-VL is prompted to describe subject/action, environment/background, objects/interactions, and overall scene context. Generation uses a maximum output length of 512 tokens and temperature 0.7, with outputs stored in JSON. The paper states segment-level storage, but it does not state frame-level language alignment beyond the 5-second segmentation, nor any finer temporal synchronization for captions.

The optical flow stream is estimated sequentially for each frame pair at native resolution with DPFlow and stored as color-encoded flow fields in MP4 videos at 25 FPS. No dataset-level thresholding or masking rules are specified for the stored flow. The depth stream is produced by Depth Anything 3 using checkpoint DA3NESTED-GIANT-LARGE-1.1. Depth is estimated at 30 FPS with a working resolution of 392 pixels on the short side, after which compressed depth arrays are loaded, bilinearly upsampled to original video resolution, clipped to the global depth range $[0.0, 5.0]$ meters, quantized to 8-bit grayscale according to

$$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$

and saved as RGB videos.

The paper characterizes the depth and optical flow labels as “high-quality pseudo-annotations” generated by state-of-the-art estimators. At the same time, it does not report quantitative label-quality metrics for DA3 or DPFlow on external benchmarks as part of dataset creation. Accordingly, the dataset should be understood as pseudo-labeled rather than manually annotated or sensor-ground-truthed.

Another possible misconception is that camera pose is a standard stored component of the dataset. The paper notes that DA3 also outputs camera pose estimation, but the dataset description does not specify storing poses or using them during training.

4. Curation pipeline and storage characteristics

The curation pipeline consists of three stated stages: data collection from the listed source datasets, preprocessing into short clips, and multimodal annotation. The multimodal annotation stage includes the Qwen3-VL captioning process, the DPFlow optical-flow estimation process, and the DA3 depth-estimation process. Beyond these steps, the paper does not describe additional cleaning, filtering criteria, deduplication procedures, or dataset-level quality-control thresholds (Zhao et al., 7 Jul 2026).

The storage characteristics are modality-specific. Caption generation is based on 5-second segments sampled at 1 FPS. Optical flow is stored as 25 FPS color-coded MP4 videos at native resolution. Depth estimation is performed at 30 FPS using the short-side 392-pixel working resolution and then resampled back to original resolution for storage as RGB-coded depth videos. Original video resolutions vary by source, but the paper does not provide their distribution.

The dataset description is notably incomplete with respect to several operational details that practitioners often expect. It does not specify canonical RGB-DF clip lengths beyond the 5-second caption segments; it does not describe file organization, naming conventions, directory layouts, or exact storage formats for raw depth arrays beyond the visualization videos; and it does not define official train, validation, or test splits. The held-out test set of 50 videos used later in model evaluation is explicitly an experiment-specific subset rather than a dataset-provided split.

This absence of declared splits and data-management conventions has practical significance. It suggests that reproducibility at the dataset-packaging level may depend on project-specific scripts or release artifacts rather than on a formally standardized dataset specification, although the paper itself does not elaborate on this point.

5. Formal 4D representation and geometric interpretation

Rynn4DDataset 1.0 is tied to a geometric interpretation in which synchronized depth and optical flow can be lifted into 3D scene dynamics. For a pixel $(u,v)$ with homogeneous image coordinate

$p_t = [u, v, 1]^\top,$

the paper defines geometric unprojection as

$P_t = D_t(u,v) \cdot K^{-1} p_t,$

where $K$ is the camera intrinsic matrix. This lifts image observations into a 3D point cloud $C_t = \{P_t^i\}$ (Zhao et al., 7 Jul 2026).

Temporal correspondence is then expressed through optical flow $f_{\mathrm{opt}} = [\Delta u, \Delta v]^\top$:

$$P_{t+1} = D_{t+1}(u+\Delta u, v+\Delta v) \cdot K^{-1} (p_t + [\Delta u, \Delta v, 0]^\top).$$

The resulting metric scene flow is defined as

$f_{3D} = P_{t+1} - P_t.$

The paper also mentions an edge-filtering refinement step in which pixels with large depth gradient $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$0 are masked, although $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$1 is not numerically specified. The geometric interpretation is central to why the dataset is described as “physically grounded”: the synchronized modalities are meant to permit explicit reasoning over 3D structure and motion rather than only appearance trajectories.

This suggests that Rynn4DDataset 1.0 is best viewed as a dataset for 4D perceptual supervision rather than merely multimodal video understanding. Its annotations are organized so that future motion can be interpreted as scene flow, even though the stored supervision itself is depth and optical flow rather than directly provided 3D point trajectories.

6. Use in training and empirical role in downstream performance

Within the RynnWorld-4D study, Rynn4DDataset 1.0 is used to train the world model through a staged curriculum. In Stage 1, the three branches for RGB, depth, and flow are trained independently with Joint Cross-Modal Attention disabled. In Stage 2, Joint Cross-Modal Attention modules are inserted every 3 layers, for 10 total, with shared $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$2; the backbone is frozen; frame-wise joint attention uses 3D RoPE; and branch dropout $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$3 is applied on $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$4. In Stage 3, the entire model is unfrozen and training continues on the full dataset (Zhao et al., 7 Jul 2026).

The shared flow-matching objective across modalities $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$5 is given by

$$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$6

and

$$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$7

with shared $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$8 across modalities to synchronize denoising trajectories, $$I = \left\lfloor \frac{d}{d_{\max}} \times 255 \right\rfloor,$$9 in Stage 1 and $(u,v)$0 thereafter, and $(u,v)$1.

The paper’s evaluation protocol uses a held-out test set of 50 sequences sampled from RoboMIND, RDT-1B, and Galaxea. Depth is evaluated using Absolute Relative Error and threshold accuracy $(u,v)$2 with median scaling; optical flow is evaluated using Average End-Point Error; and RGB is evaluated with IQ, MS, SC, Subj., SSIM, PSNR, and LPIPS. For AEPE, the implementation detail is unusual: because both predicted and reference flow are stored as Middlebury color-wheel visualizations, AEPE is computed as per-pixel $(u,v)$3 distance in normalized RGB color space between the two encoded flow maps.

The dataset’s empirical role is highlighted through ablation. The paper states that removing large-scale 4D pre-training on Rynn4DDataset 1.0 causes AEPE to surge from 0.170 to 0.729 and broadly degrades performance. It further states that cross-modal training on RGB-DF improves geometric and motion fidelity relative to independent branches, and that 3D RoPE in joint attention is crucial for pixel-wise spatial alignment across modalities. These results are presented as evidence that the dataset is critical for learning consistent spatio-temporal geometry and motion priors.

The downstream link to policy learning is indirect but central. RynnWorld-4D-Policy uses the frozen world model to extract 4D latent features and generates actions through a lightweight flow matching head in 4 ODE steps, enabling closed-loop control at approximately 9 Hz. The paper attributes improved manipulation performance, especially in tasks requiring spatial precision and temporal coordination, to this predictive 4D latent structure. Strictly speaking, these are system-level outcomes rather than dataset-only benchmarks, but they define the practical role that Rynn4DDataset 1.0 plays in the overall framework.

7. Limitations, omissions, and access conditions

The paper leaves several important aspects of Rynn4DDataset 1.0 unspecified. It does not report the number of videos or sequences, hours of footage, exact distribution ratios between human egocentric and robot manipulation data, object category inventories, dataset-level quality-control thresholds beyond depth clipping and quantization, or deduplication procedures. It also does not specify official train/validation/test splits, a dataset-specific download link, licensing terms, or usage restrictions (Zhao et al., 7 Jul 2026).

These omissions matter for interpretation. For example, the absence of official splits means that comparisons across future studies may depend on ad hoc partitioning. The lack of explicit licensing and access conditions means that the practical reusability of the dataset cannot be determined from the paper alone. Similarly, because ethical considerations and dataset biases are not discussed, no paper-backed claim can be made about representational balance, demographic coverage, or bias mitigation.

The paper does provide project URLs associated with RynnWorld-4D:

• https://alibaba-damo-academy.github.io/RynnWorld-4D.github.io
• https://github.com/alibaba-damo-academy/RynnWorld-4D
• https://huggingface.co/Alibaba-DAMO-Academy/RynnWorld-4D
• https://www.modelscope.cn/models/DAMO_Academy/RynnWorld-4D

However, it does not explicitly identify any of these as a download endpoint for Rynn4DDataset 1.0 itself. A cautious reading is therefore that the dataset is defined technically and used experimentally in the paper, but its packaging and release conditions are not fully specified in the published description.

In summary, Rynn4DDataset 1.0 occupies a specific niche: a very large RGB-DF-plus-language corpus intended for 4D embodied world modeling rather than a fully standardized robotics benchmark. Its significance in the literature of RynnWorld-4D lies in the way it operationalizes physically grounded multimodal supervision at scale, while its present limitations are chiefly those of specification, transparency, and release detail rather than of stated modeling ambition.