Robo4D-200k: 4D Robotics Dataset

• Robo4D-200k is a large-scale dataset comprising over 200k robot–world interaction episodes with detailed 4D spatiotemporal annotations.
• It integrates both real-world and synthetic demonstrations to enable research in embodied AI and sim-to-real policy transfer with rigorous quantitative benchmarks.
• The dataset provides granular RGB frames, 4D pointmaps, and precise robot kinematics to support evaluations of visual fidelity and geometric consistency.

Robo4D-200k is a large-scale, high-fidelity dataset designed to support research in 4D generative robotics, embodied AI, and sim-to-real policy transfer. Comprising over 200,000 episodes of diverse robot–world interactions, each with granular spatiotemporal annotations, Robo4D-200k facilitates the study and modeling of physically-plausible, geometry-consistent, and embodiment-agnostic multistep interactions between manipulator robots and complex real or synthetic environments. It underpins the Kinema4D action-conditioned 4D generative robotic simulator and is designed to restore the full 4D spatiotemporal nature of robot-world interactions absent from prior 2D- or static-cue-based simulators (Xu et al., 17 Mar 2026).

1. Dataset Scale and Composition

Robo4D-200k contains 201,426 robot–world interaction episodes. Each episode is uniformly 49 frames long, representing approximately two seconds at 24 frames per second, resulting in roughly 9.89 million RGB frames and ∼138 hours of total footage at standard playback rates. Data is sourced from both in-the-wild real-world robot demonstrations (DROID, Bridge, RT-1 datasets) and procedurally synthesized scenes generated within the LIBERO physics engine. This results in a balanced dataset mixing unstructured, real-world robotics with simulated scenarios, including failure cases difficult to capture in real environments.

Source/Domain	Proportion	Content Characteristics
Real (DROID+Bridge+RT-1)	~60%	Natural demonstrations, diverse environments
Synthetic (LIBERO)	~40%	Procedural success/failure, injected noise

Episodes are diverse, including pick-and-place cycles for rigid and articulated objects (e.g., blocks, plants, cups), successful and near-miss executions, cluttered workspaces, and both real-world and simulation-based interactions.

2. Robot Platforms and Interaction Scenarios

The dataset features a spectrum of robotic manipulator platforms:

• Multiple 6-DoF arms, including URDF-based robots reconstructed via ReconViaGen (e.g., UR5-like arms, YAM-Arm).
• Robots benchmarked in public datasets, notably those from the RT-1 suite.

Interaction scenarios span:

• Pick-and-place motions with rigid and articulated objects.
• Tasks exhibiting both successes and "near-miss" failures, such as gripper misalignment or unreachable targets.
• Complex workspace geometries with distractor objects, constrained passages, and occlusions.
• Both real-world demonstrations and LIBERO-simulated actions, which systematically incorporate failure modes by injecting Gaussian noise into poses (σ∈{0.5,0.8,1.2}).

More than 30 distinct object categories are present, including rigid blocks, deformable cloths, articulated plants, and transparent cups, increasing the ecological validity and diversity of contact and manipulation events.

3. Annotation Schema and Data Modalities

Each Robo4D-200k episode is annotated with a multimodal, tightly synchronized suite of signals:

• Robot kinematics: Sequences of actions $a_{1:T}$ are provided as either end-effector poses $\{T_{ee,t}\}$ or joint angles $\{q_t\}$, with full-body forward kinematics ${T_{k,t}^{recon}}_{k=1..K}=FK(q_t,M)$ for all $K$ robot links.
• Robot 4D pointmaps: $$M_{1:T}^{robot} \in \mathbb{R}^{T \times H \times W \times 3}$$, where each pixel stores a camera-space coordinate (x,y,z) for that frame and link. The projection is $$[u\cdot z, v\cdot z, z]^T = K \cdot T_{recon}^{cam} \cdot T_{k,t}^{recon} \cdot x$$, where $K$ denotes intrinsics.
• World representation: The initial frame $I_0 \in \mathbb{R}^{H\times W\times 3}$, world pointmaps $M_{1:T}^{world}$, and RGB video sequences $\{T_{ee,t}\}$0.
• Pixel-level robot mask: $\{T_{ee,t}\}$1, indicating per-frame occupancy.
• Synthetic-only data: Ground-truth depth and simulator-native state records.

Annotations are stored in standardized formats: RGB and pointmap signals in 3-channel MP4s (with pointmaps normalized per sequence), masks as single-channel videos, and kinematic/metadata JSON or NPZ/NPY files.

4. Data Structure and Access Patterns

The directory structure for each split is hierarchical, supporting efficient sharding and parallel access. Each episode encapsulates comprehensive annotation and metadata to facilitate embodied simulation and generative model training:

$\{T_{ee,t}\}$2

• Video files are provided at 24 fps for both RGB and (normalized) pointmap sequences.
• Actions, kinematics, and camera parameters are stored as structured JSON and NumPy (NPZ) objects for compatibility with Python-based robotics pipelines.
• Metadata includes robot type, scenario, domain (real/sim), and task outcome.

5. Splits, Benchmarks, and Evaluation Protocols

The dataset is partitioned into:

• Training split: 98% (~197,026 episodes)
• Validation split: 2% (3,200 episodes: 2,000 DROID, 1,000 Bridge, 100 RT-1, 100 LIBERO)
• Test split: Held-out test set matching validation scale

Benchmarking is conducted at both the 2D and 4D levels:

• 2D video metrics: PSNR, SSIM, FID, FVD, LPIPS—targeting visual and perceptual similarity between generated and ground-truth videos.
• 4D geometric metrics: Chamfer Distance (L₁, L₂), [email protected], and temporal variants, measuring 3D/4D consistency and alignment over trajectory sequences.
• Policy evaluation: Reported as success rates in pick-and-place tasks, both in simulation and out-of-distribution (OOD) real-world generalization.

6. Statistics and Distributions

Key distributional characteristics include:

• Fixed episode length: 49 frames per episode ensures temporal uniformity for sequence-based modeling.
• Action representation: Approximate 50% split between end-effector and joint-space controls.
• Domain composition: ~60% real (DROID+Bridge+RT-1) to ~40% synthetic (LIBERO). In the synthetic set, for each trajectory, 1 is a success and 9 are failures induced by noise.
• Platform and object diversity:
  • 8 distinct robotic manipulator designs from public benchmarks and reconstructions.
  • Over 30 object categories, expanding coverage of real-world manipulation scenarios.
• Failure modeling: Grasp misses, collisions, and occlusions are algorithmically synthesized for robust modeling of real-world imperfections.

7. Qualitative Visualization and Example Episodes

Visualization is a core aspect in both the dataset documentation and evaluation. Each episode is visualized as:

• Initial RGB frame (context snapshot).
• 4D point cloud sequences rendered from multiple camera frustums, color-coded by XYZ.
• Overlays of robot pointmaps atop background scenes for spatial occlusion disambiguation.

The released "demo.mp4" provides a side-by-side playback of ground-truth versus generated RGB and pointmap sequences, with special emphasis on "near-miss" failure cases. Qualitative 2D/4D grids in supplementary material enable granular comparison to baseline outputs.

A plausible implication is that the breadth of visualization modalities and systematic side-by-side comparisons facilitate both qualitative and quantitative assessment of embodied generative models’ 4D physical accuracy, particularly in failure-prone and OOD scenarios.

---

Robo4D-200k establishes a comprehensive, richly-annotated foundation for 4D spatiotemporal modeling of robot-world interactions, spanning a wide range of real and synthetic platforms, scenes, objects, and outcomes, while providing rigorous evaluation schemes for policy learning, visual generation, and zero-shot sim-to-real transfer (Xu et al., 17 Mar 2026).