Robo3R-4M: 3D Perception Corpus for Manipulation

• Robo3R-4M is a synthetic 3D perception corpus offering 4M high-fidelity annotated frames for training feed-forward reconstruction models.
• It provides precise, metric-scale scene geometry, exact camera intrinsics/extrinsics, and robot state supervision for manipulation-grade applications.
• The dataset supports robust robotic interaction with diverse objects, realistic lighting, and comprehensive annotations for depth, poses, and keypoint estimation.

Searching arXiv for the primary Robo3R paper and a possible acronym-confusion paper.

{"query":"arXiv (Yang et al., 10 Feb 2026) Robo3R Enhancing Robotic Manipulation with Accurate Feed-Forward 3D Reconstruction", "max_results": 5} {"query":"arXiv (Li et al., 2023) R^3 On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics", "max_results": 5} Robo3R-4M is a synthetic, manipulation-focused 3D perception corpus introduced to train feed-forward reconstruction systems to manipulation-grade precision. It accompanies Robo3R, a model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time, and is designed around the requirements of robotic interaction: millimeter-to-centimeter accurate geometry, reliable metric scale, and consistent camera/robot frames. The corpus provides four million high-fidelity annotated frames, exact intrinsics and extrinsics, and robot-state supervision so that multi-view, scale-invariant local geometry can be unified into metric-scale 3D in the canonical robot frame without manual calibration pipelines (Yang et al., 10 Feb 2026).

1. Motivation and problem setting

Robo3R-4M is motivated by the mismatch between the sensing requirements of robotic manipulation and the limitations of conventional depth acquisition. Robotic interaction requires millimeter-to-centimeter accurate geometry, reliable metric scale, and consistent camera/robot frames, whereas commodity depth sensors such as stereo and time-of-flight degrade on transparent and reflective materials, fine and thin objects, and adverse lighting. Existing reconstruction models, in turn, are described as lacking the precision and metric consistency required for physical interaction (Yang et al., 10 Feb 2026).

The dataset is explicitly synthetic because the target supervision is high-fidelity, perfectly labeled, and metrically consistent. Synthetic rendering with physically based path tracing provides controllable photorealism, diverse materials including glass and metals, and exact ground truth for depth, intrinsics, extrinsics, and robot states, while avoiding calibration and sensor artifacts. This design also supports learning a global similarity transform that maps scale-invariant local geometry into metric geometry in the robot base frame.

The design goals are stated as large scale; broad diversity in objects, textures, and lighting; realistic robot motion and viewpoints; rich annotations aligned to robot kinematics; strong domain randomization to foster sim-to-real generalization; and modalities sufficient to supervise depth, poses, masks, similarity scaling, and robot keypoint detection. A plausible implication is that Robo3R-4M is intended not merely as a reconstruction benchmark, but as an integrated supervision substrate for manipulation pipelines that depend on metrically consistent scene geometry.

2. Scale, composition, and scene diversity

Robo3R-4M contains 4,000,000 annotated frames spanning 100,000 scenes. The scenes are physically simulated episodes in Isaac Sim at 30 Hz, averaging approximately 40 frames per scene, and a held-out synthetic test set contains 2,000 scenes and 80,000 frames (Yang et al., 10 Feb 2026).

Component	Specification
Total corpus	4,000,000 annotated frames
Scene count	100,000 scenes
Held-out test set	2,000 scenes, 80,000 frames
Object assets	16,911 assets from DTC and Objaverse
Textures	4,710 textures
HDR environment maps	6,512 HDR environment maps
Base render resolution	640×480

The scene diversity is defined along several axes. The object set includes 16,911 assets from DTC and Objaverse, with randomized scale and placement and a mix of physically based rendering objects and glass-like materials. Materials and textures include 4,710 textures with randomized albedo, roughness, and metallic parameters; glass materials are randomized in Index of Refraction and frosting. Environments and illumination include 6,512 HDR environment maps, dome lights with HDRI intensity in $[300, 2000]$, yaw of $\pm 180^\circ$, and tilt of $\pm 20^\circ$; sphere lights with 1–3 lights, radius in $[0.05, 0.5]$ m, intensity in $[2000, 8000]$, and randomized color temperature or RGB; and distant lights with 1–3 sun-like directional sources with randomized orientation, intensity, and color (Yang et al., 10 Feb 2026).

The manipulation scenarios use industrial arms consistent with downstream use: Franka Research 3 with a parallel gripper, and bimanual UR5e arms with XHand hands. The paper states that the model trained on Robo3R-4M generalizes to other embodiments in qualitative results. Camera setups use pinhole cameras in a multi-camera system. Intrinsics are randomized per episode, including focal lengths $(f_x, f_y)$ and principal points $(c_x, c_y)$, while focus distance and f-number are randomized to simulate depth of field. Extrinsics are randomized under a look-at constraint by sampling camera positions on a spherical shell around the workspace with random radius, azimuth, and elevation, aiming orientation at a random target point in the workspace plus random roll. This keeps the task area visible while preserving strong viewpoint diversity.

Temporal continuity is provided by 30 Hz physics. However, training uses sparse monocular or binocular views with $N \in \{1,2\}$, reflecting robot perception under limited viewpoints rather than dense multi-view sensing. This suggests that the corpus is optimized for practical manipulation settings in which a robot must act from a small number of RGB viewpoints.

3. Generation pipeline and annotation schema

Robo3R-4M is generated in NVIDIA Isaac Sim, which is used for both physics and path-traced photorealistic rendering, with physics time step approximately $0.033$ s, corresponding to 30 Hz (Yang et al., 10 Feb 2026). Asset sources are DTC and Objaverse in USD format, supplemented with extensive textures and HDRIs for environment lighting. Scene and motion creation randomize object, table, and background placement under physically plausible constraints. Robot behaviors are created by sampling end-effector target poses, initializing valid joint configurations via inverse kinematics, and exploiting known joint-level kinematic chains for keypoint generation.

Domain randomization is split into visual and physical components. Visual randomization covers camera intrinsics and extrinsics, depth of field, HDRI lighting, number and placement of local lights, material properties for both PBR and glass, and textures and backgrounds. Physical randomization covers object scales and poses, as well as robot material appearance jitter in metallic, roughness, and diffuse properties. Photorealism and camera optics effects are modeled through path tracing and optics parameters, but the dataset does not inject explicit sensor noise into depth; the ground truth remains perfect metric geometry from simulation.

Per frame, the recorded modalities are an RGB image, a depth image at metric scale, semantic masks for robot, objects, and background, camera intrinsics $K$, camera extrinsics $\pm 180^\circ$0, and robot state $\pm 180^\circ$1. Training also uses labels procedurally derived from this ground truth: 3D points from depth via unprojection in camera coordinates and registered multi-view points using relative poses, surface normals computed from ground-truth point maps, and robot kinematic keypoints from forward kinematics together with projected 2D keypoint coordinates and heatmaps for supervising the keypoint head that feeds the Perspective-n-Point solver (Yang et al., 10 Feb 2026).

The coordinate-frame conventions are explicit. All translations are in meters and rotations lie in $\pm 180^\circ$2. Local geometry is learned as scale-invariant point maps in the camera frame from normalized image coordinates and depth, and global geometry is expressed in a canonical robot frame, specifically the robot base frame, through a learned global similarity transform. That canonicalization is central to the dataset’s intended downstream reuse: it converts per-view geometry into a robot-centric metric representation compatible with planning, grasping, and policy learning.

4. Geometric formalism and supervision signals

The local 3D point in the camera frame is defined from normalized image coordinates $\pm 180^\circ$3 and depth $\pm 180^\circ$4 by

$\pm 180^\circ$5

Multi-view registration uses relative pose:

$\pm 180^\circ$6

with registered points

$\pm 180^\circ$7

The mapping to the canonical robot frame uses a global similarity transform

$\pm 180^\circ$8

where $\pm 180^\circ$9, $\pm 20^\circ$0, and $\pm 20^\circ$1, implemented as $\pm 20^\circ$2, with rigid $\pm 20^\circ$3 plus scale $\pm 20^\circ$4, applied to registered points (Yang et al., 10 Feb 2026).

Camera-extrinsic refinement is based on robot keypoints and a Perspective-n-Point objective:

$\pm 20^\circ$5

where $\pm 20^\circ$6 is the projection function, $\pm 20^\circ$7 is intrinsics, $\pm 20^\circ$8 are extrinsics, $\pm 20^\circ$9 are 3D keypoints, and $[0.05, 0.5]$0 are their 2D detections. This formulation is important because the dataset provides exact intrinsics, extrinsics, robot states, and kinematic keypoints, enabling pose refinement that is tightly coupled to robot geometry.

The supervision suite used to train Robo3R on Robo3R-4M is multitask. The point loss on unprojected points with per-frame scale alignment $[0.05, 0.5]$1 is

$[0.05, 0.5]$2

The normal loss is defined by

$[0.05, 0.5]$3

The mask loss is binary cross-entropy over robot, object, and background masks,

$[0.05, 0.5]$4

The relative pose loss combines translation Huber loss and rotation angle,

$[0.05, 0.5]$5

The similarity transformation loss is

$[0.05, 0.5]$6

The keypoint loss uses heatmaps and coordinates via Soft-Argmax,

$[0.05, 0.5]$7

The overall multitask objective is

$[0.05, 0.5]$8

The paper also states that SILog and Chamfer distance/F-score are not reported in this work, which is relevant when comparing Robo3R-4M-based evaluation to monocular depth or point-cloud reconstruction benchmarks that conventionally use those metrics (Yang et al., 10 Feb 2026).

5. Role in training Robo3R

Robo3R-4M is the training corpus for Robo3R’s feed-forward reconstruction pipeline. The training inputs are monocular or binocular RGB with $[0.05, 0.5]$9 together with robot joint states $[2000, 8000]$0. Image encoding uses a frozen DINOv2 ViT-L; robot states are encoded with an MLP with GeLU and LayerNorm; the features are fused by element-wise addition, together with learnable S.T. tokens. The backbone consists of 18 Alternating-Attention transformer blocks with interleaved global and frame-wise attention. The prediction heads are a masked point head for depth, normalized image coordinates, and masks; a relative pose head; a similarity transformation head for scale plus rigid transform; and a keypoint head with a PnP extrinsic estimation module (Yang et al., 10 Feb 2026).

Optimization uses 32 RTX 4090 GPUs for approximately six days of training. Batch sizing employs dynamic views, one or two, with the batch aggregated to 384 images; mixed precision uses fp32 and bf16. The optimizer is AdamW with weight decay $[2000, 8000]$1, betas $[2000, 8000]$2, a cosine schedule with 3,000-iteration linear warmup, and a maximum learning rate of $[2000, 8000]$3.

Ablations directly expose how Robo3R-4M’s annotations affect model quality. Keypoint supervision with PnP, compared to direct extrinsics regression, yields lower absolute translation and rotation errors and higher ATA and ARA: ATE $[2000, 8000]$4 versus $[2000, 8000]$5, ARE $[2000, 8000]$6 versus $[2000, 8000]$7, [email protected] $[2000, 8000]$8 versus $[2000, 8000]$9, and ARA@0.01 $(f_x, f_y)$0 versus $(f_x, f_y)$1. Conditioning on robot states also improves point-map quality and absolute pose accuracy relative to variants without state or with self-attention fusion; for example, point and normal error are $(f_x, f_y)$2 for Robo3R versus $(f_x, f_y)$3 without state, and ARA and ATA also improve. These results are interpreted in the paper as reflecting the value of robot-keypoint supervision and robot-state labels in Robo3R-4M (Yang et al., 10 Feb 2026).

6. Empirical significance, downstream use, and scope

On the synthetic reconstruction benchmark of 80,000 frames across 2,000 scenes, Robo3R trained on Robo3R-4M achieves, for monocular point maps, point error $(f_x, f_y)$4, normal error $(f_x, f_y)$5, and scale error $(f_x, f_y)$6, outperforming VGGT, $(f_x, f_y)$7, MapAnything, and DepthAnything3; for binocular point maps, the reported values are point error $(f_x, f_y)$8, normal error $(f_x, f_y)$9, and scale error $(c_x, c_y)$0. For relative camera pose, the reported performance is RTE $(c_x, c_y)$1 m, RRE $(c_x, c_y)$2 rad, [email protected] $(c_x, c_y)$3, and [email protected] $(c_x, c_y)$4, approximately $(c_x, c_y)$5–$(c_x, c_y)$6 better than the best baseline (Yang et al., 10 Feb 2026).

The paper further reports real-world qualitative reconstructions of 1.5 mm thin objects, transparent and reflective surfaces, and cluttered bimanual scenes more cleanly than depth cameras and other feed-forward models. In imitation learning with a Maniflow policy, the task results are: Sweep Bean, Robo3R $(c_x, c_y)$7 versus RGB $(c_x, c_y)$8 and depth $(c_x, c_y)$9; Insert Screw with 2 mm clearance, Robo3R $N \in \{1,2\}$0 versus RGB $N \in \{1,2\}$1 and depth $N \in \{1,2\}$2; Breakfast, Robo3R $N \in \{1,2\}$3 versus RGB $N \in \{1,2\}$4 and depth $N \in \{1,2\}$5; and BiDex Pour, Robo3R $N \in \{1,2\}$6 versus RGB $N \in \{1,2\}$7 and depth $N \in \{1,2\}$8, while other feed-forward systems failed and the $N \in \{1,2\}$9 baseline underperformed Robo3R across tasks. In sim-to-real transfer from Isaac Sim to real deployment, Push Cube yields RGB $0.033$0, depth $0.033$1, and Robo3R $0.033$2; Pick Cube yields RGB $0.033$3, depth $0.033$4, and Robo3R $0.033$5. In grasp synthesis with AnyGrasp, normal objects yield Robo3R $0.033$6 and depth $0.033$7, transparent or reflective objects yield Robo3R $0.033$8 and depth $0.033$9, and small objects yield Robo3R $K$0 and depth $K$1. In collision-free motion planning with cuRobo, the normal setting yields Robo3R $K$2 and depth $K$3, transparent or reflective yields Robo3R $K$4 and depth $K$5, and thin yields Robo3R $K$6 and depth $K$7 (Yang et al., 10 Feb 2026).

These results support the summary claim that Robo3R-4M enables robust training of a feed-forward reconstruction model to manipulation-grade metric accuracy without depth sensors or calibration, consistent metric geometry in the robot base frame via a learned global similarity transform grounded in exact intrinsics, extrinsics, and robot states, and superior sim-to-real performance on imitation learning, grasping, and motion planning, especially for transparent, reflective, and thin objects.

The scope of the dataset is also bounded explicitly. It models only pinhole optics; fisheye and panoramic lenses are not included. Primary supervision focuses on Franka and bimanual UR5e embodiments; broader mobile and manipulator platforms and hand designs are identified as future directions. Although sequences are physically simulated, training uses sparse views, so heavily occluded or rapidly dynamic interactions could benefit from multi-view or multi-time supervision. Extreme specular and transparent edge cases and complex lighting such as caustics can still challenge feed-forward inference. Proposed future improvements include expanded camera models and calibrations, realistic sensor artifacts and rolling shutter, increased embodiment diversity, richer kinematic priors, more temporal supervision, harder occlusion scenarios, broader environment coverage with human and robot co-presence, and extended annotations such as instance masks and per-instance 6D poses (Yang et al., 10 Feb 2026).

A common source of ambiguity is the notation “R^3,” which in a separate work denotes “On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics” rather than Robo3R or Robo3R-4M (Li et al., 2023). Within Robo3R-4M itself, “4M” refers specifically to 4,000,000 annotated frames spanning 100,000 scenes (Yang et al., 10 Feb 2026).