Robo3R: Manipulation-Ready 3D Reconstruction

• Robo3R is a feed-forward, manipulation-centric 3D reconstruction model that fuses RGB images and robot states to predict metrically consistent scene geometry.
• It leverages robot priors and keypoint-based PnP refinement to deliver precise camera pose and scale alignment in a canonical robot frame.
• Evaluations show Robo3R outperforms standard depth sensors and feed-forward baselines in precision tasks like grasping and collision-free motion planning.

Robo3R is a feed-forward, manipulation-ready 3D reconstruction model that predicts accurate, metric-scale scene geometry directly from RGB images and robot states in real time. It is designed for robotic manipulation settings in which visually plausible geometry is insufficient, and where downstream modules require sharp, metrically consistent point clouds expressed in the canonical robot frame. The system jointly infers scale-invariant local geometry, relative camera poses, and a learned global similarity transformation, and it is trained on Robo3R-4M, a synthetic dataset containing four million high-fidelity annotated frames. Across imitation learning, sim-to-real transfer, grasp synthesis, and collision-free motion planning, Robo3R is presented as an alternative 3D sensing module that can outperform both selected feed-forward reconstruction baselines and commodity depth sensors, especially in transparent, reflective, thin-structure, and small-object regimes (Yang et al., 10 Feb 2026).

1. Problem setting and manipulation-centric design

Robo3R is motivated by the claim that manipulation needs a different form of 3D perception than generic scene reconstruction. The target is not merely a point cloud suitable for visualization, but accurate, metric-scale, sharp geometry in a robot-centered coordinate frame, produced at real-time speed from sparse RGB views. The paper positions this requirement against two limitations: commodity depth sensors are noisy and material-sensitive, while existing feed-forward geometry models such as DUSt3R, VGGT, MapAnything, and DepthAnything3 do not provide the metric consistency, camera-pose precision, and fine-grained geometry needed for physical interaction (Yang et al., 10 Feb 2026).

The practical examples given in the paper are explicitly manipulation-specific. A reconstruction that is acceptable for visualization may still be inadequate to insert a screw with a 2 mm tolerance, plan collision-free motion around thin obstacles, or grasp tiny objects. This framing places Robo3R within robot perception rather than generic 3D vision: its output is intended to be consumed directly by policies, grasp planners, and motion planners.

A central design choice is the use of robot priors. Robo3R combines RGB images with robot joint states and uses forward kinematics for an explicit keypoint-based camera refinement stage. This suggests a broader methodological stance: in manipulation, geometry, embodiment, and calibration should not be treated as separable post hoc steps, but as jointly learned or jointly constrained components of a sensing stack.

2. Geometric formulation and canonicalization

The input to Robo3R is one or two RGB images,

$$\{I_i\}_{i=1}^N,\quad I_i \in \mathbb{R}^{H \times W \times 3},\quad N \in \{1,2\},$$

together with robot joint states

$\mathbf{J} \in \mathbb{R}^Q.$

The model predicts dense depth maps $D$, normalized image coordinates $C$, relative camera translation and rotation $(\mathbf{t}_{\text{rel}}, \mathbf{R}_{\text{rel}})$, and a global similarity transformation $\mathbf{S}$ (Yang et al., 10 Feb 2026).

The reconstruction pipeline has three conceptual stages. First, Robo3R predicts a scale-invariant local geometric representation from each input image. Rather than regressing 3D points directly in world or robot coordinates, it predicts normalized image coordinates $\mathbf{c}=(x,y)$ and depth $d$, then unprojects them into camera coordinates:

$$\mathbf{P}_{\text{local}} = [x \cdot d,\ y \cdot d,\ d]^\top.$$

This decomposition separates ray geometry from depth and is used to learn local shape before resolving global metric scale.

Second, Robo3R estimates relative camera poses and uses them to register multi-view local point clouds:

$$\mathbf{P}_{\text{reg}} = \left\{ \mathbf{R}_{\text{rel}}^i \mathbf{P}_{\text{local}}^i + \mathbf{t}_{\text{rel}}^i \mid i = 1,\ldots,N \right\}.$$

The result is a registered scene representation that is still not yet in a globally metric robot frame.

Third, the model predicts a global similarity transformation $\mathbf{J} \in \mathbb{R}^Q.$0 and maps each registered point into the canonical robot frame:

$\mathbf{J} \in \mathbb{R}^Q.$1

The similarity transformation is parameterized as

$\mathbf{J} \in \mathbb{R}^Q.$2

where $\mathbf{J} \in \mathbb{R}^Q.$3 is a rigid transformation and $\mathbf{J} \in \mathbb{R}^Q.$4 resolves metric scale. This canonicalization is central to the paper’s practical claim: downstream manipulation modules can consume 3D data in the robot’s own frame without requiring traditional calibration pipelines.

Robo3R further refines camera alignment with an extrinsic estimation module based on robot keypoints and Perspective-n-Point. Robot keypoints are predefined along the kinematic chains, their 3D positions in the robot frame are computed from forward kinematics, and a dedicated keypoint head predicts their 2D image coordinates. Camera extrinsics are then estimated by solving a PnP problem. The paper presents this as more robust and accurate than direct extrinsic prediction.

3. Architecture and supervision

Robo3R uses separate image and robot-state encoders. Images are encoded with DINOv2 ViT-L, producing per-view patch features

$\mathbf{J} \in \mathbb{R}^Q.$5

while robot state is encoded by a multilayer perceptron with GeLU activations into

$\mathbf{J} \in \mathbb{R}^Q.$6

The two modalities are fused by element-wise addition, yielding

$\mathbf{J} \in \mathbb{R}^Q.$7

Learnable similarity-transformation tokens are appended to this sequence to aggregate global information for metric-scale alignment in the robot frame (Yang et al., 10 Feb 2026).

The backbone is a transformer using the Alternating-Attention mechanism from VGGT. It stacks 18 blocks alternating between global attention and frame-wise attention. This organization is used to propagate information both within an image and across multiple views.

Several task-specific heads are attached to the backbone. The masked point head is a main contribution. It predicts depth, normalized image coordinates, and masks separately, with masks partitioning the scene into robot, objects, and background. The head consists of a five-layer transformer decoder followed by three MLP heads—depth, ray, and mask—with pixel shuffle upsampling. This decomposition is intended to avoid the over-smoothing often seen in dense geometric prediction and to preserve thin structures and sharp discontinuities.

The relative pose head predicts translation as a 3D vector and rotation in a 9D representation, later reshaped to a $\mathbf{J} \in \mathbb{R}^Q.$8 matrix and orthogonalized via SVD. The similarity transformation head shares the pose-head architecture and adds an MLP for scale prediction. The keypoint head predicts a heatmap

$\mathbf{J} \in \mathbb{R}^Q.$9

from which a differentiable Soft-Argmax extracts sub-pixel coordinates

$D$0

Training uses a multi-task objective covering local geometry, normals, masks, relative pose, global similarity transformation, and keypoints (Yang et al., 10 Feb 2026). The local point loss is

$D$1

where $D$2 aligns predicted local points to ground truth in a scale-invariant manner. The normal loss is

$D$3

with

$D$4

Mask supervision uses binary cross-entropy,

$D$5

Relative pose is supervised through derived relative transforms,

$D$6

with loss

$D$7

Global metric alignment is supervised by

$D$8

and keypoints by

$D$9

The full objective is

$C$0

4. Robo3R-4M dataset and optimization regime

Robo3R-4M is the synthetic training corpus used to learn manipulation-grade geometry. It is generated in NVIDIA Isaac Sim using path tracing for photorealistic rendering and physics simulation at 30 Hz. The dataset contains 100,000 scenes and 4 million frames, with 16,911 objects from DTC and Objaverse, 4,710 textures, and 6,512 environment maps. It records RGB images, depth images, semantic masks, robot states, and camera intrinsics and extrinsics (Yang et al., 10 Feb 2026).

The data-generation pipeline emphasizes diversity and domain randomization. Cameras are pinhole sensors at $C$1 resolution with randomized intrinsics $C$2, randomized focus distance and f-number, and randomized extrinsics sampled on a spherical shell with a “look-at” constraint toward the workspace. Lighting includes dome lights with HDRI environment maps, sphere lights, and distant lights, all randomized in intensity, orientation, and color. Objects vary in pose and scale and span standard PBR materials as well as glass-like transparent materials with randomized index of refraction and frosting. Robot materials and tabletop backgrounds are also randomized.

The model has 969.5M parameters. The DINOv2 image encoder is frozen during training, and the transformer backbone is initialized from $C$3. Input images are $C$4 during both training and inference. Training uses either one or two viewpoints, with random cropping, color jitter, and Gaussian blur. Optimization is performed on 32 RTX 4090 GPUs for about six days using AdamW with weight decay 0.05, betas $C$5, a maximum learning rate of $C$6, cosine decay, and a 3,000-step linear warmup. Mixed precision with fp32 and bf16 is used, and the mini-batch size is 384 images.

At inference time, Robo3R runs at 43.5 Hz for monocular input and 18.7 Hz for binocular input on a single RTX 4090. Since the real robot system in the paper operates at 10 Hz, the reconstruction module satisfies the stated real-time requirement.

5. Reconstruction benchmarks and ablation findings

Evaluation is conducted on a test set of 2,000 scenes and 80,000 frames. Point-map metrics are scale-invariant point error, normal error, and scale error. Relative camera pose metrics are relative translation error (RTE), relative rotation error (RRE), relative translation accuracy (RTA), and relative rotation accuracy (RRA) (Yang et al., 10 Feb 2026).

The reported benchmark results are as follows.

Setting	Robo3R	Strongest baseline reported
Monocular point-map: point / normal / scale	0.006 / 0.080 / 0.007	$C$7: 0.061 / — / —
Binocular point-map: point / normal / scale	0.005 / 0.079 / 0.004	$C$8: 0.032 / 0.129 / 0.483
Binocular relative pose: RTE / RRE / [email protected] / [email protected]	0.014 / 0.013 / 0.951 / 0.899	$C$9: 0.116 / 0.073 / 0.110 / 0.245

In monocular point-map estimation, Robo3R achieves a point error of 0.006, normal error of 0.080, and scale error of 0.007. The best baseline on point error is $(\mathbf{t}_{\text{rel}}, \mathbf{R}_{\text{rel}})$0 at 0.061, while MapAnything, DepthAnything3, and VGGT are reported at 0.078, 0.075, and 0.126 respectively. In binocular reconstruction, Robo3R improves to 0.005, 0.079, and 0.004, while $(\mathbf{t}_{\text{rel}}, \mathbf{R}_{\text{rel}})$1, MapAnything, DepthAnything3, and VGGT remain substantially worse on scale recovery and overall point quality. The paper therefore frames metric consistency as a defining advantage rather than a secondary improvement.

For binocular relative camera pose, Robo3R reports RTE = 0.014, RRE = 0.013, [email protected] = 0.951, and [email protected] = 0.899. The strongest baseline, $(\mathbf{t}_{\text{rel}}, \mathbf{R}_{\text{rel}})$2, reports 0.116, 0.073, 0.110, and 0.245. This indicates that the improvement is not limited to depth estimation; inter-view registration and alignment are also materially stronger.

Qualitative comparisons are made against $(\mathbf{t}_{\text{rel}}, \mathbf{R}_{\text{rel}})$3, LingBot-Depth, and a RealSense D455. The paper states that Robo3R reconstructs objects as narrow as 1.5 mm even when they occupy only 1–2 image pixels, and that it handles transparent and reflective objects that “blind” depth sensors. These examples are consistent with the masked point head’s stated purpose of preserving sharp boundaries and fine structures.

The ablation studies isolate several design decisions. Keypoint-based PnP refinement reduces absolute translation error from 0.018 to 0.016 and absolute rotation error from 0.018 to 0.016, while improving ATA@0.01 from 0.334 to 0.442 and ARA@0.01 from 0.359 to 0.415. Conditioning on robot state also improves results: without robot state, point error is 0.006, normal error 0.081, ATA 0.903, and ARA 0.831; self-attention fusion yields 0.006, 0.086, 0.900, and 0.821; the reported fusion method yields 0.005, 0.079, 0.903, and 0.838. This suggests that robot priors are especially helpful for surface quality and rotation accuracy.

6. Downstream manipulation performance

Robo3R is evaluated as a sensing front-end rather than only as a reconstruction benchmark. In imitation learning, it is paired with Maniflow, a manipulation policy that can operate on RGB or 3D point clouds. The tasks are Sweep Bean, Insert Screw, Breakfast, and BiDex Pour. Robo3R + Maniflow achieves 14/16, 15/16, 12/16, and 16/16, respectively. The corresponding results for $(\mathbf{t}_{\text{rel}}, \mathbf{R}_{\text{rel}})$4 are 11/16, 4/16, 4/16, and 12/16; for RGB-based Maniflow, 10/16, 2/16, 5/16, and 9/16; for depth-camera Maniflow, 4/16, 7/16, 11/16, and 16/16; and for Maniflow with other feed-forward reconstructions, the result is reported as infeasible (“-”) (Yang et al., 10 Feb 2026).

These results separate two issues that are often conflated. First, a 3D representation can be preferable to raw RGB for geometry-sensitive manipulation. Second, the quality of that 3D representation matters: the paper reports that generic feed-forward reconstructions are not precise enough to support feasible actions, whereas Robo3R is. Depth remains competitive on BiDex Pour and close on Breakfast, but Robo3R is stronger on Sweep Bean and Insert Screw, which stress tiny objects and high-precision insertion.

In sim-to-real transfer, policies are trained in simulation on Push Cube and Pick Cube and then deployed in reality. The reported success counts are 3/16 and 2/16 for RGB camera input, 7/16 and 5/16 for depth camera input, and 16/16 and 12/16 for Robo3R. The paper interprets this as evidence that reconstructed 3D geometry from RGB can be more domain-invariant than either raw RGB or simulated depth.

For grasp synthesis, AnyGrasp is run on Robo3R point clouds. In the normal, transparent/reflective, and small-object categories, depth-camera input achieves 12/16, 7/16, and 6/16, while Robo3R achieves 14/16, 10/16, and 11/16. Other feed-forward reconstructions are again reported as infeasible. For collision-free motion planning with cuRobo, the results are 5/5, 2/5, and 2/5 for depth-camera input on normal, transparent/reflective, and thin obstacles, and 5/5, 4/5, and 5/5 for Robo3R. This is significant because physically incorrect geometry in planning directly translates into collision risk.

Taken together, these downstream experiments position Robo3R as a robot-aware 3D sensing stack rather than a standalone reconstruction model. Its contribution lies in producing geometry that is sufficiently sharp, metrically aligned, and temporally practical for use in closed-loop robotic systems.

7. Limitations and relation to adjacent research

The stated limitations are specific. Robo3R currently supports pinhole cameras and only a limited set of robot embodiments. It relies on robot joint states and forward kinematics for keypoint-based refinement, so it assumes accurate robot state and a known kinematic model. The core training distribution centers on manipulation setups with Franka and UR5e/XHand systems, even though the appendix reportedly shows some generalization to a wheeled dual-arm robot and robot-free indoor scenes. Severe occlusion, novel embodiments, and non-pinhole optics remain likely challenge areas. The model size—969.5M parameters—also implies deployment on hardware comparable to an RTX 4090 rather than lightweight onboard compute (Yang et al., 10 Feb 2026).

Within adjacent research, Robo3R occupies a distinct position. Auto3R is an active robotic scanning system that automates next-view and next-path selection through data-driven uncertainty prediction over rendered RGB and depth, with a reconstruction backend based on 3D Gaussian Splatting; its emphasis is iterative acquisition and uncertainty-guided scanning rather than feed-forward manipulation-ready geometry in the canonical robot frame (Shen et al., 4 Dec 2025). Robo360, as described in its abstract, is a dataset featuring robotic manipulation with dense view coverage and a diverse set of objects with various physical and optical properties, intended to enable high-quality 3D neural representation learning; however, the supplied document does not provide a substantive dataset description beyond that abstract-level characterization (Liang et al., 2023).

This suggests a three-way distinction within robot-centric 3D perception. Robo360 is positioned as a dense multi-view manipulation dataset, Auto3R as an uncertainty-guided active scanning framework, and Robo3R as a feed-forward RGB-based reconstruction foundation model for manipulation. Robo3R’s defining feature is therefore not active viewpoint selection or dataset coverage alone, but the combination of scale-invariant local geometry, relative pose estimation, global similarity alignment to the canonical robot frame, masked dense prediction for sharp geometry, and explicit PnP-based extrinsic refinement.