Calib3R: Unified Robot Calibration

• Calib3R is a unified robotics calibration-and-reconstruction system that estimates camera-to-robot transforms and recovers metric 3D scenes from RGB images and robot poses.
• It leverages dense geometric correspondences from the MASt3R model and integrates robot motion constraints in a single optimization to address scale ambiguity and conventional calibration limits.
• Designed for both single and multi-camera setups on manipulators and mobile robots, Calib3R enables efficient, patternless calibration that is immediately usable for robot-centric perception and planning.

Calib3R is a unified robotics calibration-and-reconstruction pipeline for jointly estimating camera-to-robot extrinsics and recovering a 3D scene that is both metric-scaled and expressed in the robot reference frame, using only RGB images and robot poses. It is presented as a patternless method for single-camera and multi-camera systems on robot arms or mobile robots, and it builds on the 3D foundation model MASt3R to extract dense geometric correspondences from image pairs. The central premise is that camera-to-robot calibration and dense 3D reconstruction depend on the same RGB-derived geometric correspondences and can therefore be solved in a single optimization rather than as separate stages (Allegro et al., 10 Sep 2025).

1. Problem setting and conceptual scope

Calib3R addresses a setting in which a robot platform is equipped with one or more RGB cameras and provides, as input, a sequence of RGB images from each camera together with the corresponding robot poses from kinematics or odometry. The stated goals are to estimate the rigid transform between each camera and the robot reference frame, recover a globally consistent 3D reconstruction of the observed static scene, recover metric scale, and do so for both single-camera and multi-camera systems without calibration patterns or depth sensors (Allegro et al., 10 Sep 2025).

The paper frames this problem against two standard limitations. First, ordinary RGB-only multi-view reconstruction produces geometry in arbitrary scale and in an arbitrary coordinate frame. Second, classical hand-eye or motion-based calibration typically assumes metric camera motion derived from a target or pattern. Calib3R is designed to close that gap by fusing robot motion, which is already metric through kinematics or odometry, with dense 3D correspondence information derived from RGB data (Allegro et al., 10 Sep 2025).

The scope includes both manipulators and mobile robots. For a robot arm, the robot reference frame can be the end-effector or base depending on the setup; for a mobile robot, the reference frame is the robot base. The reconstruction is expressed in the initial robot reference frame, $\mathcal{W}=\mathcal{R}_0$, which is what makes the final scene both metric and robot-aligned. A plausible implication is that the method is intended not merely for calibration in isolation but for calibration that is immediately usable in robot-centric perception and planning pipelines.

2. Use of MASt3R and local 3D correspondence extraction

Calib3R builds on MASt3R, which takes two RGB images and predicts dense pointmaps, confidence maps, and feature maps. For two images

$$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$

MASt3R predicts pointmaps such as

$$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$

confidence maps

$$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$

and feature maps

$$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$

Dense matches are obtained by nearest-neighbor matching in feature space, producing correspondences

$$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$

These quantities provide the geometric substrate on which the downstream optimization operates (Allegro et al., 10 Sep 2025).

As in MASt3R-SfM, Calib3R first constructs a sparse co-visibility graph over image pairs. A pairwise matcher estimates whether two images overlap and yields a connectivity graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$. Only informative overlapping pairs are kept. This design choice is explicitly motivated by scalability, since optimization over all possible pairs would be unnecessary and computationally expensive.

For each image $n$, Calib3R forms a canonical pointmap by aggregating pointmaps from multiple overlapping pairs through a confidence-weighted average:

$\tilde{\mathbf{X}^{n} = \frac{\sum_{e\in\mathcal{E}^{n} \mathbf{C}^{n,n}(e)\,\mathbf{X}^{n,n}(e)} {\sum_{e\in \mathcal{E}^{n} \mathbf{C}^{n,n}(e)}.$

These canonical pointmaps remain local reconstructions. To place them in a common world frame $\mathcal{W}$, the method introduces constrained pointmaps $$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$0 that depend on camera intrinsics, camera pose, and a scale factor through inverse reprojection. This suggests that Calib3R uses MASt3R not as a terminal reconstruction stage but as a dense geometric prior whose outputs are constrained and rescaled inside a robot-aware optimization.

3. Joint objective and calibration formulation

The defining feature of Calib3R is a single objective that jointly estimates the global poses of all camera views in the world frame, one scale factor per camera trajectory, the camera-to-robot transform or transforms, and, in multi-camera systems, inter-camera constraints. The overall loss is

$$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$1

where $$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$2 is the number of cameras (Allegro et al., 10 Sep 2025).

The scene-geometry part comprises a 3D matching term and a 2D reprojection term. The 3D matching loss enforces agreement between 3D correspondences:

$$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$3

The 2D reprojection loss refines consistency by projecting points back into the images:

$$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$4

The confidence weight $$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$5 makes reliable correspondences contribute more heavily.

The calibration component is based on the classic hand-eye equation

$$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$6

with optimization form

$$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$7

In Calib3R, $$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$8 is the robot motion between poses $$\mathcal{I}^{i}, \mathcal{I}^{j}\in\mathbb{R}^{W\times H\times 3},$$9 and $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$0, $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$1 is the camera motion inferred from the reconstructed camera trajectory, and $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$2 is the unknown camera-to-robot rigid transform. Because RGB-only reconstruction yields camera motion only up to scale, Calib3R augments the hand-eye loss with a per-camera scale parameter:

$$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$3

The scaled camera motion is

$$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$4

The paper emphasizes that scale is not recovered in a separate post-processing step or through grid search. Instead, it is optimized jointly with the calibration parameters. This is the mechanism by which the reconstruction becomes metric-scaled and aligned to the robot frame. In technical terms, Calib3R couples the gauge freedom of monocular or RGB-only reconstruction directly to robot-motion constraints inside a single gradient-based estimation problem.

4. Multi-camera rigidity, coordinate frames, and scale recovery

For multiple cameras rigidly mounted on the same robot, Calib3R adds cross-camera constraints. If $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$5 and $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$6 are two cameras, their relative transform should remain fixed over time. The paper formulates this as a cross-camera calibration constraint in the spirit of $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$7, with $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$8 the unknown fixed transform between cameras and the two motion sequences supplying the $$\mathbf{X}^{i,i}, \mathbf{X}^{i,j}\in\mathbb{R}^{W\times H\times 3},$$9 and $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$0 terms (Allegro et al., 10 Sep 2025).

This cross-camera term preserves rigidity among the cameras and helps align all camera trajectories in the same robot-centric world frame. In the single-camera case, $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$1 and $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$2 vanishes; the framework reduces to geometry, hand-eye calibration, and scale recovery. The same formulation therefore subsumes both single-camera and multi-camera systems without requiring separate algorithmic variants.

The estimated outputs are explicitly enumerated as camera poses $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$3 for each camera $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$4 and time $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$5, one scale factor $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$6 per camera, and rigid camera-to-robot transforms $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$7. The world frame is chosen as the initial robot reference frame, $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$8. This coordinate convention is important: it means that the final scene reconstruction is not merely metrically scaled but directly represented in a frame native to the robot.

Scale recovery is presented as the key bridge between unscaled RGB reconstruction and robotics. The robot motion $$\mathbf{C}^{i,i}, \mathbf{C}^{i,j}\in\mathbb{R}^{W\times H\times 1},$$9 is metric because it comes from odometry or kinematics, whereas the camera trajectory is only defined up to scale. The optimizer therefore solves for the scale that makes the camera motion consistent with robot motion under the hand-eye-style loss. The paper explicitly contrasts this with prior work that used grid search over scale values and states that Calib3R estimates scale directly in the same gradient-based optimization. A plausible implication is that this tighter coupling improves both numerical efficiency and consistency between geometry and calibration.

5. Experimental evaluation and quantitative results

The evaluation spans manipulator datasets and mobile-robot datasets. The manipulator datasets are Franka Pattern, Franka Object, and GraspNet-1Billion. Franka Pattern uses a Franka Emika Panda arm with an Azure Kinect, 25 poses, and a $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$0 checkerboard with 3 cm squares. Franka Object uses the same robot and poses but replaces the checkerboard with arbitrary objects. GraspNet-1Billion uses 25 images from scene_0100. The mobile-robot datasets are MEMROC, OpenLORIS, and CSE Benchmark. MEMROC contains real and synthetic multi-camera mobile robot sequences with a checkerboard; OpenLORIS contains single-camera indoor service-robot sequences; CSE Benchmark provides synthetic multi-camera data from Isaac Sim. Calib3R and foundation-model methods ran on an NVIDIA A40 GPU, while classical baselines ran on CPU (Allegro et al., 10 Sep 2025).

Calibration performance is reported via translation and rotation errors for camera-to-robot transforms:

$$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$1

Metric scale error is evaluated using checkerboard geometry:

$$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$2

On Franka Pattern, Calib3R achieves $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$3 cm and $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$4 rad. The paper states that this is better than all reconstruction-based baselines and best in rotation among all methods, while Evangelista’s pattern-based method slightly beats it in translation on that dataset. On Franka Object, where no pattern is present, Calib3R is reported as clearly best among reconstruction-based methods, with $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$5 cm and $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$6 rad. On GraspNet-1Billion, it achieves $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$7 cm and $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$8 rad (Allegro et al., 10 Sep 2025).

For metric scale accuracy on Franka Pattern, Calib3R achieves mean scale error $$\mathbf{D}^{i,i}, \mathbf{D}^{i,j}\in\mathbb{R}^{W\times H\times d}.$$9 cm, standard deviation $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$0 cm, and relative error $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$1. The paper states that this is better than DUSt3R + Calib, MASt3R-SfM + Calib, and VGGT + Calib, with VGGT + Calib as the nearest competitor at $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$2.

On mobile robots, Calib3R is described as competitive with or better than state-of-the-art pattern-free methods on MEMROC, OpenLORIS, and CSE Benchmark. It notably outperforms Zuniga’s method by a wide margin and outperforms SensorX2Car because it also estimates translation. On the real MEMROC dataset, it obtains $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$3 cm and $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$4 rad.

The compared baselines include classical pattern-based hand-eye methods—Tsai, Park, Horaud, Andreff, Daniilidis, Evangelista—patternless reconstruction-based pipelines—COLMAP + Calib, DUSt3R + Calib, MASt3R-SfM + Calib, VGGT + Calib—and mobile-robot calibration baselines—Zuniga, MEMROC, Joint-MEMROC, SensorX2Car. The paper highlights that many baselines either require a calibration pattern, estimate only rotation, or do not produce a full metric reconstruction. That contrast is central to Calib3R’s positioning within the calibration literature.

6. Data efficiency, assumptions, limitations, and practical role

A major empirical claim is that Calib3R remains accurate with very few images. In ablations on Franka Object and MEMROC, the number of images is reduced from 25 down to 5. The paper reports that Calib3R stays below 5 cm translation error and below 0.04 rad rotation error with only 5 images on the Franka Object experiment, reaches below 1 cm translation error with about 15 images, and maintains strong performance with fewer than 10 images per camera in mobile-robot experiments (Allegro et al., 10 Sep 2025).

The paper presents this as a practical advantage for fast recalibration. Because the method is patternless and data-efficient, it can plausibly reduce the burden of frequent recalibration in settings where sensors are remounted, robot kinematics drift, or camera rigs undergo minor mechanical disturbances.

The assumptions are explicit. The scene is assumed to be static during image capture; robot poses must be available from kinematics or odometry; enough overlap must exist between views for MASt3R-based matching; and, for mobile robots, $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$5-translation is not observable from planar motion alone. To address the mobile-robot observability issue, Calib3R estimates the ground plane from the reconstructed 3D scene using Grounding DINO + SAM2 to segment the floor and then back-projects the floor points. The $$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$6-height is then estimated as the average height,

$$\mathcal{M}^{i,j}=\{y_{p}^{i}\leftrightarrow y_{p}^{j}\}_{p=1,\dots,|\mathcal{M}^{i,j}|}.$$7

This is a specific correction for a known degeneracy rather than a generic component of the optimization.

The method also depends on a strong foundation model, and the paper states that computation is heavier than in classical calibration. The reported experiments are offline rather than real-time, and future work is identified in real-time operation and dynamic environments. These constraints delimit the current applicability of the method: it is a high-capacity RGB-only calibration-and-reconstruction system, but not yet a real-time or dynamic-scene solution.

The practical implication is that Calib3R can provide both a calibrated camera rig aligned with the robot frame and a metric 3D scene map using only ordinary RGB images and robot poses. The paper specifically identifies manipulation and navigation as relevant settings, where robot-frame 3D understanding is critical for planning, grasping, obstacle avoidance, and multi-camera perception. Because it avoids fiducial markers, depth sensors, and multi-stage calibration pipelines, it is presented as more deployable in real-world settings where recalibration may be frequent. This suggests that Calib3R belongs to a broader trend in robotics toward folding geometric estimation, calibration, and scene reconstruction into a single learned-plus-optimized pipeline rather than treating them as isolated modules.