DUSt3R: Uncalibrated Stereo 3D Reconstruction

• DUSt3R is a geometric 3D vision framework that enables dense stereo reconstruction from image collections without requiring known camera calibration or poses.
• The framework uses a transformer-based network to regress "pointmaps"—dense mappings from image pixels to 3D coordinates in a canonical frame—directly from raw pixel data.
• DUSt3R supports unified solutions for tasks like dense 3D reconstruction, depth estimation, camera pose recovery, and pixel correspondence, demonstrating state-of-the-art performance on various benchmarks.

DUSt3R is a geometric 3D vision framework that enables dense, unconstrained stereo reconstruction from arbitrary image collections, operating without known camera calibration or viewpoint poses. In contrast to traditional multi-view stereo (MVS), which requires explicit knowledge of camera intrinsics and extrinsics to triangulate corresponding image points, DUSt3R regresses so-called “pointmaps”—dense mappings from image pixels to 3D coordinates in a canonical reference frame—using only the images as input. This design unifies monocular depth estimation, binocular stereo, and multi-view 3D reconstruction within a single transformer-based network and allows for end-to-end learning of spatial geometry and related quantities directly from pixel data. DUSt3R eliminates the cumbersome pre-calibration step and simplifies the 3D vision pipeline, enabling robust performance on a broad range of tasks, including dense scene reconstruction, camera pose estimation, pixel correspondence, and monocular/multi-view depth estimation.

1. Pointmap Regression and Formulation

At the core of DUSt3R is the regression of pointmaps: for an image $I \in \mathbb{R}^{W \times H \times 3}$, the network predicts $X \in \mathbb{R}^{W \times H \times 3}$ where $X_{i,j}$ gives the 3D coordinates of the scene point observed by pixel $(i,j)$. This is achieved without access to camera intrinsics $K$ or extrinsics $P$ at test time, a marked departure from the MVS convention where

$$X_{i, j} = K^{-1} \begin{pmatrix} i D_{i, j} \ j D_{i, j} \ D_{i, j} \end{pmatrix}$$

Given two images $I^1, I^2$, DUSt3R regresses two pointmaps, both expressed in the reference frame of $I^1$ via cross-attention, i.e. $X^{1 \to 1}, X^{2 \to 1}$. For monocular depth, the same image is used for both input branches. For multi-view reconstruction, pairwise reconstructions for all image pairs are fused through a global alignment step.

2. Transformer-Based Architecture

DUSt3R employs a transformer backbone:

• Encoder: Shared Vision Transformer (ViT) encodes each image into a sequence of features.
• Decoder: Two intertwined transformer decoders, one per image, exchange information at every block via cross-attention, ensuring pointmaps align in the same reference frame.

Let $F^v$ denote encoder outputs for view $v$: $$F^1 = \mathrm{Encoder}(I^1), \qquad F^2 = \mathrm{Encoder}(I^2)$$ Decoder blocks process features as: $$G^1_i = \mathrm{DecoderBlock}^1(G_{i-1}^1, G_{i-1}^2), \quad G^2_i = \mathrm{DecoderBlock}^2(G_{i-1}^2, G_{i-1}^1)$$ with regression heads producing pointmap and confidence maps.

3. Unified Loss and Training

Supervision leverages only ground-truth pointmaps, with a confidence-weighted regression loss: $$\ell(v, i) = \left\| \frac{1}{z} \hat{X}^{v \to 1}_i - \frac{1}{z} X^{v \to 1}_i \right\|$$ where $z = \mathrm{norm}(X^{v \to 1})$ normalizes for scale ambiguity. Final loss aggregates pixelwise errors weighted by predicted confidences $\hat{C}^{v \to 1}_i$ and penalizes overconfidence: $$L = \sum_{v \in \{1,2\}} \sum_{i \in D^v} \hat{C}^{v \to 1}_i \, \ell(v, i) - \alpha \log \hat{C}^{v \to 1}_i$$ This encourages the network to output low confidence in regions of geometric ambiguity or high prediction error.

4. Multi-View Reconstruction via Global Alignment

For collections of $N$ images, DUSt3R processes all relevant pairs to build a graph $G(V, E)$ where each edge corresponds to an overlapping view pair. Predicted pointmaps for each pair are related by unknown rigid transforms $P_e$ and scales $\sigma_e$. DUSt3R then solves for globally aligned pointmaps $\chi$ by minimizing: $$\min_{\chi, P, \sigma} \sum_{e \in E} \sum_{v \in e} \sum_{i=1}^{HW} \hat{C}^{v \to e}_i \left\| \chi^v_i - \sigma_e P_e \hat{X}^{v \to e}_i \right\|$$ with $\prod_e \sigma_e = 1$ to prevent degenerate solutions. This operates directly in 3D rather than minimizing image-space reprojection error, making it robust and computationally efficient—converging within seconds on a GPU.

5. Downstream Tasks and Capabilities

DUSt3R supports multiple geometric vision tasks through its unified output:

• Dense 3D Reconstruction: Fused pointmaps yield globally consistent 3D point clouds, with completeness and accuracy rivaling methods requiring explicit calibration.
• Depth Estimation: Pointmap norms provide depth, usable both in monocular and multi-view setups.
• Camera Pose and Intrinsics Recovery: By aligning pointmaps across views or analyzing the structure within $X^{1 \to 1}$, focal lengths and relative/absolute camera poses can be estimated via Procrustes or PnP procedures.
• Pixel Correspondence: Since corresponding pixels should map to the same 3D point, correspondences are readily established via nearest-neighbor matching in 3D space, enabling robust matching over wide baselines.

6. Empirical Performance and Benchmarks

DUSt3R demonstrates high performance across a diverse set of public benchmarks:

• Monocular and Multi-View Depth Estimation: Achieves state-of-the-art or near SOTA results on NYUv2, TUM, BONN (indoor), KITTI, DDAD (outdoor), DTU, ETH-3D, Tanks & Temples, ScanNet. Particularly, it achieves $\sim$2.7mm accuracy and 0.8mm completeness on DTU, outperforming many calibrated approaches.
• Relative Pose Estimation: On CO3Dv2, achieves mAA@30°=76.7%, surpassing previous methods (PoseDiffusion: 66.5%), and is highly robust even with wide baselines or little overlap.
• Visual Localization: Matches or exceeds SOTA on 7Scenes and Cambridge Landmarks, without requiring known camera models at test time.
• Runtime: Global alignment is efficient and amenable to gradient-based optimization, avoiding the iterative, slow processes of bundle adjustment.

7. Contributions, Advantages, and Limitations

DUSt3R’s primary contributions include:

1. Unified formulation: The regression of pointmaps directly enables a common framework for monocular, stereo, and multi-view 3D tasks.
2. Elimination of explicit camera calibration: By predicting geometry and correspondence in a learned canonical frame, DUSt3R avoids reliance on any known camera intrinsics or poses.
3. Efficient multi-view alignment: The global 3D alignment approach is both faster and, for many tasks, more robust than conventional bundle adjustment.
4. Transformer-based geometry reasoning: Leverages pretrained Transformer encoders and cross-attention to robustly relate features across widely varying viewpoints.
5. State-of-the-art empirical results: Outperforms or matches SOTA on a range of geometry tasks, often at lower computational cost and higher flexibility.

A plausible implication is that, while DUSt3R’s pairwise formulation and decoupled alignment are highly effective, methods building on DUSt3R (e.g., direct multi-view or global models) may address residual challenges of error accumulation and scalability for extremely large image collections.

References to Key Formulas

• 3D regression loss: $$\ell(v,i) = \left\Vert \frac{1}{z}\hat{X}^{v \to 1}_i - \frac{1}{z}X^{v \to 1}_i \right\Vert$$

$$L = \sum_{v \in \{1,2\}} \sum_{i \in D^v} \hat{C}^{v \to 1}_i \, \ell(v, i) - \alpha \log \hat{C}^{v \to 1}_i$$

• Multi-view global alignment: $$\chi^* = \arg\min_{\chi,P,\sigma} \sum_{e \in E} \sum_{v \in e} \sum_{i=1}^{HW} \hat{C}_i^{v \to e} \left\Vert \chi^v_i - \sigma_e P_e \hat{X}^{v \to e}_i \right\Vert$$

Conclusion

DUSt3R represents a paradigm shift in geometric 3D computer vision by replacing explicit geometric reasoning with transformer-based regression of 3D structure from raw images, requiring no camera calibration, and supporting a range of downstream tasks through a single unified output. Its architecture and pointmap-centric design enable robust, efficient, and accurate reconstruction and pose estimation across diverse real-world scenarios, as validated by extensive empirical benchmarking. The framework provides a basis for subsequent advances in learning-based 3D perception.