DUSt3R: Geometric 3D Vision Made Easy

Abstract: Multi-view stereo reconstruction (MVS) in the wild requires to first estimate the camera parameters e.g. intrinsic and extrinsic parameters. These are usually tedious and cumbersome to obtain, yet they are mandatory to triangulate corresponding pixels in 3D space, which is the core of all best performing MVS algorithms. In this work, we take an opposite stance and introduce DUSt3R, a radically novel paradigm for Dense and Unconstrained Stereo 3D Reconstruction of arbitrary image collections, i.e. operating without prior information about camera calibration nor viewpoint poses. We cast the pairwise reconstruction problem as a regression of pointmaps, relaxing the hard constraints of usual projective camera models. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. In the case where more than two images are provided, we further propose a simple yet effective global alignment strategy that expresses all pairwise pointmaps in a common reference frame. We base our network architecture on standard Transformer encoders and decoders, allowing us to leverage powerful pretrained models. Our formulation directly provides a 3D model of the scene as well as depth information, but interestingly, we can seamlessly recover from it, pixel matches, relative and absolute camera. Exhaustive experiments on all these tasks showcase that the proposed DUSt3R can unify various 3D vision tasks and set new SoTAs on monocular/multi-view depth estimation as well as relative pose estimation. In summary, DUSt3R makes many geometric 3D vision tasks easy.

• The paper introduces DUSt3R, which eliminates the need for explicit camera calibration by inferring parameters directly from image contents.
• It employs a regression approach on pointmaps to directly decode rich geometric details for accurate multi-view depth estimation.
• The architecture integrates multiple 3D vision tasks into one pipeline, achieving state-of-the-art performance on benchmarks like DTU, Tanks and Temples, and ETH-3D.

Introduction to DUSt3R

The paper introduces DUSt3R, a novel approach aimed at simplifying the complex tasks involved in geometric 3D vision. Traditional Multi-view Stereo Reconstruction (MVS) processes typically require careful estimation of camera parameters, which is cumbersome and error-prone. DUSt3R stands out by not needing such prior information, instead operating directly on the image contents to infer camera parameters, pixel correspondences, depthmaps, and create a full 3D reconstruction.

Unconstrained 3D Reconstruction

DUSt3R tackles the challenge of 3D reconstruction from images without predefined information about camera calibration or viewpoint poses. The system formulates pairwise reconstruction as regression of pointmaps, which diversifies from the standard projective camera models. The network efficiently decodes these pointmaps with rich geometric details from pairs of images, simplifying the process of extracting detailed scene geometry.

Seamless Integration of Multiple Tasks

An impressive feature of DUSt3R is its ability to unify various 3D vision tasks, traditionally handled separately, into a single, simplified pipeline. The architecture leverages pretrained models and a fully data-driven approach to learn powerful geometric and shape priors. This process results in direct 3D models of scenes that also lend themselves to tasks like pose estimation and monocular and multi-view reconstructions.

Flexibility and State-of-the-Art Performance

Experimentation with datasets such as DTU, Tanks and Temples, and ETH-3D demonstrates that DUSt3R works without known camera parameters. The network can handle monocular reconstruction and align multiple image pairs in a common reference frame. The evaluations position DUSt3R as setting new standards, achieving state-of-the-art results across a range of tasks including multi-view depth estimation and camera pose estimation.

Conclusion

The paper presents DUSt3R as an influential advancement in 3D geometric vision, offering a substantial simplification over traditional methods. The results underline its potential and versatility in handling diverse 3D vision challenges without the exhaustive and meticulous steps of estimating and calibrating camera parameters, which represents a significant stepping stone for future developments in the field.