MUSt3R: Multi-view Network for Stereo 3D Reconstruction (2503.01661v1)

Abstract: DUSt3R introduced a novel paradigm in geometric computer vision by proposing a model that can provide dense and unconstrained Stereo 3D Reconstruction of arbitrary image collections with no prior information about camera calibration nor viewpoint poses. Under the hood, however, DUSt3R processes image pairs, regressing local 3D reconstructions that need to be aligned in a global coordinate system. The number of pairs, growing quadratically, is an inherent limitation that becomes especially concerning for robust and fast optimization in the case of large image collections. In this paper, we propose an extension of DUSt3R from pairs to multiple views, that addresses all aforementioned concerns. Indeed, we propose a Multi-view Network for Stereo 3D Reconstruction, or MUSt3R, that modifies the DUSt3R architecture by making it symmetric and extending it to directly predict 3D structure for all views in a common coordinate frame. Second, we entail the model with a multi-layer memory mechanism which allows to reduce the computational complexity and to scale the reconstruction to large collections, inferring thousands of 3D pointmaps at high frame-rates with limited added complexity. The framework is designed to perform 3D reconstruction both offline and online, and hence can be seamlessly applied to SfM and visual SLAM scenarios showing state-of-the-art performance on various 3D downstream tasks, including uncalibrated Visual Odometry, relative camera pose, scale and focal estimation, 3D reconstruction and multi-view depth estimation.

• The paper introduces MUSt3R, an efficient multi-view network for uncalibrated stereo 3D reconstruction that significantly improves performance and computational efficiency over previous pairwise methods.
• MUSt3R employs a symmetrical decoder, multi-layer memory, and enhanced prediction heads to scale efficiently and provide both local and global pointmaps for tasks like visual odometry and SLAM.
• Empirical results demonstrate MUSt3R's superior performance on benchmarks like TUM RGB-D and ETH3D, enabling real-time 3D reconstruction at scale for applications like AR and mobile systems.

MUSt3R: A Technical Examination of a Multi-view Network for Stereo 3D Reconstruction

The paper "MUSt3R: Multi-view Network for Stereo 3D Reconstruction" presents an evolved architectural approach in the field of geometric computer vision, extending the functionalities of the previously established DUSt3R framework. The salient contribution of this research is the development of MUSt3R, which transitions from a pair-based stereo reconstruction model to a highly efficient multi-view system.

Key Advancements and Methodology

DUSt3R initially introduced a method for dense, unconstrained Stereo 3D Reconstruction of image collections without requiring camera calibration or viewpoint poses. However, it was limited by its pairwise processing nature, creating quadratic computational complexity that hinders large-scale applications. MUSt3R addresses these limitations through an architectural redesign that enables direct multi-view predictions and incorporates a memory-efficient mechanism.

1. Symmetrical Decoder Design: The MUSt3R architecture simplifies the previous dual-decoder structure to a symmetric, weight-shared decoder. This design choice reduces the overall model complexity and effectively scales to multiple views while maintaining high efficiency in computation.
2. Multi-layer Memory Mechanism: By incorporating a multi-layer memory, MUSt3R scales efficiently across large datasets and supports both offline and online tasks. This mechanism reduces computational cost by keeping track of previous frame representations, alleviating the need for pairwise global alignment.
3. Enhanced Prediction Heads: The addition of a secondary prediction head allows MUSt3R to output both local and global pointmaps for each view. This enhancement facilitates rapid estimation of camera parameters such as depth and focal length, crucial for applications in real-time visual odometry (VO) and SLAM.
4. Loss Function Refined in Log Space: To balance the scale of predicted 3D coordinates, the authors implement a regression loss computed in log space. This modification bolsters the model’s performance in predicting metric-defined 3D structures, which is critical for maintaining accuracy over varying scales.

Empirical Results

MUSt3R demonstrates superior performance across several benchmark datasets, including TUM RGB-D and ETH3D, showcasing its effective handling of VO and SLAM tasks in a fully uncalibrated regime. The results yield marked improvements in Relative Rotation Accuracy (RRA) and relative translation direction accuracy (RTA) over conventional methods, such as Spann3R, confirming the efficacy of MUSt3R's approach in real-world scenarios characterized by diverse camera geometries and transformations.

The memory optimization strategies employed by MUSt3R allow for significant reductions in computational costs associated with alignment and frame processing, resulting in real-time throughput on high-resolution image sequences. This performance characteristic positions MUSt3R as a viable solution for systems necessitating robust 3D reconstructions at scale.

Implications and Future Directions

MUSt3R's contributions significantly reduce the computational overhead related to 3D reconstruction tasks, potentially translating into improvements in mobile applications where processing power and battery life are constraints. The scalable architecture further opens prospects for MUSt3R’s application in the burgeoning area of augmented reality (AR), where precise environmental mapping is crucial.

Looking forward, advancements could explore integration with hardware acceleration platforms for enhanced real-time performance. Additionally, extending MUSt3R's framework to adaptive learning paradigms could facilitate dynamic modeling of environments, improving operational robustness in novel scenes or under changing lighting conditions.

In summary, MUSt3R provides a substantial leap in multi-view stereo 3D reconstruction by balancing model complexity, computational efficiency, and performance, thereby laying a foundational block for further innovations in 3D vision technologies.