Regist3R: Incremental Registration with Stereo Foundation Model (2504.12356v1)

Abstract: Multi-view 3D reconstruction has remained an essential yet challenging problem in the field of computer vision. While DUSt3R and its successors have achieved breakthroughs in 3D reconstruction from unposed images, these methods exhibit significant limitations when scaling to multi-view scenarios, including high computational cost and cumulative error induced by global alignment. To address these challenges, we propose Regist3R, a novel stereo foundation model tailored for efficient and scalable incremental reconstruction. Regist3R leverages an incremental reconstruction paradigm, enabling large-scale 3D reconstructions from unordered and many-view image collections. We evaluate Regist3R on public datasets for camera pose estimation and 3D reconstruction. Our experiments demonstrate that Regist3R achieves comparable performance with optimization-based methods while significantly improving computational efficiency, and outperforms existing multi-view reconstruction models. Furthermore, to assess its performance in real-world applications, we introduce a challenging oblique aerial dataset which has long spatial spans and hundreds of views. The results highlight the effectiveness of Regist3R. We also demonstrate the first attempt to reconstruct large-scale scenes encompassing over thousands of views through pointmap-based foundation models, showcasing its potential for practical applications in large-scale 3D reconstruction tasks, including urban modeling, aerial mapping, and beyond.

Overview of Regist3R: Incremental Registration with Stereo Foundation Model

The paper "Regist3R: Incremental Registration with Stereo Foundation Model" tackles the persistent challenges of multi-view 3D reconstruction, addressing key limitations such as computational inefficiency and cumulative errors in current methods, particularly when scaling to large image sets. Regist3R, the proposed solution, represents a novel approach centered around a stereo foundation model which focuses on efficient and scalable incremental 3D reconstruction.

Key Contributions and Methodology

Regist3R fundamentally changes the approach to multi-view 3D reconstruction by leveraging an incremental reconstruction paradigm. Traditional Structure from Motion (SfM) methods, both global and incremental, face significant challenges: global methods struggle with sparse features and initial geometry reliability, while incremental methods can be prohibitively computationally expensive, often plagued by error propagation. Regist3R circumvents these issues by proposing an inference-only model that facilitates registration without the need for global alignment or exhaustive optimization.

The model architecture integrates a two-stream, transformer-based network which processes images and their associated pointmaps. In its operation, Regist3R autoregressively updates the 3D reconstruction as new images are introduced, effectively building a pointmap body within a unified world coordinate system. This allows the system to maintain consistency across multiple views, while avoiding the computational pitfalls of earlier methods. The training of Regist3R employs a unique auto-regressive strategy that simulates realistic scenarios where ground truth pointmaps might contain inaccuracies, further enhancing the system's robustness against feature noise and drift errors.

For efficient inference, the model employs a minimum spanning tree (MST) strategy to minimize the number of view pairwise comparisons needed during reconstruction. This drastically reduces computational load, achieving highly efficient reconstruction of scenes by requiring only $N-1$ inferences for a dataset of $N$ images. Additionally, Regist3R incorporates a tree compression mechanism to mitigate cumulative errors typical in deeper reconstruction chains.

Experimental Evaluation

The performance of Regist3R has been subjected to rigorous evaluation across several public datasets, including DTU, NRGBD, and 7Scenes, as well as a unique aerial dataset, CS-Drone3D. The results from these benchmark tests underscore the model's efficiency and accuracy, showing that Regist3R can outperform or match traditional optimization-heavy methods like DUSt3R and MASt3R-SfM, while operating with significantly reduced computational complexity. Notably, Regist3R's ability to manage large-scale reconstructions, demonstrated by its application to the challenging CS-Drone3D dataset, showcases its practical applicational strength in dealing with urban modeling and aerial mapping.

Implications and Future Directions

The introduction and success of the Regist3R model in managing complex multi-view 3D reconstructions efficiently suggest several powerful implications for practical applications in computer vision. Specifically, its adoption for urban modeling and aerial mapping highlights its potential for various industrial domains requiring large-scale, quick, and reliable 3D reconstructions, without the overhead of conventional optimization procedures.

Theoretically, the paper sheds light on the potential for future development within the field of 3D reconstruction, particularly with models that are capable of balancing accuracy with scalable efficiency. Regist3R paves the way for exploration into further automation in 3D modeling, possibly combining aspects of both incremental and global frameworks to handle diverse types of image collections more flexibly.

Looking forward, areas for potential improvement include expanding the model to support scenarios with varied intrinsic camera parameters, improving robustness in sparse view setups, and enhancing general applicability across different environments. These advancements would further consolidate the model's efficiency and adaptiveness, reinforcing its role in the evolving landscape of 3D computer vision tasks.