MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds (2412.06974v1)

Abstract: Recent sparse multi-view scene reconstruction advances like DUSt3R and MASt3R no longer require camera calibration and camera pose estimation. However, they only process a pair of views at a time to infer pixel-aligned pointmaps. When dealing with more than two views, a combinatorial number of error prone pairwise reconstructions are usually followed by an expensive global optimization, which often fails to rectify the pairwise reconstruction errors. To handle more views, reduce errors, and improve inference time, we propose the fast single-stage feed-forward network MV-DUSt3R. At its core are multi-view decoder blocks which exchange information across any number of views while considering one reference view. To make our method robust to reference view selection, we further propose MV-DUSt3R+, which employs cross-reference-view blocks to fuse information across different reference view choices. To further enable novel view synthesis, we extend both by adding and jointly training Gaussian splatting heads. Experiments on multi-view stereo reconstruction, multi-view pose estimation, and novel view synthesis confirm that our methods improve significantly upon prior art. Code will be released.

• The paper introduces a single-stage network that processes multiple views simultaneously, eliminating pairwise processing and global optimization.
• It enhances performance with cross-reference-view attention blocks, achieving up to 78× faster reconstruction and a 3.2× reduction in Chamfer distance.
• Integration of Gaussian splatting heads enables accurate novel view synthesis, outperforming baseline methods in photometric evaluations.

Overview of MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views

The paper introduces MV-DUSt3R and its enhanced variant MV-DUSt3R+, innovative single-stage networks designed for reconstructing 3D scenes from a sparse set of images without any prior knowledge of camera intrinsics or poses. These networks aim to overcome the limitations of existing methods, such as DUSt3R and MASt3R, by avoiding pairwise processing of views and the subsequent need for global optimization.

Key Contributions

• MV-DUSt3R Network: This network processes multiple views simultaneously in a single feed-forward pass, eliminating the need for pairwise view processing and global optimization, which are typical in existing solutions. It leverages multi-view decoder blocks to learn pairwise relationships among all input views and aligns predictions to a consistent reference camera coordinate system.
• MV-DUSt3R+ Enhancement: Building upon MV-DUSt3R, the MV-DUSt3R+ network introduces cross-reference-view attention blocks, allowing it to select and process multiple reference views. This adaptation improves the robustness and quality of scene reconstructions, especially when dealing with complex scenes with significant inter-view changes.
• Novel View Synthesis (NVS): Both networks are extended for NVS via the integration of Gaussian splatting heads, which predict per-view 3D Gaussian attributes. This allows the models to synthesize new viewpoints with enhanced accuracy.

Experimental Evaluation

The networks were evaluated on multiple datasets, including HM3D, ScanNet, and MP3D, demonstrating significantly improved performance over prior methods:

• Multi-View Stereo (MVS) Reconstruction: MV-DUSt3R demonstrated substantial gains in speed (48× to 78× faster than DUSt3R) while reducing Chamfer distance by up to 3.2× on various dataset evaluations, indicating more precise 3D reconstructions.
• Multi-View Pose Estimation (MVPE): In terms of pose estimation accuracy, MV-DUSt3R+ exhibited a remarkable reduction in mean average error, outperforming DUSt3R across all input configurations.
• Novel View Synthesis: The Gaussian splatting extension enabled more accurate reconstruction of novel views, outperforming baseline approaches in photometric evaluations, attributed to improved predictions of Gaussian parameter locations.

Implications and Future Directions

The findings underscore the advantage of avoiding traditional pairwise view processing and the associated global optimization by utilizing a simultaneous multi-view approach. The MV-DUSt3R+ improves upon this by integrating multiple reference frames, which is particularly beneficial for reconstructing large and complex scenes accurately.

The paper's results suggest potential future developments in AI, particularly in the areas of real-time scene understanding and interactive 3D applications. Practical applications could range from augmented and virtual reality to autonomous systems where rapid and precise environmental mapping is crucial.

In the future, exploration of different neural representations or integration with even larger datasets might further enhance the performance and applicability of these models. Additionally, given the substantial performance of these models in zero-shot scenarios, there is room for investigating their integration with generative models for broader application contexts.

In conclusion, MV-DUSt3R and MV-DUSt3R+ represent a significant stride in efficient and high-quality 3D scene reconstruction, offering a flexible and scalable solution suitable for a variety of complex visual environments.