M^3VSNet: Unsupervised Multi-metric Multi-view Stereo Network (2004.09722v2)

Abstract: The present Multi-view stereo (MVS) methods with supervised learning-based networks have an impressive performance comparing with traditional MVS methods. However, the ground-truth depth maps for training are hard to be obtained and are within limited kinds of scenarios. In this paper, we propose a novel unsupervised multi-metric MVS network, named M^3VSNet, for dense point cloud reconstruction without any supervision. To improve the robustness and completeness of point cloud reconstruction, we propose a novel multi-metric loss function that combines pixel-wise and feature-wise loss function to learn the inherent constraints from different perspectives of matching correspondences. Besides, we also incorporate the normal-depth consistency in the 3D point cloud format to improve the accuracy and continuity of the estimated depth maps. Experimental results show that M3VSNet establishes the state-of-the-arts unsupervised method and achieves comparable performance with previous supervised MVSNet on the DTU dataset and demonstrates the powerful generalization ability on the Tanks and Temples benchmark with effective improvement. Our code is available at https://github.com/whubaichuan/M3VSNet.

• The paper introduces an unsupervised MVS framework that eliminates the need for ground-truth depth maps using a novel multi-metric loss function.
• It leverages pyramid feature aggregation, variance-based cost volume generation, and 3D U-Net regularization to enhance feature extraction and depth estimation.
• Empirical results on DTU and Tanks and Temples benchmarks show competitive reconstruction accuracy and robust generalization without supervised data.

Evaluation of MVSNet and M$^3$VSNet for Multi-view Stereo Reconstruction

Recent advancements in Multi-view Stereo (MVS) technologies have underscored the potential of leveraging deep learning models to enhance 3D dense point cloud reconstruction. The paper "M$^3$VSNet: Unsupervised Multi-metric Multi-view Stereo Network" by Huang et al. introduces an unsupervised learning paradigm aimed at addressing inherent limitations in supervised MVS systems that rely on ground-truth depth maps. This research is pivotal for applications in fields such as augmented reality, virtual reality, and robotics.

The primary contribution of the paper is the development of the M$^3$VSNet framework, which obviates the necessity for labeled training data by employing a novel multi-metric loss function. This function encapsulates both pixel-wise and feature-wise losses to optimize matching correspondences from multiple perspectives. In addition to this, a significant innovation is the incorporation of normal-depth consistency, which enhances depth map accuracy by ensuring orthogonality between local surface tangents and normals.

Methodological Insights

The proposed M$^3$VSNet architecture is articulated into several components: pyramid feature aggregation, variance-based cost volume generation, and 3D U-Net regularization. The pyramid feature aggregation mechanism effectively integrates multi-level contextual information, enhancing the robustness of the extracted features. This allows M$^3$VSNet to surpass the single-scale features used in MVSNet, offering more informative feature maps for constructing cost volumes.

The normal-depth consistency addresses challenges posed by anomalous matching correspondences and continuity errors that proliferate in feature-poor environments. This is accomplished through a post-processing step that refines the initial depth maps, significantly improving their reliability.

Empirical Evaluation

The M$^3$VSNet was rigorously validated on the \textsl{DTU} dataset, where it demonstrated a comparable performance to the supervised MVSNet architecture, with an impressive overall accuracy in dense point cloud reconstruction. By eliminating the requirement for supervised depth maps, M$^3$VSNet establishes a robust benchmark for unsupervised methodologies, displaying superior performance metrics relative to existing unsupervised alternatives like $\mathrm{MVS^2}$ and Unsup_MVS.

Further validation on the \textsl{Tanks and Temples} benchmark, without any fine-tuning on this new dataset, highlighted M$^3$VSNet's generalization capabilities in handling large-scale, complex environments. These results underscore its applicability in real-world scenarios, offering enhanced adaptability across varied datasets and conditions.

Future Implications

The research opens avenues for scaling MVS applications in situations where labeled data is scarce or unavailable. Future work could explore the extension of M$^3$VSNet to incorporate domain adaptation techniques, ensuring robustness across diverse environmental contexts. Moreover, integrating multi-task learning paradigms could enable simultaneous execution of ancillary tasks such as depth completion and scene understanding, further broadening the utility of MVS technologies.

In conclusion, the M$^3$VSNet signifies a substantial leap forward in MVS research by providing an unsupervised alternative that maintains high reconstruction quality. The intelligent combination of multi-metric losses and normal-depth consistency sets a precedent for future endeavors aimed at refining 3D reconstruction frameworks. This innovative approach not only mitigates dependency on exhaustive datasets but also enhances the scalability of MVS applications in both academic and industrial domains.