Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry (2112.08177v2)

Abstract: Multi-view depth estimation methods typically require the computation of a multi-view cost-volume, which leads to huge memory consumption and slow inference. Furthermore, multi-view matching can fail for texture-less surfaces, reflective surfaces and moving objects. For such failure modes, single-view depth estimation methods are often more reliable. To this end, we propose MaGNet, a novel framework for fusing single-view depth probability with multi-view geometry, to improve the accuracy, robustness and efficiency of multi-view depth estimation. For each frame, MaGNet estimates a single-view depth probability distribution, parameterized as a pixel-wise Gaussian. The distribution estimated for the reference frame is then used to sample per-pixel depth candidates. Such probabilistic sampling enables the network to achieve higher accuracy while evaluating fewer depth candidates. We also propose depth consistency weighting for the multi-view matching score, to ensure that the multi-view depth is consistent with the single-view predictions. The proposed method achieves state-of-the-art performance on ScanNet, 7-Scenes and KITTI. Qualitative evaluation demonstrates that our method is more robust against challenging artifacts such as texture-less/reflective surfaces and moving objects. Our code and model weights are available at https://github.com/baegwangbin/MaGNet.

Overview of Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

The paper presented, authored by Gwangbin Bae et al. from the University of Cambridge, introduces MaGNet, a novel framework for multi-view depth estimation that fuses single-view depth probability with traditional multi-view geometry. The paper addresses critical challenges inherent to multi-view depth estimation methods, including high memory and computational costs and failure modes related to texture-less surfaces, reflective surfaces, and moving objects. By incorporating single-view depth probabilities, MaGNet aims to enhance the accuracy, robustness, and efficiency of depth estimation.

Key Contributions

1. Probabilistic Depth Sampling: Unlike traditional methods that evaluate a fixed, uniform set of depth candidates, MaGNet samples depth candidates probabilistically, based on the single-view depth probability distribution. This results in fewer evaluated candidates but with higher accuracy, requiring only 5 candidates compared to 64 in methods like DPSNet while maintaining a 92% thinner cost-volume.
2. Depth Consistency Weighting: The proposed method incorporates a depth consistency weighting mechanism to improve multi-view matching. This ensures that depth candidates are consistent with single-view predictions, thus improving robustness, particularly in challenging situations such as scenes with texture-less or reflective surfaces.
3. Iterative Refinement: To handle initial inaccuracies in single-view depth predictions, MaGNet implements iterative refinement, updating depth distributions over multiple passes to refine accuracy and reduce uncertainty. The iterative approach allows for the consideration of wider depths if initial predictions show high variance, enhancing resilience against errors.

Experimental Results and Analysis

MaGNet demonstrates state-of-the-art performance on several established benchmarks including ScanNet, 7-Scenes, and KITTI datasets. The method surpasses existing techniques by effectively balancing computational efficiency and accuracy. Notably, MaGNet's ability to outperform other methods on cross-dataset evaluations suggests superior generalization abilities, likely attributed to its smaller, focused depth search space and single-view geometric reasoning.

Implications and Future Directions

The integration of single-view and multi-view depth estimation paradigms offers significant advances in efficiently handling depth inference under challenging visual conditions. This approach not only enhances real-time applications due to reduced computational overhead but also addresses accuracy concerns where stereo geometric cues might fail. Future iterations may explore extending this methodology's applicability to broader computer vision tasks such as real-time 3D mapping and augmented reality, especially in dynamic environments or varying lighting conditions.

By minimizing dependency on large-scale multi-view computational resources and enhancing predictive reliability across mixed-content scenes, MaGNet contributes a solid foundation for future explorations into hybrid depth estimation frameworks that leverage both probabilistic single-view insights and geometric depth cues. Researchers may further investigate integrating additional probabilistic metrics or cross-domain adaptation techniques to bolster the framework's robustness and versatility.