Learning-based Multi-View Stereo: A Survey (2408.15235v2)

Abstract: 3D reconstruction aims to recover the dense 3D structure of a scene. It plays an essential role in various applications such as Augmented/Virtual Reality (AR/VR), autonomous driving and robotics. Leveraging multiple views of a scene captured from different viewpoints, Multi-View Stereo (MVS) algorithms synthesize a comprehensive 3D representation, enabling precise reconstruction in complex environments. Due to its efficiency and effectiveness, MVS has become a pivotal method for image-based 3D reconstruction. Recently, with the success of deep learning, many learning-based MVS methods have been proposed, achieving impressive performance against traditional methods. We categorize these learning-based methods as: depth map-based, voxel-based, NeRF-based, 3D Gaussian Splatting-based, and large feed-forward methods. Among these, we focus significantly on depth map-based methods, which are the main family of MVS due to their conciseness, flexibility and scalability. In this survey, we provide a comprehensive review of the literature at the time of this writing. We investigate these learning-based methods, summarize their performances on popular benchmarks, and discuss promising future research directions in this area.

• The paper provides a systematic review and taxonomy of learning-based MVS approaches for reconstructing dense 3D geometry.
• It examines diverse methodologies, including depth map estimation, voxel grids, NeRF adaptations, and transformer-based models.
• The survey highlights benchmarks, challenges, and future directions that are crucial for advancing AR/VR, robotics, and autonomous navigation.

Learning-based Multi-View Stereo: A Survey

The paper "Learning-based Multi-View Stereo: A Survey" by Fangjinhua Wang \textit{et al.} encompasses a comprehensive review and a systematic analysis of more recent learning-based approaches for Multi-View Stereo (MVS) methodologies. Multi-View Stereo aims to reconstruct dense 3D geometry from multiple images captured at different viewpoints and has been pivotal in various fields, including AR/VR, robotics, and autonomous driving.

Taxonomy and Categorization

The authors subdivide learning-based MVS methods into several categories, focusing mainly on:

• Depth map-based methods: The primary focus area due to their relative simplicity and scalability.
• Voxel-based methods: Discretize 3D space into voxels and estimate geometry directly, offering high accuracy but at the cost of memory consumption.
• NeRF-based methods: Adapt neural radiance fields for surface extraction.
• 3D Gaussian Splatting-based methods: Utilize 3D Gaussians for novel view synthesis and efficient rendering.
• Large feed-forward methods: Leverage large-scale transformer models for direct 3D representation learning.

Each category is analyzed based on methodological pipeline, performance, and practicality metrics.

Depth Map-based Methods

Depth map-based methods remain the mainstay due to their decoupling of complex 3D reconstruction into more manageable tasks of depth estimation and fusion:

1. Feature Extraction: Using CNNs (e.g., ResNet, FPN) and Transformers to extract deep features from images.
2. Cost Volume Construction: Plane sweep algorithms construct 3D or 4D cost volumes. Statistical metrics or variance-based strategies are used for feature matching.
3. Cost Volume Regularization: Strategies including 3D CNN and RNN regularizations refine cost volumes to aggregate context information effectively.
4. Depth Estimation and Refinement: Employ softargmax or argmax operations typically for final depth map prediction while auxiliary refinement steps improve precision further.

Online methods focus on real-time applications and utilize lightweight architectures. Conversely, offline methods optimize for high-quality outputs and integrate more computationally intensive processes.

Other Learning-based Methods

The survey explores other expressive families of learning-based MVS:

• Voxel-based Methods like Atlas and NeuralRecon leverage TSDF and volumetric representations.
• NeRF-based Methods build upon neural radiance fields to model and render volumetric data.
• 3D Gaussian Splatting-based Approaches such as 3DGS which encode the scene explicitly with 3D Gaussians facilitating efficient rasterization.
• Large feed-forward Methods utilize large transformers to capture object-level semantics and geometry directly from image inputs.

Benchmarks and Evaluation Metrics

The paper compares depth map-based methods extensively on standard benchmarks like DTU, Tanks and Temples, and ETH3D databases. The evaluation focuses significantly on 2D and 3D metrics:

• 2D Metrics: Typically assess depth maps based on mean absolute error, inlier ratios, etc.
• 3D Metrics: Measure point cloud alignments using precision, recall, and F1 scores, often after alignment using iterative closest point (ICP) algorithms.

Challenges and Directions for Future Research

The survey identifies several key challenges and proposes future research directions:

• Dataset and Benchmarks: The requirement for high-quality, diverse, and large-scale datasets to train and evaluate learning-based MVS methods.
• View Selection: Strategies to select optimal views to balance computational efficiency and reconstruction accuracy.
• Depth Fusion: Employing advanced methods for robust depth map fusion leveraging deep learning.
• Feature Enhancement: Exploring the potential of advanced feature extractors like Vision Transformers.
• Efficiency: Critical for applications demanding low-latency outputs, the focus on pruning, quantization, and distillation methods to develop efficient models.
• Prior Assistance: Incorporating priors such as surface normals, geometric constraints, or semantic information to improve MVS algorithms, especially in textureless and challenging regions.

Conclusion

The review comprehensively covers methodologies, results, and insights into learning-based MVS, focusing prominently on depth map-based methods. Depth map methodologies achieve a favorable balance between performance and computational feasibility, whereas emerging techniques like NeRF-based methods show promise in handling more complex real-world scenarios. This survey is a valuable resource to guide future innovations and improvements in learning-based MVS research.