A Survey on Deep Stereo Matching in the Twenties (2407.07816v1)

Abstract: Stereo matching is close to hitting a half-century of history, yet witnessed a rapid evolution in the last decade thanks to deep learning. While previous surveys in the late 2010s covered the first stage of this revolution, the last five years of research brought further ground-breaking advancements to the field. This paper aims to fill this gap in a two-fold manner: first, we offer an in-depth examination of the latest developments in deep stereo matching, focusing on the pioneering architectural designs and groundbreaking paradigms that have redefined the field in the 2020s; second, we present a thorough analysis of the critical challenges that have emerged alongside these advances, providing a comprehensive taxonomy of these issues and exploring the state-of-the-art techniques proposed to address them. By reviewing both the architectural innovations and the key challenges, we offer a holistic view of deep stereo matching and highlight the specific areas that require further investigation. To accompany this survey, we maintain a regularly updated project page that catalogs papers on deep stereo matching in our Awesome-Deep-Stereo-Matching (https://github.com/fabiotosi92/Awesome-Deep-Stereo-Matching) repository.

• The paper systematically categorizes deep stereo matching architectures into five paradigms, emphasizing the effectiveness of iterative optimization methods.
• It demonstrates through benchmark evaluations that models like RAFT-Stereo achieve robust accuracy and efficiency in processing high-resolution images.
• The study addresses key challenges such as domain shift and over-smoothing while outlining promising future directions like end-to-end learning and cross-spectral matching.

A Survey on Deep Stereo Matching in the 2020s

A Survey on Deep Stereo Matching in the 2020s, authored by Fabio Tosi, Luca Bartolomei, and Matteo Poggi, provides a comprehensive examination of the advancements and challenges in deep stereo matching as of 2022. This essay explores the core findings and insights from the paper, highlighting the significant developments, challenges, and implications for future research in the domain.

Overview of Architectural Advancements

The paper organizes recent deep stereo matching architectures into five primary categories: CNN-based cost volume aggregation, Neural Architecture Search (NAS)-based architectures, iterative optimization-based architectures, Vision Transformer (ViT)-based architectures, and Markov Random Field (MRF)-based architectures. These categories represent the diverse approaches explored to improve stereo matching accuracy and efficiency over the past decade.

1. CNN-based Cost Volume Aggregation: Traditional approaches leveraging correlation layers or cost-volume concatenation have been enhanced with innovative techniques such as adaptive patch matching and multi-scale fusion. Examples include AANet, which uses deformable convolutions to mitigate edge-fattening, and CFNet, which introduces a fused and cascade cost volume representation to capture global and structural information.
2. Neural Architecture Search (NAS): NAS has recently been applied to automate the design of efficient stereo matching networks. LEAStereo and EASNet exemplify this category, with LEAStereo automatically optimizing the architecture through hierarchical search spaces and EASNet enabling adaptability to various device constraints.
3. Iterative Optimization-based Architectures: These approaches, inspired by the RAFT architecture, iteratively update disparity estimates using lightweight cost volumes without predefined disparity ranges. RAFT-Stereo, ORStereo, and CREStereo illustrate the effectiveness of this paradigm, providing a balance between accuracy and computational efficiency.
4. Vision Transformer (ViT)-based Architectures: Transformers, which excel at capturing global context via attention mechanisms, have been adapted for stereo matching. Models like STTR and ELFNet leverage these capabilities to enhance disparity estimation by focusing on long-range dependencies and global coherence.
5. Markov Random Field (MRF)-based Architectures: NMRF represents a novel approach combining deep learning with MRFs to enforce spatial coherence. The integration of learned potential functions and neural message passing improves the robustness and accuracy of disparity estimation.

Addressing Key Challenges

Despite the significant advancements in architecture, several challenges persist in deep stereo matching:

1. Domain Shift: Models trained on synthetic data often perform poorly on real-world scenes due to domain discrepancies. Techniques like domain-agnostic feature modeling, cost volume construction, and the integration of geometric cues have been proposed. Notable approaches include DSMNet, which normalizes feature distributions, and GraftNet, which utilizes pre-trained broad-spectrum features.
2. Over-Smoothing: Over-smoothing of depth boundaries remains a critical issue. Methods like SMD-Nets introduce multi-modal distribution modeling to capture both foreground and background disparities accurately.
3. Handling Non-Lambertian Materials: Transparent and reflective surfaces pose substantial challenges. Approaches like Depth4ToM combine monocular and stereo depth estimation to handle these complex materials.
4. Efficiency: Real-time performance on resource-constrained devices is crucial. Efficient architectures like HITNet and MobileStereoNet leverage lightweight cost volumes and hierarchical processing to achieve significant speed-ups.

Experimental Results

The paper also evaluates the performance of various models on benchmark datasets such as KITTI 2015, Middlebury v3, and the Robust Vision Challenge. Iterative optimization-based models, particularly those derived from RAFT-Stereo, consistently outperform others due to their robust generalization capabilities and efficient handling of high-resolution images.

Implications and Future Directions

The advancements and challenges outlined in this paper have several implications for future research:

• Foundational Models: Similar to efforts in single-image depth estimation, developing foundational models for stereo matching could provide robust pre-trained architectures adaptable to various applications.
• Cross-Spectral Matching: Expanding stereo matching to leverage inputs from diverse modalities, such as thermal or multi-spectral cameras, can significantly enhance performance under challenging conditions.
• End-to-End Learning: The continued integration of geometric priors and learned models through end-to-end frameworks can further bridge the gap between classical methods and deep learning.

Conclusion

The survey by Tosi, Bartolomei, and Poggi provides a thorough examination of the state-of-the-art in deep stereo matching as of the 2020s. The evolution of architectures, response to key challenges, and promising results in benchmarks underscore the rapid advancements and ongoing innovation in the field. Future research is poised to build on these developments, pushing the boundaries of what is achievable in stereo vision.

For a comprehensive understanding and thorough investigation of these advancements, readers are encouraged to refer to the full survey paper and accompanying supplementary materials. The authors provide an invaluable resource for both newcomers and seasoned researchers in deep stereo matching.