Guided Depth Map Super-resolution: A Survey (2302.09598v2)

Abstract: Guided depth map super-resolution (GDSR), which aims to reconstruct a high-resolution (HR) depth map from a low-resolution (LR) observation with the help of a paired HR color image, is a longstanding and fundamental problem, it has attracted considerable attention from computer vision and image processing communities. A myriad of novel and effective approaches have been proposed recently, especially with powerful deep learning techniques. This survey is an effort to present a comprehensive survey of recent progress in GDSR. We start by summarizing the problem of GDSR and explaining why it is challenging. Next, we introduce some commonly used datasets and image quality assessment methods. In addition, we roughly classify existing GDSR methods into three categories, i.e., filtering-based methods, prior-based methods, and learning-based methods. In each category, we introduce the general description of the published algorithms and design principles, summarize the representative methods, and discuss their highlights and limitations. Moreover, the depth related applications are introduced. Furthermore, we conduct experiments to evaluate the performance of some representative methods based on unified experimental configurations, so as to offer a systematic and fair performance evaluation to readers. Finally, we conclude this survey with possible directions and open problems for further research. All the related materials can be found at \url{https://github.com/zhwzhong/Guided-Depth-Map-Super-resolution-A-Survey}.

• The paper surveys guided depth map super-resolution techniques by categorizing methods into filtering-based, optimization-based, and learning-based approaches.
• It details methodologies including bilateral and guided filters, graph-based regularization, and multi-scale deep architectures for improved depth reconstruction.
• Experimental evaluations show that advanced learning-based methods yield superior performance, benefiting applications like 3D reconstruction, autonomous driving, and augmented reality.

Overview of "Guided Depth Map Super-resolution: A Survey"

The paper presents a thorough survey of Guided Depth Map Super-resolution (GDSR) approaches. GDSR aims to reconstruct high-resolution (HR) depth maps from low-resolution (LR) observations, assisted by paired HR color images. This is a pivotal problem in computer vision and image processing, notably influential for applications like 3D reconstruction, semantic understanding, and autonomous driving. The manuscript categorizes existing GDSR methods into filtering-based, optimization-based, and learning-based approaches and systematically evaluates their developments, strengths, and limitations.

Filtering-based Methods

Filtering-based techniques generally operate by refining the depth of pixels through weighted averages of neighboring pixels. This segment includes various adaptations of bilateral filters, non-local means filters, guided filters, and dynamic filters.

1. Bilateral Filters and Its Variants: These employ spatial and range filters to smooth images while maintaining edges. Joint bilateral upsampling (JBU) extends this idea by using the HR color image to influence the range filter, although it is susceptible to artifacts from inconsistency in edge information between depth and color images.
2. Non-Local Means Filters: This class calculates pixel affinities based on the similarity of image patches rather than single pixels, aiming to better capture detailed textures.
3. Guided Filters: Proposed to improve upon bilateral filters, guided filters apply a linear model to generate the filtered depth map, introducing computational efficiency and reduced gradient reversal artifacts.
4. Dynamic Filters: Address the static nature of traditional filters by dynamically adjusting based on local structures or guidance images, aiming to better handle complex structures and reduce artifacts.

Optimization-based Methods

Optimization-based methods aim to solve for a depth map that minimizes a combination of data fidelity and regularization terms. These approaches incorporate various priors to manage the ill-posed nature of GDSR.

1. Markov Random Fields (MRF): MRF-based techniques utilize undirected graphical models to impose smoothness and consistency constraints across pixels, while adaptations attempt to manage texture-copying issues.
2. Auto-regressive Models: These models aim to represent depth signals as linear combinations of neighboring signals, tailored adaptively by evaluating color-intensity gradients.
3. Total Variation: Regularization using Total Generalized Variation (TGV) and related approaches is leveraged to preserve edges while addressing the staircasing artifacts commonly observed in TV regularization.
4. Graph Laplacians: These methods utilize graph-based regularization to promote intra-image smoothness and inter-image structural consistency, reflecting spectral characteristics of depth structures.

Learning-based Methods

Recent advances in learning-based methods include dictionary learning and deep learning, which leverage large datasets and advanced models for enhanced performance.

1. Dictionary Learning: Methods like joint intensity-depth sparse representations and multi-modal dictionary learning capture cross-modal interdependencies through basis expansions.
2. Deep Learning: These methods include architectures designed to utilize multi-scale feature extraction, attention mechanisms, and feature fusions, significantly advancing state-of-the-art performance. The paper surveys various deep, learned approaches such as multi-task learning frameworks, attention-based fusion models, and GAN-based frameworks.

Experimental Evaluation

The paper also provides a comprehensive experimental evaluation of representative GDSR methods across multiple benchmark datasets. The experiments measure performance using standard depth quality metrics such as RMSE and demonstrate the efficacy of newer learning-based methods.

Implications and Future Directions

The implications of GDSR research are substantial for downstream applications like robotics, augmented reality, and self-driving cars, where HR depth information is crucial. Future research is encouraged to explore lightweight models suitable for mobile devices, develop robust evaluation metrics sensitive to perceptual quality, and integrate GDSR with high-level visual tasks for further improvements. Additionally, there is a guiding interest in investigating weakly-supervised or unsupervised frameworks to address the lack of large, high-quality paired datasets in real-world scenarios.

In conclusion, by providing a thorough overview and classification of current methodologies, this survey lays a groundwork for understanding the intricate methodologies of GDSR and sets a trajectory for future explorations in this vibrant research area.