Deep Depth Completion of a Single RGB-D Image (1803.09326v2)

Abstract: The goal of our work is to complete the depth channel of an RGB-D image. Commodity-grade depth cameras often fail to sense depth for shiny, bright, transparent, and distant surfaces. To address this problem, we train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation. This method was chosen over others (e.g., inpainting depths directly) as the result of extensive experiments with a new depth completion benchmark dataset, where holes are filled in training data through the rendering of surface reconstructions created from multiview RGB-D scans. Experiments with different network inputs, depth representations, loss functions, optimization methods, inpainting methods, and deep depth estimation networks show that our proposed approach provides better depth completions than these alternatives.

• The paper's main contribution is a two-stage method that uses CNN-predicted surface normals and occlusion boundaries to infer missing depth data.
• It leverages only RGB information for training, ensuring sensor-agnostic performance and improved accuracy with lower relative errors.
• A new benchmark dataset of over 105,000 RGB-D images validates that the method outperforms traditional inpainting techniques in challenging conditions.

Deep Depth Completion of a Single RGB-D Image: An Expert Overview

In the field of computer vision, depth completion for RGB-D images poses significant challenges due to the limitations of commodity-level depth cameras which fail to capture accurate depth details in complex scenarios such as shiny, bright, or distant surfaces. The paper "Deep Depth Completion of a Single RGB-D Image" by Yinda Zhang and Thomas Funkhouser addresses these challenges by introducing a novel approach that employs deep learning techniques to predict and complete the missing depth information within RGB-D images captured by standard cameras like the Microsoft Kinect and Intel RealSense.

The primary contribution of this work is the introduction of a two-stage process for depth completion. The proposed method first leverages a convolutional neural network (CNN) to predict surface normals and occlusion boundaries solely from the RGB channels of the input image. Subsequently, these predictions are combined with the raw depth data in a global optimization framework to infer and complete the missing depth for all pixels. This strategy diverges from conventional depth inpainting methods that often use hand-crafted approaches or direct estimation of depth from RGB, which typically struggle with large holes and noisy data.

To validate the effectiveness of this approach, the authors introduce a new benchmark dataset consisting of 105,432 RGB-D images paired with rendered depth completions. These were derived from surface reconstructions obtained from multiview scans in 72 diverse indoor environments. Experimental analysis demonstrates that the proposed method significantly outperforms alternative depth completion techniques, yielding higher accuracy in terms of relative error and RMSE metrics. Specifically, the completed depths achieved smaller relative errors and performed better across various thresholds, indicating robustness and precision even in scenarios where raw depth is scarce.

A notable aspect of this proposed system is its reliance solely on color input for training the prediction network, which ensures that the model's performance is independent of the specific depth sensor used. This generalizes the applicability of their method to different sensors or environments without necessitating retraining. Furthermore, experiments reveal that predicting surface normals provides a more reliable depth estimation framework than directly predicting depth values or derivatives, owing to the localized and orientation-specific nature of normals.

By framing the completion problem as an overview of local geometric predictions with global spatial coherence facilitated by a linear optimization approach, this paper elucidates a flexible and scalable solution for real-world depth completion applications. While the research presents a comprehensive methodology and dataset, the implications extend beyond immediate practical utility. Theoretically, this work raises compelling questions about the interplay between color information and spatial geometry in deep networks, suggesting a promising direction for future exploration in both depth sensing and computer vision at large.

Looking forward, potential developments inspired by this research may involve exploring integrative models that simultaneously leverage not only surface normals and colors but also additional modalities such as texture and materials to further enhance depth completion frameworks. Additionally, further research could focus on refining the global optimization process to incorporate more dynamic contexts or scene-specific constraints, thus pushing the boundaries of depth completion quality and performance even further.