Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints (1910.06727v1)

Abstract: Depth completion aims to recover dense depth maps from sparse depth measurements. It is of increasing importance for autonomous driving and draws increasing attention from the vision community. Most of existing methods directly train a network to learn a mapping from sparse depth inputs to dense depth maps, which has difficulties in utilizing the 3D geometric constraints and handling the practical sensor noises. In this paper, to regularize the depth completion and improve the robustness against noise, we propose a unified CNN framework that 1) models the geometric constraints between depth and surface normal in a diffusion module and 2) predicts the confidence of sparse LiDAR measurements to mitigate the impact of noise. Specifically, our encoder-decoder backbone predicts surface normals, coarse depth and confidence of LiDAR inputs simultaneously, which are subsequently inputted into our diffusion refinement module to obtain the final completion results. Extensive experiments on KITTI depth completion dataset and NYU-Depth-V2 dataset demonstrate that our method achieves state-of-the-art performance. Further ablation study and analysis give more insights into the proposed method and demonstrate the generalization capability and stability of our model.

• The paper introduces a CNN framework that leverages depth-normal constraints and anisotropic diffusion to significantly improve depth estimation from sparse LiDAR data.
• It employs an encoder-decoder architecture with a confidence prediction branch to effectively reduce sensor noise and enhance the reliability of depth predictions.
• Experimental evaluations on KITTI and NYU-Depth-V2 datasets demonstrate state-of-the-art performance and strong generalization across outdoor and indoor scenes.

Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints

The paper "Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints" addresses the challenge of generating dense depth maps from sparse LiDAR inputs, a task critical for effective autonomous driving systems. In conventional depth completion methods, the limited use of 3D geometric constraints poses difficulties, particularly when it comes to handling sensor noise inherent in LiDAR data. This paper introduces a novel convolutional neural network (CNN) framework that enhances robustness against noise and effectively utilizes geometric constraints to improve depth completion performance.

Methodology

The proposed framework consists of an encoder-decoder architecture which predicts the surface normals, coarse depth, and confidence of the sparse LiDAR inputs. These predictions are processed through a diffusion refinement module that exploits the geometric relationship between depth and surface normals. The framework's central innovation lies in its anisotropic diffusion model, which operates on a plane-origin distance subspace, assuming that 3D scenes comprise piecewise planar surfaces. This assumption aids in regularizing the depth completion process and taking full advantage of the sparse inputs.

Through a confidence prediction branch, the system estimates the reliability of sparse depth inputs, mitigating the propagation of noise. This allows the network to selectively refine predictions using the diffusion module, guided by the confidence map produced by the encoder-decoder network. The paper emphasizes the efficacy of coupling depth and normal predictions during training, enforcing constraints that enhance depth estimation accuracy.

Experimental Evaluation

The paper validates the proposed method using the KITTI Depth Completion and NYU-Depth-V2 datasets, which represent challenging outdoor and indoor environments, respectively. The results show that this method achieves state-of-the-art performance, demonstrating robustness and capability in handling both scenarios. Notably, testing on the NYU-Depth-V2 dataset reveals the model's excellent generalization from the outdoor to the indoor scenes despite being primarily trained for outdoor applications.

Several metrics, including RMSE, MAE, iRMSE, and iMAE, were used to evaluate model performance. The results indicate superior performance compared to baseline methods and previous state-of-the-art techniques, particularly in challenging conditions where noise is prevalent.

Ablation Study and Analysis

Extensive ablation studies further substantiate the effectiveness of key components, including the impact of the diffusion refinement module and the confidence prediction scheme. The research investigates different configurations for the diffusion module and confirms that the asymmetric conductance function performs better than its alternatives. Additionally, varying the confidence prediction parameter affects performance, highlighting the necessity of carefully balancing model tightness and tolerance to noise.

Implications and Future Directions

The proposed approach presents significant implications for real-time depth estimation in autonomous systems. Its ability to integrate geometric constraints efficiently and handle noise robustly is pivotal in advancing depth completion technology. Future research could explore the application of similar techniques in more dynamic and varied environmental settings, extending to broader applications beyond autonomous driving, such as augmented reality and robotics.

The framework underscores the potential advancements in neural network-based depth completion when geometric properties of the scene are leveraged effectively. This paper also points to the broader application of anisotropic diffusion and confidence prediction in multimedia and computer vision contexts, suggesting fruitful areas for further exploration and development in AI and machine perception.