Noise-Aware Unsupervised Deep Lidar-Stereo Fusion (1904.03868v1)

Abstract: In this paper, we present LidarStereoNet, the first unsupervised Lidar-stereo fusion network, which can be trained in an end-to-end manner without the need of ground truth depth maps. By introducing a novel "Feedback Loop'' to connect the network input with output, LidarStereoNet could tackle both noisy Lidar points and misalignment between sensors that have been ignored in existing Lidar-stereo fusion studies. Besides, we propose to incorporate a piecewise planar model into network learning to further constrain depths to conform to the underlying 3D geometry. Extensive quantitative and qualitative evaluations on both real and synthetic datasets demonstrate the superiority of our method, which outperforms state-of-the-art stereo matching, depth completion and Lidar-Stereo fusion approaches significantly.

• The paper introduces LidarStereoNet, a novel unsupervised deep network that fuses Lidar and stereo data for 3D perception, using a feedback loop to clean noisy Lidar points automatically.
• Numerical results on the KITTI dataset show LidarStereoNet significantly outperforms existing methods, achieving over 50% performance improvement for 3D perception tasks.
• The work has significant implications for applications like autonomous driving by eliminating the need for ground truth depth maps for training and enhancing adaptability.

Noise-Aware Unsupervised Deep Lidar-Stereo Fusion

The paper "Noise-Aware Unsupervised Deep Lidar-Stereo Fusion" presents an innovative approach to enhance 3D perception using Lidar-Stereo fusion without requiring ground truth depth maps. The authors introduce LidarStereoNet, a fusion network trained end-to-end in an unsupervised manner, which stands out by addressing the common issues faced by Lidar and stereo sensors — noise in Lidar data, and misalignment between sensors. Most notable is the introduction of a novel "Feedback Loop" mechanism which is pivotal in automatically cleaning erroneous Lidar measurements for improved fusion fidelity.

Methodology

1. Feedback Loop Design: The feedback loop is a central element of LidarStereoNet. By connecting network outputs to inputs, it effectively identifies and removes noisy Lidar points during training. This process meticulously calibrates inputs, enhancing stereo matching accuracy.
2. Loss Functions: The method employs several loss functions — image warping loss, Lidar loss, smoothness loss, and the novel plane fitting loss, which are crucial for unsupervised learning given the absence of ground truth data. The plane fitting loss enforces a geometric constraint by modeling disparities within segments as slanting planes, enhancing structural representation.
3. Core Architecture: The network features separate layers for extracting features from dense stereo images and sparse Lidar inputs, followed by a fusion step. The architecture includes a feature-matching block using a stack-hourglass structure and a disparity computing layer using the soft-argmin operation.

Numerical Results

This work reports compelling numerical performance improvements. On the KITTI dataset, LidarStereoNet outperforms existing methods for Lidar-Stereo fusion, as well as stereo matching and depth completion approaches. Notably, when Lidar data was completely missing, the network still achieved state-of-the-art results, showcasing its robustness. The method exhibited superiority with a significant margin, reporting over 50% improvement in performance compared to previous methods.

Implications and Future Work

The implications of this work are considerable, particularly in application domains reliant on precise environmental perception, such as autonomous driving. The elimination of dependency on ground truth depth maps for training enhances adaptability and reduces the overhead associated with data acquisition. Future work could explore the extension of feedback loop principles to other sensor fusion tasks, investigate unsupervised strategies across broader modalities, or enhance the network's ability to generalize across diverse environments.

Furthermore, additional research into optimizing computational efficiency could make such models more suitable for deployment in real-time systems, considering the 0.5 fps processing time recorded in this paper. This research could significantly influence advancements in unsupervised learning paradigms and the practical application of AI in complex environments.

In conclusion, the paper delivers strong theoretical and practical innovations in Lidar-Stereo fusion, propelling the capabilities of automated 3D perception toward smarter, more efficient systems ready to tackle real-world challenges.