Gated2Depth: Real-time Dense Lidar from Gated Images (1902.04997v3)

Abstract: We present an imaging framework which converts three images from a gated camera into high-resolution depth maps with depth accuracy comparable to pulsed lidar measurements. Existing scanning lidar systems achieve low spatial resolution at large ranges due to mechanically-limited angular sampling rates, restricting scene understanding tasks to close-range clusters with dense sampling. Moreover, today's pulsed lidar scanners suffer from high cost, power consumption, large form-factors, and they fail in the presence of strong backscatter. We depart from point scanning and demonstrate that it is possible to turn a low-cost CMOS gated imager into a dense depth camera with at least 80m range - by learning depth from three gated images. The proposed architecture exploits semantic context across gated slices, and is trained on a synthetic discriminator loss without the need of dense depth labels. The proposed replacement for scanning lidar systems is real-time, handles back-scatter and provides dense depth at long ranges. We validate our approach in simulation and on real-world data acquired over 4,000km driving in northern Europe. Data and code are available at https://github.com/gruberto/Gated2Depth.

• The paper proposes the Gated2Depth framework, which uses a learning-based method with a convolutional neural network on three gated image slices to estimate dense depth maps.
• Extensive validation shows the Gated2Depth framework achieves real-time dense depth estimation comparable to traditional scanning lidar systems at ranges up to 80 meters.
• This framework offers a lower-cost, power-efficient, and more weather-resilient alternative to traditional lidar for applications like autonomous navigation.

Overview of "Gated2Depth: Real-Time Dense Lidar From Gated Images"

The paper "Gated2Depth: Real-Time Dense Lidar From Gated Images" proposes a novel imaging framework that transforms images obtained from a CMOS-based gated camera into dense depth maps. The depth accuracy of these maps is on par with conventional pulsed lidar systems, which are the current standard for spatial measurement in various autonomous and remote sensing applications.

Key Contributions and Methodology

The primary contribution of this research is the Gated2Depth framework, which challenges traditional lidar systems that depend on mechanical scanning to gather spatial data, resulting in low spatial resolution at larger distances. The paper addresses the shortcomings of pulsed lidar, such as high cost, power consumption, large form-factor, and susceptibility to backscatter. By utilizing a gated camera—characterized by time-gated exposure based on pulsed illumination—the proposed system captures dense depth information by learning from three sequentially captured gated slices.

Key methodological innovations include:

• Image Formation and Depth Estimation Model: The paper introduces an image formation model that uses a learning-based approach to estimate dense depth. This approach doesn't require dense depth labels, relying instead on a synthetic discriminator loss.
• Architecture Design: A convolutional neural network (CNN) architecture is employed to exploit semantic contexts across the gated slices. This architecture is trained to yield depth maps that provide detailed understanding at long ranges.
• Training Regimen: The framework leverages a discriminator pre-trained on synthetic data to transfer dense depth estimation to real-world data. This method allows dense mapping without densely annotated data.
• Extensive Validation: The method is validated using both synthetic data and real-world data collected across over 4,000 km of driving in diverse weather and lighting conditions in northern Europe.

Numerical Results

The authors assert that their framework achieves a range of at least 80 meters. In simulation and experimental validation, the Gated2Depth framework exhibits depth estimation accuracy comparable to traditional scanning lidar systems but offers a significant advantage in terms of resolution and real-time processing capability. This is crucial for applications requiring high-density spatial data for scene understanding, such as autonomous vehicle navigation and robotics.

Implications and Future Work

The practical implications of this research are substantial. By reducing the reliance on expensive and resource-heavy traditional lidar systems, the Gated2Depth framework opens the door for more widespread adoption of advanced depth sensing in cost-sensitive markets. The method's resilience against adverse weather conditions such as fog, snow, and strong ambient light positions it as an invaluable tool in environments challenging for traditional lidar.

From a theoretical standpoint, this work underscores the potential of gated imagers, combined with neural network architectures, to produce reliable and dense depth information across various applications. The research invites further exploration into how similar techniques could be applied to other domains requiring accurate depth perception with minimal data acquisition overhead.

Future research directions could focus on enhancing the integration of Gated2Depth with other sensing modalities, like RGB imaging, to further enhance context and reduce any remaining ambiguities inherent to gated imaging. Additionally, extending the range beyond the stated 80 meters and improving performance under varying intensities of ambient light could further bolster the framework’s applicability.

In summary, the Gated2Depth introduces a compelling alternative to traditional lidar by leveraging underutilized sensor technology through advanced machine learning models capable of delivering detailed, real-time depth estimation suitable for dynamic real-world environments.