LiDAR2Map: In Defense of LiDAR-Based Semantic Map Construction Using Online Camera Distillation (2304.11379v2)

Abstract: Semantic map construction under bird's-eye view (BEV) plays an essential role in autonomous driving. In contrast to camera image, LiDAR provides the accurate 3D observations to project the captured 3D features onto BEV space inherently. However, the vanilla LiDAR-based BEV feature often contains many indefinite noises, where the spatial features have little texture and semantic cues. In this paper, we propose an effective LiDAR-based method to build semantic map. Specifically, we introduce a BEV feature pyramid decoder that learns the robust multi-scale BEV features for semantic map construction, which greatly boosts the accuracy of the LiDAR-based method. To mitigate the defects caused by lacking semantic cues in LiDAR data, we present an online Camera-to-LiDAR distillation scheme to facilitate the semantic learning from image to point cloud. Our distillation scheme consists of feature-level and logit-level distillation to absorb the semantic information from camera in BEV. The experimental results on challenging nuScenes dataset demonstrate the efficacy of our proposed LiDAR2Map on semantic map construction, which significantly outperforms the previous LiDAR-based methods over 27.9% mIoU and even performs better than the state-of-the-art camera-based approaches. Source code is available at: https://github.com/songw-zju/LiDAR2Map.

• The paper introduces LiDAR2Map, a framework that leverages online camera-to-LiDAR distillation, achieving a 27.9% improvement in mIoU.
• It employs a BEV Feature Pyramid Decoder to refine LiDAR BEV features and mitigate noise for robust multi-scale segmentation.
• Experimental validation on the nuScenes dataset demonstrates its efficiency in advanced vehicle and map segmentation for autonomous driving.

An Expert Overview of "LiDAR2Map: In Defense of LiDAR-Based Semantic Map Construction Using Online Camera Distillation"

The paper "LiDAR2Map: In Defense of LiDAR-Based Semantic Map Construction Using Online Camera Distillation" presents a method for constructing semantic maps in autonomous driving contexts using LiDAR-based techniques augmented by a novel online distillation scheme from camera images. The research addresses the inherent limitations of using LiDAR data alone, particularly the lack of rich semantic information, which is naturally abundant in camera images.

Technical Contributions

The primary contribution of the research is the development of LiDAR2Map, a framework that leverages LiDAR's precise 3D spatial data and enriches it with semantic cues distilled from camera images. The system comprises two main innovations:

1. BEV Feature Pyramid Decoder (BEV-FPD): This component is designed to enhance the multi-scale BEV features extracted from LiDAR data. It addresses the noise issues commonly associated with LiDAR-based BEV features, thereby significantly boosting segmentation accuracy. The BEV-FPD allows for robust multi-scale feature learning, which is critical for accurate semantic map construction.
2. Online Camera-to-LiDAR Distillation Scheme: This scheme consists of both feature-level and logit-level distillation processes that allow the LiDAR-based framework to absorb semantic richness typically found in camera images. The scheme introduces a Position-Guided Feature Fusion Module (PGF$^2$M) that efficiently integrates features from both modalities by encoding spatial relationships, thus facilitating seamless feature fusion.

Experimental Validation

The effectiveness of the proposed framework is validated using the nuScenes dataset, a comprehensive dataset regularly used in autonomous driving research. The paper demonstrates that LiDAR2Map achieves superior performance in both map and vehicle segmentation tasks under various challenging settings. Notably, LiDAR2Map surpasses previous LiDAR-only methods by a significant margin, achieving a 27.9% improvement in mean Intersection-over-Union (mIoU).

Implications and Future Perspectives

LiDAR2Map positions itself as a competitive alternative to camera-based and fusion methods by achieving enhanced semantic map construction performance while relying mainly on LiDAR data for inference. This efficiency is particularly advantageous as it suggests a reduction in the computational and data burdens associated with using high-resolution camera feeds in real-time autonomous systems.

Practically, the method offers a promising avenue for constructing high-definition maps required for advanced navigation and path-planning tasks in autonomous vehicles. Theoretically, it broadens the understanding of multi-modal distillation processes in the context of BEV perception tasks. Looking forward, the ideas presented in this work could be extended to other perception tasks like 3D object detection and motion prediction within BEV frameworks, contributing further to the development of robust autonomous vehicle systems.

Overall, the research contributes a well-structured approach to enhancing LiDAR-based perception systems, thus reinforcing the potential of LiDAR as a core technology in the autonomous driving sector.