BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection (2211.09386v1)

Abstract: 3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding. Owing to its low cost and high efficiency, multi-view 3D object detection has demonstrated promising application prospects. However, accurately detecting objects through perspective views is extremely difficult due to the lack of depth information. Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment. Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise localization. In this paper, we explore the incorporation of LiDAR-based detectors for multi-view 3D object detection. Instead of directly training a depth prediction network, we unify the image and LiDAR features in the Bird-Eye-View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm. To this end, we propose \textbf{BEVDistill}, a cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection. Extensive experiments demonstrate that the proposed method outperforms current KD approaches on a highly-competitive baseline, BEVFormer, without introducing any extra cost in the inference phase. Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-art in comparison with various image-based detectors. Code will be available at https://github.com/zehuichen123/BEVDistill.

Cross-Modal BEV Knowledge Distillation for Multi-View 3D Object Detection

The paper presents BEVDistill, a novel approach for enhancing multi-view 3D object detection by leveraging cross-modal knowledge distillation (KD) from LiDAR detectors. It addresses the challenge posed by 3D object detection using images, which lack intrinsic depth information, by integrating LiDAR insights that provide robust spatial cues. This is accomplished without incurring additional computational costs during the inference phase, thereby maintaining the efficiency necessary for practical applications such as autonomous driving.

Methodology Overview

The BEVDistill framework integrates information from image- and LiDAR-based modalities in the Bird-Eye-View (BEV) space, effectively allowing feature alignment and knowledge transfer despite the inherent differences between the two modalities. This is operationalized through a teacher-student paradigm wherein a LiDAR-based detector acts as the teacher, imparting knowledge to the image-based detector, the student.

The methodology employed is two-pronged, incorporating both dense and sparse supervision mechanisms. The dense feature distillation focuses on non-homogenous feature alignment by projecting both 2D and 3D features into the BEV space, facilitating mutual information transfer while maintaining each modality's intrinsic structure. Significantly, the approach utilizes a foreground-guided distillation strategy, focusing on regions of interest that contribute most to the detection task.

The sparse instance distillation addresses cross-modal divergence by adopting a quality-score mechanism that prioritizes more credible teacher predictions. The distillation of this knowledge through maximizing mutual information between teacher and student penultimate layer representations furthers the effectiveness of knowledge transfer.

Experimental Results

The framework was evaluated extensively on the nuScenes dataset, demonstrating significant improvements over the baseline BEVFormer model. Specifically, BEVDistill achieved a notable 59.4 NDS on the nuScenes test leaderboard, showcasing its superior performance against existing multi-view 3D detection methods.

Experiments confirmed the enhanced model ability in localizations, such as translation and orientation accuracy, due to the distillation of spatial cues from the LiDAR teacher model. This points to the efficacy of BEVDistill in addressing the sensor gap between image and LiDAR data.

Implications and Future Work

The BEVDistill framework extends the capabilities of multi-view 3D object detectors, enhancing their practicality in real-world scenarios by integrating the spatial accuracy of LiDAR with the efficiency and cost-effectiveness of image-based methods. This approach opens pathways for further research in cross-modal knowledge transfer techniques, potentially extending to other sensor modalities and application domains.

Future work could focus on exploring adaptive and real-time KD mechanisms and investigating the scalability of BEVDistill to larger and more diverse datasets, further establishing its utility in autonomous systems and other technologically advanced fields.