UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View (2303.15083v1)

Abstract: In the field of 3D object detection for autonomous driving, the sensor portfolio including multi-modality and single-modality is diverse and complex. Since the multi-modal methods have system complexity while the accuracy of single-modal ones is relatively low, how to make a tradeoff between them is difficult. In this work, we propose a universal cross-modality knowledge distillation framework (UniDistill) to improve the performance of single-modality detectors. Specifically, during training, UniDistill projects the features of both the teacher and the student detector into Bird's-Eye-View (BEV), which is a friendly representation for different modalities. Then, three distillation losses are calculated to sparsely align the foreground features, helping the student learn from the teacher without introducing additional cost during inference. Taking advantage of the similar detection paradigm of different detectors in BEV, UniDistill easily supports LiDAR-to-camera, camera-to-LiDAR, fusion-to-LiDAR and fusion-to-camera distillation paths. Furthermore, the three distillation losses can filter the effect of misaligned background information and balance between objects of different sizes, improving the distillation effectiveness. Extensive experiments on nuScenes demonstrate that UniDistill effectively improves the mAP and NDS of student detectors by 2.0%~3.2%.

UniDistill: Advancements in Cross-Modality Knowledge Distillation for 3D Object Detection

In "UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye View," the authors propose an innovative approach to enhance single-modality 3D object detectors by leveraging a universal framework for cross-modality knowledge distillation (KD). The paper addresses the intricacies of multi-modality versus single-modality detectors, highlighting the balance between the system complexity inherent in multi-modal methods and the relatively lower accuracy of single-modal counterparts. This work presents a significant step in optimizing the trade-off between these options in the autonomous driving sector.

Overview of Methodology

UniDistill capitalizes on the commonality across detection paradigms in various modalities within the Bird's-Eye View (BEV) space. During the training phase, the framework projects features from both teacher and student detectors into the BEV, applying three distinct distillation losses to align the foreground features effectively. These modalities include LiDAR-to-camera, camera-to-LiDAR, fusion-to-LiDAR, and fusion-to-camera paths, ensuring broad applicability.

1. Feature Distillation: This loss function transfers semantic knowledge by aligning features at nine critical points within each foreground bounding box, eschewing global alignment to better focus on relevant object data.
2. Relation Distillation: High-level structural knowledge is transferred by aligning relationships between key data points, computed as cosine similarities, across both the teacher and student arrays.
3. Response Distillation: This loss mitigates the prediction gap by aligning the response features in a Gaussian mask, emphasizing feature alignment in proximity to foreground objects.

Experimental Evaluation

The UniDistill framework is rigorously tested on the nuScenes dataset, using various detector configurations, including BEVDet backed by ResNet-50 for camera-based detection and CenterPoint for LiDAR-based models. Notably, the LiDAR-camera combination, which employs post-fusion processes to leverage both modalities, presents noteworthy improvements. The LiDAR-only detectors' mean Average Precision (mAP) and nuScenes Detection Score (NDS) metrics show enhancements in the 2.0% to 3.2% range through the application of UniDistill.

Implications and Future Directions

The implications of this framework are significant, particularly in reducing computational overhead at inference time while maintaining competitive accuracy rates. The introduction of distillation losses that are mindful of semantic content and size variability among detected objects speaks to the method's nuanced approach.

Looking forward, the paper suggests potential efficiencies in deploying block-wise distillation techniques, hinting at faster training times and reduced memory burdens. This direction holds promise for the broader field of AI and machine learning, signifying advancements in efficient model training and potentially impacting related domains in AI requiring complex sensor integration and data fusion.

Conclusion

"UniDistill" stands as a robust framework, addressing both the flexibility and efficiency challenges in cross-modality knowledge distillation. Its potential to enhance performance in single-modality detectors without increasing operational costs marks a meaningful contribution to the deployment of 3D object detection models in real-world autonomous driving platforms.