Robust Bird's Eye View Segmentation by Adapting DINOv2 (2409.10228v1)

Abstract: Extracting a Bird's Eye View (BEV) representation from multiple camera images offers a cost-effective, scalable alternative to LIDAR-based solutions in autonomous driving. However, the performance of the existing BEV methods drops significantly under various corruptions such as brightness and weather changes or camera failures. To improve the robustness of BEV perception, we propose to adapt a large vision foundational model, DINOv2, to BEV estimation using Low Rank Adaptation (LoRA). Our approach builds on the strong representation space of DINOv2 by adapting it to the BEV task in a state-of-the-art framework, SimpleBEV. Our experiments show increased robustness of BEV perception under various corruptions, with increasing gains from scaling up the model and the input resolution. We also showcase the effectiveness of the adapted representations in terms of fewer learnable parameters and faster convergence during training.

• The paper demonstrates that adapting DINOv2 with LoRA in SimpleBEV significantly enhances BEV segmentation robustness under environmental corruptions.
• It shows that the LoRA-adapted DINOv2 model converges faster and requires fewer training iterations while maintaining performance across varying resolutions.
• Experimental results on nuScenes confirm that this approach outperforms traditional methods in handling challenges like motion blur, brightness shifts, and fog.

Robust Bird's Eye View Segmentation by Adapting DINOv2

The paper Robust Bird's Eye View Segmentation by Adapting DINOv2 presents a strategy to improve the robustness of Bird's Eye View (BEV) segmentation for autonomous driving applications by leveraging the DINOv2 visual foundational model. This work primarily addresses the problem of performance degradation under various environmental corruptions by adapting DINOv2 using Low Rank Adaptation (LoRA) within the SimpleBEV framework.

Introduction and Motivation

Autonomous driving systems rely heavily on accurate 3D perception of their surroundings, traditionally achieved using costly and less scalable LIDAR sensors. Camera-based alternatives, specifically those extracting BEV representations from multiple camera images, provide a more scalable solution but are vulnerable to various corruptions, such as changes in brightness, weather, and camera failures. The authors propose enhancing the robustness of BEV perception by integrating DINOv2, a large vision model known for its strong general-purpose features, and adapting it for the BEV task using LoRA.

Methodology

The core of the proposed method is the integration of DINOv2 into the SimpleBEV architecture. The backbone of SimpleBEV, which originally employs ResNet-101 for feature extraction, is replaced by DINOv2. The adaptation is achieved using LoRA, where only a small set of weight matrices within the attention layers of DINOv2 are updated, while the majority of the model remains frozen. This approach reduces the number of learnable parameters significantly, thus enhancing training efficiency and convergence speed.

The analytical focus lies on multiple key aspects:

1. Input Resolution: Evaluation of the models under different image resolutions to assess their robustness and efficiency.
2. Feature Resolution: Comparison of feature map resolutions to understand the preservation of spatial information.
3. Convergence Speed: Analysis of the number of updates required to reach optimal performance.

Experimental Results

The paper presents a comprehensive evaluation on the nuScenes dataset and its corruption benchmark, nuScenes-C. The experiments demonstrate that:

• The DINOv2 adaptation can match or exceed the performance of SimpleBEV even with lower input resolutions and feature resolutions. Notably, ViT-L adaptation achieves a significant performance boost over SimpleBEV at corresponding resolutions.
• The LoRA-adapted DINOv2 model requires fewer training iterations to converge, illustrating the efficiency of incorporating pre-trained large-scale vision models.
• The robustness evaluations highlight that the DINOv2 adaptations outperform SimpleBEV in most corruption scenarios, particularly in handling motion blur, brightness changes, and fog, thus substantiating the model’s increased resilience against real-world environmental variances.

Ablation Studies

Ablation studies reveal critical insights:

• Comparison of different adaptation strategies (frozen, fine-tuning, and LoRA) showed that LoRA provided an optimal balance between parameter efficiency and performance. Specifically, LoRA with DINOv2 significantly reduces learnable parameters while maintaining performance.
• Varying the rank of LoRA indicated that rank 32 offered the best performance, suggesting that there is an optimal complexity level for the adapted weights that captures a balance between robustness and retaining useful pre-trained information.

Implications and Future Directions

The paper signifies the potential of leveraging pre-trained foundational models like DINOv2 in autonomous driving tasks, specifically for robust BEV segmentation. The findings imply that future work could further explore and benchmark other foundational models, such as Stable Diffusion, for BEV and related perception tasks. Additionally, there is room for optimizing adaptation techniques to harness the full capabilities of pre-trained models while minimizing training complexity.

In conclusion, this paper offers a significant contribution by demonstrating that robust and efficient BEV segmentation can be achieved through careful adaptation of large-scale pre-trained vision models using parameter-efficient techniques. The insights gleaned lay the groundwork for future research in enhancing the robustness and scalability of visual perception systems in autonomous vehicles.