VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

Abstract: The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-LLMs (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-LLMs with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.

• The paper introduces a plug-and-play cross-view 3D geometric enabler (CVGE) that fuses rich 3D features into a vision-language model.
• The model achieves significant improvements in trajectory planning, motion prediction, and risk perception across autonomous driving benchmarks.
• It leverages a hierarchical adaptive injection mechanism to deeply integrate spatial information within the model’s layers, enhancing navigation tasks.

VGGDrive: Empowering Vision-LLMs with Cross-View Geometric Grounding for Autonomous Driving

Introduction

The paper introduces VGGDrive, a novel vision-language framework specifically designed for autonomous driving. The vision-LLM (VLM) is augmented with cross-view 3D geometric capabilities through the use of a plug-and-play Cross-View 3D Geometric Enabler (CVGE). This advancement addresses a fundamental limitation of existing VLMs, namely, their inability to comprehend 3D spatial information essential for navigation tasks like trajectory planning, motion prediction, and cross-view risk perception.

Figure 1: Existing relevant paradigms vs. our VGGDrive. (a) The VLA paradigm for trajectory planning. (b) Two existing paradigms for integrating 3D foundation models (VGGT) with VLMs. (c) Our VGGDrive leverages the VGGT model to empower the basic VLM with cross-view geometric grounding capabilities.

Core Contributions

Cross-View Geometric Grounding

The VGGDrive architecture integrates a frozen visual 3D foundation model (VGGT) into VLMs through the CVGE. This component decouples the base VLM architecture and implements a hierarchical adaptive injection mechanism, which injects 3D geometric features deep into the model's layers. This process enriches the VLM with spatial perception proficiency required for complex driving environments.

Figure 2: Quantitative comparison of VGGDrive with specific state-of-the-art methods across autonomous driving benchmarks.

Enhanced Autonomy in Driving

VGGDrive showcases significant improvements in autonomous driving capabilities. It consistently outperforms state-of-the-art methods across several benchmarks by equipping VLMs with essential 3D features. Tasks including cross-view risk perception, motion prediction, and trajectory planning benefit from this comprehensive geometric understanding, providing enhanced scene interpretation and situational awareness.

Figure 3: Overview of VGGDrive. The frozen visual 3D foundation model extracts geometrically consistent 3D features, while the CVGE facilitates integration with 2D visual representations in the VLM.

Methodology

Base VLM and Integration

VGGDrive utilizes the Qwen2.5-VL-7B model as its core, which processes visual and textual inputs through a unified transformer decoder. The CVGE then bridges 3D features with 2D visual embeddings via a hierarchical adaptive approach, ensuring robust incorporation of spatial data into the model.

Hierarchical Adaptive Injection

The CVGE's adaptive injection mechanism enables cross-layer enhancement of the VLM by exploiting the VGGT's 3D geometric insights. The mechanism extracts and enriches hidden states with 3D features, supporting deep model architecture engagement and geometric foundation establishment within VLMs.

Experimental Evaluation

The experimental results demonstrate VGGDrive’s superior performance across a range of autonomous driving benchmarks. Notably, VGGDrive significantly advances past limitations associated with spatial perception in VLMs, illustrating improvements in real-world driving task efficiency.

Figure 4: Visualization of VGGDrive's performance across various autonomous driving attribute evaluation tasks.

Future Directions

VGGDrive's integration of 3D foundation models with VLMs opens new vistas for research in autonomous systems, especially those requiring robust spatial understanding. Future work could explore scalable deployment in diverse environments, extending foundational capabilities to leverage real-time dynamic scene reconstruction while emphasizing efficiency and computational resource management.

Conclusion

The presented work, VGGDrive, represents a significant step in integrating 3D geometric capabilities within vision-LLMs for autonomous driving. By addressing inherent limitations of existing models, VGGDrive establishes a new technical pathway, enhancing both the theoretical and practical aspects of autonomous navigation tasks.

Figure 5: Qualitative results on the closed-loop trajectory planning task showing left-turn, right-turn, and straight-ahead scenarios highlighting VGGDrive's competence in complex driving conditions.