VGGDrive: 3D Geometric Fusion for Autonomous Driving

• VGGDrive is an architectural framework that integrates cross-view 3D geometric cues into vision-language models for enhanced autonomous driving performance.
• It employs a frozen 3D foundation model, a base VLM, and a plug-and-play CVGE module to fuse 2D and 3D features progressively at each decoder layer.
• Empirical evaluations on multiple benchmarks demonstrate significant improvements in risk perception, object matching, and trajectory planning over conventional models.

VGGDrive is an architectural framework for integrating cross-view 3D geometric understanding into Vision-LLMs (VLMs) for autonomous driving. It addresses the key limitation of existing VLMs, which is their inability to leverage 3D geometric cues—essential for robust perception, motion prediction, and planning in complex driving scenarios. VGGDrive achieves this by hierarchically injecting features from a frozen 3D foundation model into a VLM backbone through a specialized Cross-View 3D Geometric Enabler (CVGE), resulting in substantial improvements over prior VLM-based approaches across multiple industry-standard driving benchmarks (Wang et al., 24 Feb 2026).

1. Architectural Components

VGGDrive comprises three principal modules:

1. Frozen 3D Foundation Model (VGGT): The VGGT network processes surround-view camera images $I = \{I_c\}_{c=1}^C$ and generates high-dimensional cross-view 3D feature representations $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$, where $B$ is the batch size, $C$ the number of views, $N_1$ the count of 3D tokens per view, and $D_1$ the 3D feature dimension.
2. Base Vision-LLM (VLM): Specifically, Qwen2.5-VL-7B is used. Its visual encoder converts the same multi-view images into 2D feature embeddings $$V_0^{2d} \in \mathbb{R}^{B \times (C \cdot N_2) \times D_2}$$; the text encoder embeds task instructions $L$ into tokens $T_0$. The resulting sequence $x_0 = [V_0^{2d}; T_0]$ is processed by $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$0 autoregressive decoder layers $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$1 for $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$2.
3. Plug-and-Play Cross-View 3D Geometric Enabler (CVGE): CVGE is incorporated with a Hierarchical Adaptive Injection Mechanism at each decoder layer. This mechanism:
   • Extracts the 2D visual tokens from the hidden state using a mask $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$3;
   • Fuses 2D features $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$4 with cross-view 3D features $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$5 using CVGE to yield $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$6;
   • Replaces the original 2D tokens in $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$7 with $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$8, and adds a residual connection for the next layer’s input.

This hierarchical, layer-by-layer injection enables the VLM to progressively incorporate genuine 3D geometric context, overcoming the limitations of conventional 2D VLMs.

2. Cross-View 3D Geometric Enabler (CVGE) Mechanism

Within each decoder layer $$V^{3d} \in \mathbb{R}^{B \times C \times N_1 \times D_1}$$9, the CVGE performs a learnable cross-modal attention for geometric grounding:

• Token Flattening and Alignment: The VGGT output is reorganized so that its token dimension matches the 2D embedding ($B$0).
• Dimensionality Projection: Two MLPs map 2D and 3D tokens into a shared $B$1-dimensional space, where $B$2.
• Camera Extrinsic/Intrinsic Prior Injection: For each camera view, the camera-to-LiDAR transformation matrix $B$3 is encoded and injected as a specialized “camera token” into the cross-attention mechanism.
• Cross-Attention: Multi-head cross-attention ($B$4 heads) is computed as $B$5, where $B$6 derives from 2D features, $B$7/$B$8 from realigned 3D features with camera tokens.
• Dimensionality Restoration: The result $B$9 is projected back to the original 2D embedding space via an MLP.

This mechanism allows 2D imagistic representations to “pull in” geometric cues derived from 3D perception, thereby enabling spatially and geometrically coherent vision-language reasoning.

3. Hierarchical Adaptive Injection Across Decoder Layers

At each decoder layer $C$0, VGGDrive applies the following procedure:

1. State Update: $C$1.
2. 2D Token Extraction: $C$2.
3. Cross-View 3D Fusion: $C$3.
4. Re-injection with Residual: The new hidden states $C$4 are formed by replacing the masked 2D portion of $C$5 with $C$6 and adding the original $C$7 as a residual.

This process is repeated at every decoder layer, enabling deep, progressive 3D grounding of the sequential representations. Ablations demonstrate that multi-layer injection achieves maximal performance, while employing single-layer injection at layer 11 achieves substantial gains.

4. Training Procedure

Training is structured in two distinct stages, always with the VGGT frozen:

• Stage 1 (CVGE Warmup): The VLM is frozen. CVGE modules are trained for 2 epochs with a learning rate of $C$8.
• Stage 2 (Full Fine-Tuning): Jointly fine-tune the VLM and CVGE for 2 additional epochs at $C$9.

The training objective is the standard autoregressive vision-language cross-entropy loss: $N_1$0 No auxiliary geometric or distillation losses are applied; geometric supervision is imparted implicitly via the frozen VGGT features.

5. Benchmark Evaluation Protocols

VGGDrive is evaluated on five compositional autonomous-driving benchmarks with diverse input and output modalities:

Benchmark	Primary Task(s)	Input Modality	Key Metrics
NuInstruct	Cross-view risk perception, regression, captioning	6-view NuScenes, text prompts	MAE↓, Accuracy↑, mAP↑, BLEU↑
DriveLM	Object matching, action prediction	6-view NuScenes, language	Accuracy↑, ChatGPT-score↑
OmniDrive	Scene captioning, Q&A	6-view NuScenes	BLEU↑, CIDEr↑, ROUGE↑
NuScenes-Plan	Open-loop trajectory planning	6-view images, ego state, commands	L2 error↓, Collision↓, Intersection↓
NAVSIM	Closed-loop planning	8-camera, LiDAR, commands	PDMS

For NAVSIM, the Predictive Driver Model Score (PDMS) is computed as: $N_1$1 where $N_1$2 is no-fault collisions, $N_1$3 is drivable-area compliance, $N_1$4 is ego progress, $N_1$5 is time-to-collision, and $N_1$6 is comfort.

6. Empirical Results and Comparative Analysis

VGGDrive demonstrates substantial performance gains relative to both the base VLM and to alternative fusion strategies (VGGT-Dist, VGGT-Add):

• NAVSIM (closed-loop): VGGDrive achieves PDMS = 88.76 (Base VLM: 86.04; VGGT-Dist/Add: 86.68/86.10).
• NuInstruct (cross-view risk): VGGDrive reaches mAP = 37.49% (Base: 6.15%; VGGT-Dist/Add: 28.51%/30.12%).
• DriveLM (action & planning): VGGDrive obtains 77.50% Accuracy (Base: 64.35%; prior SOTA: 71–73%).
• OmniDrive (captioning): VGGDrive matches prior language performance (BLEU/CIDEr/ROUGE avg ≈ 52.85 vs. 52.64 for Base).
• NuScenes-Plan (open-loop): VGGDrive reduces avg L2 error from 0.36 m to 0.31 m, collision from 0.37% to 0.22%, intersection from 1.59% to 2.31%.

Ablation studies confirm the superiority of multi-head cross-attention in CVGE over simple addition or distillation, and that hierarchical injection (across all decoder layers) yields the strongest gains. The addition of camera transform tokens is verified to contribute an additional +0.4 PDMS in NAVSIM. Qualitative analyses show that generated trajectories exhibit enhanced drivable-area compliance and fewer collisions, with improved language-based identification of spatially distant agents.

7. Limitations and Future Research Directions

Several open questions and future research avenues remain:

• The approach is dependent on accurate and static camera calibration; adaptation to dynamic changes in extrinsics could improve robustness.
• Joint multimodal pre-training of VLM and 3D perception (as opposed to post-hoc fusion) may further unify linguistic and geometric reasoning.
• Extension to additional sensor modalities, including LiDAR and radar integration at the CVGE stage, could augment safety-critical perception.
• Incorporating auxiliary geometric losses (e.g., enforcing depth consistency or view-alignment via contrastive approaches) may enable partial unfreezing and continual adaptation of the 3D foundation model.
• Integration of VGGDrive into closed-loop control stacks and validation in real-vehicle operation remain essential for practical deployment.

VGGDrive establishes a paradigm for plug-and-play, hierarchical infusion of frozen 3D model features into VLMs, enabling cross-view geometric grounding and delivering state-of-the-art results across heterogeneous autonomous driving tasks (Wang et al., 24 Feb 2026).