Distilling Multi-modal Large Language Models for Autonomous Driving

Abstract: Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage LLMs as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.

• The paper introduces DistillDrive, a framework that distills multi-modal LLM knowledge into a vision-based planner, significantly improving planning in long-tail driving scenarios.
• The paper employs joint training, surrogate tasks, and KL-divergence-based distillation to align scene representations, reducing trajectory errors by 37% and collisions by 80%.
• The paper demonstrates robust VQA integration for scene reasoning, offering interpretable planning outputs while maintaining low inference latency at deployment.

DistillDrive: Distilling Multi-modal LLMs for Autonomous Driving

Introduction

The paper presents DistillDrive, a framework for end-to-end autonomous driving that leverages the world knowledge of multi-modal LLMs (MLLMs) while maintaining the computational efficiency of vision-based planners. The motivation stems from the limitations of vision-only planners in handling long-tail scenarios and the prohibitive inference cost of LLM-based planners. DistillDrive addresses these challenges by distilling knowledge from an MLLM into a vision-based planner through joint training and a suite of surrogate tasks, enabling robust planning and visual question answering (VQA) without requiring the LLM at inference.

Framework Overview

DistillDrive consists of two main components: a vision-based planner and an MLLM. The vision-based planner is responsible for fast trajectory prediction and acts as a tokenizer for the MLLM, providing structured scene representations. The MLLM receives these structured inputs and is trained for planning, VQA, and several surrogate tasks designed to enrich and ground the scene representations.

Figure 1: Overview of DistillDrive, showing the flow from multi-view images and text prompts through the scene encoder and planning transformer, with structured BEAM token embeddings shared with the MLLM.

Vision-based Planner

The planner is decomposed into a scene encoder and a planning transformer. The scene encoder produces BEAM (Bird's-eye-view, Ego, Agent, Map) token embeddings, which are high-dimensional representations of scene components. These embeddings are used both for trajectory prediction and as input to the MLLM. Unlike prior works, the scene encoder is jointly trained with the MLLM, allowing for semantically grounded and task-aligned representations.

Multi-modal LLM

The MLLM comprises component-specific Q-former adapter layers, a LLM (LLaVA-v1.5-7B), and task-specific decoder heads. The Q-formers project BEAM tokens into a shared embedding space, compressing and structuring the input for the LLM. The MLLM is supervised on planning, VQA, and surrogate tasks:

• Masked token reconstruction: The MLLM reconstructs masked BEV tokens, enriching visual representations.
• Future token prediction: The MLLM predicts future BEV tokens, learning spatio-temporal cues for planning.
• Scene editing: The MLLM reasons about scene modifications (addition/deletion of agents) and their impact on planning, with corresponding QA pairs.

  Figure 2: Examples of scene editing, illustrating addition and deletion of agents and the associated question-answer pairs for grounding edits in language.

Distillation Mechanism

A key innovation is the distillation loss, which aligns the penultimate layer features of the planning transformer and the MLLM via KL-divergence minimization. This facilitates knowledge transfer from the MLLM to the vision-based planner, improving robustness to rare events.

Experimental Results

DistillDrive is evaluated on the nuScenes open-loop planning benchmark, using both standardized and VAD evaluation protocols. The framework demonstrates substantial improvements over baseline vision-based planners (VAD, UniAD) and recent MLLM-based planners (PARA-Drive, TOKEN, DriveVLM).

Figure 3: Comparison of planning performance in long-tail scenarios from nuScenes, showing DistillDrive-VAD's superior robustness in overtaking and 3-point turn maneuvers.

Quantitative Performance

• Trajectory error: DistillDrive achieves a 37% reduction in L2 trajectory error and a 44% reduction in long-tail scenarios compared to vision-only planners.
• Collision rate: An 80% reduction in collision rate is observed.
• Efficiency: DistillDrive matches or exceeds the accuracy of LLM-based planners while maintaining the low inference latency of vision-based planners, as the LLM is not required at test time.

Long-tail Scenario Robustness

DistillDrive consistently outperforms baselines in zero-shot and rare maneuvers, such as 3-point turns and overtaking, demonstrating the effectiveness of knowledge distillation from the MLLM.

Figure 4: Visual comparison of planning performance between DistillDrive (VAD-Tiny) and VAD-Tiny, highlighting improved trajectory prediction in challenging scenarios.

Visual Question Answering

The MLLM branch of DistillDrive supports VQA, enabling interpretable reasoning about the scene and planned actions. Qualitative results show accurate responses to complex queries about perception, prediction, and planning.

Figure 5: Visualization of planning and VQA by the MLLM branch, with predicted trajectory and example LLM response from the DriveLM test dataset.

Ablation and Analysis

Ablation studies confirm the importance of joint training, structured scene tokens, distillation, and surrogate tasks. Each component contributes to improved planning accuracy and collision avoidance. The surrogate tasks, particularly scene editing and future prediction, are critical for learning robust, grounded representations.

Implementation Considerations

• Training: Two-stage training is employed: pre-training the vision planner, followed by joint training with the MLLM using LoRA for efficient LLM fine-tuning.
• Resource requirements: The framework is designed to be scalable, with Q-former adapters and token sequence length constraints to manage memory consumption.
• Deployment: At inference, only the vision-based planner is required, ensuring real-time performance suitable for deployment in autonomous vehicles.

Implications and Future Directions

DistillDrive demonstrates that structured knowledge distillation from MLLMs can significantly enhance the robustness and interpretability of vision-based autonomous driving systems without incurring the computational cost of LLM inference. The approach suggests that future research should focus on further enriching scene representations, exploring more advanced surrogate tasks, and extending the framework to closed-loop and real-world driving scenarios. The integration of VQA capabilities also opens avenues for human-in-the-loop and explainable AI in autonomous driving.

Figure 6: Visualization of VQA and planning prediction by the MLLM branch, illustrating the model's ability to answer complex scene queries and predict safe trajectories.

Conclusion

DistillDrive introduces a principled framework for distilling multi-modal LLM knowledge into efficient vision-based planners for autonomous driving. Through joint training, structured scene encoding, and targeted surrogate tasks, the system achieves state-of-the-art planning performance and robustness in long-tail scenarios, while supporting interpretable reasoning via VQA. The results indicate that structured distillation from MLLMs is a promising direction for scalable, robust, and explainable autonomous driving systems.