Distilling Multi-modal Large Language Models for Autonomous Driving (2501.09757v1)

Abstract: Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage LLMs as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark.

Distilling Multi-modal LLMs for Autonomous Driving: An Analytical Review

In advancing the field of autonomous driving, the paper "Distilling Multi-modal LLMs for Autonomous Driving" introduces DistillDrive, an end-to-end system aimed at enhancing the planning performance of vision-based planners by distilling knowledge from multi-modal LLMs (MLLMs). This framework emphasizes the challenge of achieving robust planning, particularly in "long-tail" scenarios—rare but complex situations that accrue significant planning and safety requisites.

Summary of Approach

DistillDrive emerges as an overview of a powerful language-based reasoning framework and efficient vision-based planning mechanisms. The authors propose a joint training framework that incorporates an MLLM to guide a vision-based planner such as VAD or UniAD. By using the scene encoder of the vision-based planner as a "trainable tokenizer," the system produces structured scene representations, referred to as bird's-eye-view, ego, agent, and map (BEAM) token embeddings. This configuration allows the architecture to distill the nuanced world knowledge embedded within MLLMs into more efficient models grounded on visual inputs, thereby reducing computational overhead during inference while still leveraging the sophisticated reasoning capability of these LLMs.

Key Features and Methodology

Key innovations set this work apart:

1. Structured Scene Encoding: By introducing BEAM token embeddings, DistillDrive creates semantically rich, structured representations that feed into the MLLM, thus providing contextually sophisticated inputs derived from visual data.
2. Joint Training with Surrogate Tasks: The system deploys a suite of surrogate tasks, including masked token reconstruction, future token prediction, and scene editing, which are central to aligning the learning objectives of vision and MLLM-based planners. These tasks encourage the learning of spatial and temporal cues essential for successful planning and prediction.
3. Distillation Mechanism: The framework distills knowledge by aligning the feature distributions between the multi-modal LLM and the planning transformer, promoting richer feature learning transferable to an efficient vision-only inference.

Numerical Results and Evaluation

The paper presents rigorous quantitative assessments using the nuScenes dataset. Training with DistillDrive achieved a 37% reduction in L2 trajectory error and an 80% reduction in collision rates across general scenarios. In challenging long-tail scenarios, which further test the robustness of planners, the reductions in trajectory errors and collision rates are even more pronounced, underscoring the framework's ability to generalize from rare event data that typically challenge vision-based systems.

Furthermore, DistillDrive demonstrates superior performance in long-tail scenarios like zero-shot 3-point turns and overtaking maneuvers. This performance is noteworthy as success in such scenarios indicates superior generalization capabilities that are vital for real-world deployment.

Theoretical and Practical Implications

The theoretical contributions of this paper revolve around the alignment of vision-based representations with language-based reasoning structures, presenting a hybrid model that optimizes the strengths of both domains. Practically, this work bolsters the pathway toward deploying more efficient, yet robust, autonomous driving systems. The method balances the often-conflicting demands of computational efficiency and planning robustness, which are crucial as AI systems transition from experimental stages to deployment in densely dynamic urban environments.

Speculation on Future Developments

Looking ahead, the integration of multi-modal frameworks such as DistillDrive heralds a new era of AI systems where cross-domain contextual learning becomes central. Future research may further elucidate the granularity of how visual and textual information interact at deeper levels, potentially driving advancements not only in autonomous systems but in AI reasoning frameworks at large. Additionally, exploring scalability and adaptation in varied driving environments, beyond urban intersections to rural, less-structured environments, poses intriguing opportunities for extending this line of work.

In conclusion, the paper "Distilling Multi-modal LLMs for Autonomous Driving" provides significant insights and methodologies for enhancing the design and implementation of resilient autonomous driving systems through the novel integration of vision-based and language-informed frameworks. This work serves as a foundational step toward increasingly sophisticated AI applications in real-world risky decision-making scenarios.