Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models

Abstract: Leveraging the general world knowledge of LLMs holds significant promise for improving the ability of autonomous driving systems to handle rare and complex scenarios. While integrating LLMs into Vision-Language-Action (VLA) models has yielded state-of-the-art performance, their massive parameter counts pose severe challenges for latency-sensitive and energy-efficient deployment. Distilling LLM knowledge into a compact driving model offers a compelling solution to retain these reasoning capabilities while maintaining a manageable computational footprint. Although previous works have demonstrated the efficacy of distillation, these efforts have primarily focused on relatively simple scenarios and open-loop evaluations. Therefore, in this work, we investigate LLM distillation in more complex, interactive scenarios under closed-loop evaluation. We demonstrate that through a combination of latent feature distillation and ground-truth trajectory supervision, an efficient vision-only student model \textbf{Orion-Lite} can even surpass the performance of its massive VLA teacher, ORION. Setting a new state-of-the-art on the rigorous Bench2Drive benchmark, with a Driving Score of 80.6. Ultimately, this reveals that vision-only architectures still possess significant, untapped potential for high-performance reactive planning.

• The paper demonstrates that a compact student transformer can mimic a 7B LLM’s reasoning, yielding state-of-the-art autonomous driving performance.
• It employs joint latent feature mimicry and trajectory supervision to achieve a 150x inference speedup and improved driving scores on Bench2Drive.
• The study validates that a shallow decoder with retained vision encoders balances efficiency and robust closed-loop planning in complex scenes.

Efficient LLM Reasoning Distillation for Vision-Only Autonomous Driving: A Summary of "Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models" (2604.08266)

Introduction and Motivation

The integration of LLMs within vision-language-action (VLA) models has markedly advanced autonomous driving, primarily by endowing autonomous driving systems with enhanced world knowledge and explicit causal reasoning. However, these models demand enormous computational resources at inference time, restricting practical deployment in latency and energy-sensitive domains. This work addresses these limitations by proposing a distillation framework that compresses the reasoning ability of a 7B-parameter LLM (teacher) into a lightweight, vision-only transformer decoder (student), Orion-Lite, and demonstrates that this student not only retains but surpasses the driving performance of its teacher in complex, interactive, closed-loop scenarios.

Figure 1: Overview of the proposed distillation framework demonstrating joint distillation and trajectory supervision with Orion-Lite, achieving 3$\times$ inference speedup and state-of-the-art Bench2Drive performance.

Distillation Framework: Technical Approach

The central technical insight of this work is that in state-of-the-art VLAs (e.g., ORION), the LLM is effectively reduced to a feature extractor within the direct inference mode, bypassing autoregressive reasoning chains in favor of generating latent planning features via templated prompting. Orion-Lite replaces this 7B-parameter autoregressive LLM with a compact, randomly initialized, shallow transformer decoder, and relies on latent feature mimicry and ground-truth trajectory supervision as synergistic training signals.

Specifically:

• The teacher's multimodal processing stack (frozen vision encoder and temporal QT-Former) is retained for feature extraction.
• The student receives visual scene embeddings and driving commands, and is trained to match the teacher’s intermediate planning features (via $\mathcal{L}_1$ regression mimic loss), with additional direct supervision from ground-truth trajectories, collision penalties, and VAE KL-regularization.
• The vision encoder and QT-Former are frozen to maintain learned spatial-temporal priors and to isolate the reasoning module as the sole locus of distillation and learning.

This approach allows the replacement of a massive LLM with a drastically more efficient transformer decoder, without auxiliary encoders or distributional regularizers.

Bench2Drive Benchmark and Metric Analysis

All experiments are conducted on Bench2Drive, a closed-loop evaluation suite constructed atop CARLA V2, specifically designed for assessing E2E driving under high-complexity, high-diversity, and scenario-driven regimes. Bench2Drive outpaces prior open-loop and closed-loop protocols with 44 interactive scenarios, 23 atmospheric conditions, and 12 towns, demanding robust planning, control, and context-awareness from models.

Key closed-loop metrics include Driving Score (DS), Success Rate (SR), Efficiency, Comfortness, and per-scenario Multi-Ability scores (e.g., merging, overtaking, emergency braking).

Results and Empirical Findings

Orion-Lite achieves the following when compared to the 7B-parameter ORION teacher and other VLA/RL/World Model methods:

• Inference Speed and Efficiency: Orion-Lite’s reasoning module attains a 150$\times$ speedup over the LLM teacher, and overall system speedup is 3$\times$, with memory usage dropping from 31GB to 8GB.

Figure 2: Orion-Lite attains massive reduction in inference latency over its teacher while improving the main Driving Score; latency is measured as average inference step time on CARLA (A6000 GPU).

• Driving Performance: Orion-Lite reaches a Driving Score of 80.6 (+2.9 over the teacher), SR of 55.5% (+0.9), and mean Multi-Ability of 60.5% (+5.8), establishing a new state-of-the-art on closed-loop Bench2Drive.
• Behavioral Analysis: In controlled scenarios (merging, overtaking), Orion-Lite exhibits more decisive and robust planning, consistently avoiding the hesitation and failures observed in the teacher when negotiating dynamic obstacles and edge cases.

Figure 3: Rollout trajectories show Orion teacher's indecision and failures in complex scenes, whereas Orion-Lite executes robust, intervention-free, and smooth maneuvers.

Ablation Studies and Model Analysis

• Necessity of Joint Supervision: Ablation reveals that both latent mimic loss and direct ground-truth trajectory supervision are required for optimal performance; each alone yields sub-SOTA but competitive results, while their combination enables the performance leap.
• Model Capacity and Generalization: Increasing transformer decoder layers beyond six degrades generalization, indicating the distillation task requires low to moderate model capacity and is susceptible to overfitting with overparameterization.

Figure 4: Increasing depth of student decoder past six layers reduces both Driving Score and Multi-Ability, indicating the optimality of a shallow design.

• Impact of Encoder Initialization: Retaining the teacher-trained encoder and freezing it during student training yields higher driving performance than fine-tuning a generic encoder, supporting the claim that joint vision-language training imbues necessary semantic priors for autonomous driving.
• Loss Function Choice: Standard $\mathcal{L}_1$ regression yields lower collision rates and more robust training than $\mathcal{L}_2$, KL-divergence, or Huber loss, due to its resilience to outlier latent activations.

Implications and Theoretical Considerations

The work empirically challenges the necessity for large autoregressive LLMs at inference-time in high-performing E2E autonomous driving. The critical finding is that, for reactive, closed-loop driving with strong vision encoders and trajectory planners, shallow transformer decoders are sufficient to internalize and apply the reasoning patterns previously encoded by LLMs, given proper distillation. This points toward a paradigm where model efficiency, modular distillation, and targeted architectural simplification can break the prevalent dogma of ever-increasing parameterization for closed-loop vision-action tasks.

From a practical perspective, this unlocks feasible deployment on hardware with stringent computational and energy constraints without sacrificing closed-loop autonomy or handling of complex, long-tail scenarios. Theoretically, it raises new questions about the boundary between explicit (parametric) knowledge and implicit (feature-based) knowledge in VLA architectures, and how general world reasoning can be modularly injected or induced in compact visual planning stacks.

Limitations and Prospective Directions

The main limitation remains the reliance on the existence of a fully trained, computationally intensive VLA teacher. Additionally, the vision encoder becomes the new computational bottleneck once the reasoning module is minimized. The current empirical validation scope is limited to Bench2Drive, necessitating broader cross-benchmark verification and studies on scenarios with even more complexity and long-tail events.

Future work should focus on:

• Architectures that allow direct injection of LLM world knowledge into visual planners without expensive teacher models.
• Joint optimization of visual feature extractors during the distillation process.
• Investigating the limits of purely vision-based reasoning and the minimal necessary model capacity for various autonomous driving regimes.
• Extending analysis to rare, uncurated scenarios to further pressure-test the limits of vision-only architectures.

Conclusion

This work presents a robust knowledge distillation framework demonstrating that a compact transformer decoder can supplant a 7B LLM for closed-loop vision-only autonomous driving, without incurring a performance penalty and in fact establishing state-of-the-art results on a highly challenging benchmark. The findings highlight the substantial untapped potential in efficient, vision-only architectures for E2E planning, emphasizing training regimes and knowledge transfer protocols over raw model scaling. This suggests novel trajectories for research into specialized, efficient, and deployable intelligence for autonomous systems.

(2604.08266)