LUNA-AD: Lightweight Uncertainty-Aware Language Model with Lifelong Learning for Autonomous Driving

Abstract: While LLMs offer promising reasoning capabilities, their integration into safety-critical driving systems is hindered by limited reasoning diversity, high computational overhead, and static learning paradigms. To address these challenges, we propose LUNA-AD, a lightweight uncertainty-aware LLM with lifelong learning for autonomous driving (AD). LUNA-AD features a tri-system architecture that reconciles complex multimodal behavioral reasoning, efficient deployment, and continual refinement. We design a multi-agent analytical system to generate uncertainty-aware decision-making demonstrations through diverse hypothesis exploration. A dual-head lightweight heuristic model is distilled to unify the inference of decision distributions and textual explanations while enabling efficient deployment. Furthermore, a reflection-driven lifelong learning mechanism operates on multimodal decision outputs and preserves strategic diversity, allowing for the refinement of candidate decisions and rationales via closed-loop feedback to enhance driving robustness. Extensive experiments on nuPlan benchmarks demonstrate that LUNA-AD achieves state-of-the-art success rates under both non-reactive and reactive modes, with drastically reduced inference latency compared to existing knowledge-driven AD frameworks.

• The paper introduces a tri-system framework combining multi-agent analytical reasoning, lightweight dual-head inference, and reflective lifelong learning for autonomous driving.
• It employs uncertainty-aware distillation and calibrated ensemble methods to enhance decision-making and robustness, achieving success rates above 89% on challenging benchmarks.
• The framework optimizes onboard computational efficiency with sub-200 ms inference latency, enabling safe, adaptive, real-world autonomous driving under resource constraints.

LUNA-AD: Lightweight Uncertainty-Aware LLM with Lifelong Learning for Autonomous Driving

Introduction and Motivation

LUNA-AD presents a cohesive framework designed to reconcile the demands of high-fidelity multimodal reasoning, lightweight deployment, and continual adaptation in autonomous driving decision-making. This addresses key deficiencies in prior LLM- and VLM-based approaches: insufficient uncertainty modeling, excessive computational burden, and static (non-adaptive) learning. The framework is motivated by the necessity for socially compliant, contextually rich decision processes that remain tractable for onboard hardware and robust against long-tail, out-of-distribution encounter scenarios.

Figure 1: Architecture overview showing the tri-system decision-making pipeline integrating multi-agent analytical reasoning, lightweight heuristic inference, and reflection-driven lifelong learning for closed-loop autonomous driving.

Tri-System Framework Architecture

LUNA-AD is structured around three synergistic subsystems:

1. Multi-Agent Analytical System (MAAS): Implements ensemble-based hypothesis generation through diverse agent querying, each performing structured chain-of-thought reasoning and explicit confidence assessment. This approach produces uncertainty-calibrated, multimodal demonstrations, rather than one-hot, single-hypothesis outputs common in chain-of-thought or majority-vote LLM protocols.
2. Dual-Head Lightweight Heuristic System: A parameter-efficient, retrieval-augmented model trained via uncertainty-aware distillation from MAAS outputs. Its dual-head architecture jointly predicts decision probability distributions (DPP) and generates natural language rationales, supporting both robust downstream planning and human-interpretable explanations with low-latency inference.
3. Reflection System: This module closes the learning loop by leveraging event-triggered feedback via a progress monitor. Upon detecting a system-level failure (e.g., collision, stall), this system invokes detailed reasoning over the causal decision and suggests revised explanations or corrections, persistently updating the knowledge base via memory bank augmentation.

Multi-Agent Analytical Reasoning and Uncertainty-Aware Demonstrations

The multi-agent analytical system operationalizes multimodal behavioral reasoning as a three-stage collaborative process: action voting (with stochastic decoding), confidence assessment (deterministic, hypothesis-specific), and summarization. Each agent is guided by precisely engineered instructional prompts; action voting agents perform detailed agent-centric scene interpretation and decision justification, confidence assessors provide calibrated probability estimates, and summarization agents synthesize outcome distributions in a manner that preserves semantic context without workflow leakage.

Figure 2: Three-stage collaborative reasoning by MAAS, yielding diverse hypothesis exploration, calibrated confidence aggregation, and uncertainty-aware rationales for downstream distillation.

Memory accumulation is realized by archiving each multimodal decision query, scene embedding, and summarized demonstration, forming a retrieval-augmented memory bank for downstream heuristic model supervision.

Lightweight Dual-Head Heuristic System

To address onboard computational limits, LUNA-AD distills the analytical system's "dark knowledge" into a lightweight dual-task model. The architecture extends a pre-trained LLM backbone with two heads: one for DPP and one for LM. The training objective couples cross-entropy for language modeling with KL divergence for uncertainty-preserving DPP, balanced by a scalar parameter.

During inference, DPP operates at the decision layer at regular low latency, while high-cost textual explanation is only triggered upon adverse events, optimizing resource allocation and adaptability.

Figure 3: Dual-head parameter-efficient architecture supporting simultaneous uncertainty-aware decision distribution prediction and linguistically rich rationale generation.

Decision-Guided Planning and Adaptive Feedback

The adaptive planner leverages the predicted decision distribution to construct a probability-thresholded action set, generating and optimizing candidate trajectories via diffusion-based sampling. Trajectory arbitration directly incorporates both soft decision probabilities and motion scoring, producing socially and contextually smooth execution behaviors.

Closed-Loop Lifelong Learning with Reflection

Robustness is promoted by integrating a reflection subsystem triggered on detection of collisions or stalls. Upon failure, the system reconstructs the dialogue (decision, rationale, system state) and invokes high-capacity LLMs to analyze the failure, suggest corrections, and expand the memory bank with revised demonstrations. This enables continuous knowledge evolution, responding adaptively to rare or previously unseen operational hazards.

Figure 4: Closed-loop reflection process for dynamic self-correction and knowledge update upon failure detection and analysis.

Empirical Evaluation

Quantitative Closed-Loop Performance

LUNA-AD demonstrates state-of-the-art performance on nuPlan Test14-Hard and Test14-Random with success rates of 89.71% (NR-SR) and 90.44% (R-SR) on long-tail benchmarks, outperforming all compared knowledge-driven, hybrid, and pure learning-based baselines. Noteworthy is the substantially reduced inference latency (192 ms, 1.7B parameters) relative to competing LLM-based planners (multi-B parameter regimes, >2.9–5.6 s latency).

Ablation and Open-Loop Analysis

Ablation studies confirm that each tri-system component (uncertainty-aware voting, explicit confidence aggregation, summarization, and reflection) contributes monotonically to end-to-end robustness and coverage. Removing retrieval-augmented generation or event-triggered explanation impairs both interactive stability and performance under distribution shift.

Open-loop few-shot analysis indicates that retrieval of 3 few-shot demonstrations yields optimal calibration and textual explanation quality for the heuristic model, with test-time adaptability to varying context lengths.

Figure 5: Influence of retrieval-augmented few-shot context size on decision accuracy, calibration, language quality, and latency; highlighting the trade-off space and optimal configuration.

Vision-Domain Transfer and Field Deployments

Experiments with multi-view RGB input confirm that the analytical module generalizes to raw visual observations, achieving Acc@1 of 65% and Acc@3 of 78.33% on DriveLM-NuScenes. Field-collected campus trials demonstrate robust handling of unstructured layouts and ambiguous agent interactions, maintaining interpretable, uncertainty-modulated decisions.

Figure 6: Six-camera system for field data capture, empowering vision-based extension and real-world deployment studies.

Theoretical Implications and Future Directions

LUNA-AD makes several strong claims which contradict prevalent LLM deployment strategies for AD:

• Cascaded tri-system architectures with explicit uncertainty modeling outperform both deterministic LLM planners and end-to-end VLA models under safety-critical constraints, challenging the convention of single-hypothesis autoregressive planning.
• Parameter-efficient, uncertainty-aware models can match or surpass substantially larger LLMs in success rate and robustness when equipped with retrieval memory and event-triggered adaptation, countering general scaling law assumptions.

The reflection module embodies a paradigm shift toward continual memory growth and closed-loop semantic self-improvement, setting the stage for more adaptive, generalizable embodied LMs.

Conclusion

LUNA-AD establishes a new standard for interpretable, robust, and efficient autonomous driving cognition by unifying multimodal analytical exploration, lightweight knowledge distillation, and reflection-driven lifelong learning. The framework's tri-system design demonstrates principled synergy between high-capacity LLM reasoning and hardware-constrained deployment, with robust uncertainty modulation and behavioral diversification. Empirical results exemplify its competitiveness and efficiency on canonical closed- and open-loop AD benchmarks, suggesting immediate viability for real-world, resource-constrained applications. Advancements in direct vision-language integration and further migration towards end-to-end perception-reasoning architectures constitute promising directions for expansions upon this work (2606.08470).