LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design

Abstract: Vision-radar fusion is central to robust autonomous driving, combining dense visual semantics with precise range and velocity measurements from radar. However, real-world fusion quality is fundamentally challenged by dynamically varying input quality, stemming from occlusion, adverse weather, and channel noise. To address this, we re-frame the problem from static data fusion to channel-aware semantic reasoning and propose a LLM-centric Semantic-layer Channel-aware Integrated Perception (LM-SCIP) framework. It places a LLM as a central reasoning core to fuse a local visual stream with a quality-varying external radar stream used to cover perception-blind spots. Concretely, LM-SCIP couples a hierarchical radar-vision encoder with a Channel-Adaptive Semantic Module (CASM) that maps link indicators into a "Channel Prompt" to dynamically gate external radar features. A parameter-efficient, LoRA-tuned LLM, in conjunction with a heterogeneous Mixture-of-Experts (H-MoE), then arbitrates between local visual cues and the channel-conditioned radar context. Finally, a decoupled multi-task decoder outputs localization, trajectory forecasting, and image reconstruction. Experiments on nuScenes and VIRAT validate our approach. On nuScenes, under a controlled toggle of radar input, LM-SCIP reduces localization RMSE by 40.0% versus a vision-only baseline. On VIRAT, the model attains a 0.214m localization RMSE and 0.179m minFDE (k=1). These results reveal that the proposed LM-SCIP enables a robust vision-dominant fallback at low SNR and synergistic fusion at high SNR.

• The paper presents LM-SCIP, a novel fusion framework that uses channel state indicators to dynamically adjust multimodal perception.
• It employs a multimodal semantic encoder and a heterogeneous mixture-of-experts LLM to balance vision and radar inputs based on real-time wireless conditions.
• Experimental results show over 40% RMSE reduction in localization and improved robustness against occlusion and low SNR compared to vision-only methods.

Semantic-Layer Multimodal Fusion for Autonomous Driving with Channel-Adaptive LLM Reasoning

Introduction

The integration of multimodal perception, specifically vision and radar fusion, is pivotal for enhancing safety and robustness in autonomous driving, particularly under challenging conditions involving occlusion and adverse weather. Traditional systems typically assume static, reliable sensors and fixed fusion policies, which are inadequate for real-world cooperative perception scenarios where input quality, especially via V2X links, is time-varying due to wireless channel fluctuations. The paper "LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design" (2607.01772) introduces LM-SCIP, a novel framework that elevates multimodal fusion to the semantic layer and leverages a LLM to perform channel-adaptive reasoning, dynamically arbitrating the trust across modalities based on real-time link quality.

Framework Architecture and Semantic Layer Design

LM-SCIP is architected to operate in infrastructure-centric cooperative perception settings, addressing occlusion and blind spots by integrating input from a local camera and a remote radar obtained via V2X communication. The system comprises four principal modules:

1. Multimodal Semantic Encoder (MSE): Utilizes modality-specific backbones (hierarchical ViT for vision, complex CNN for radar I/Q waveform). Bidirectional cross-modal attention aligns feature spaces without explicit BEV calibration.
2. Channel-Adaptive Semantic Module (CASM): Encodes link quality indicators (SNR, modulation) into a Channel Prompt via multi-layer embeddings and transformers, gating the contribution of remote radar features through adaptive masking mechanisms. This module ensures that unreliable features are dynamically attenuated when the wireless channel degrades.
3. Heterogeneous Mixture-of-Experts LLM (H-MoE): Features a token-level router dispatching representations to task-specialized MLP/Transformer experts, with task arbitration performed by a LoRA-tuned GPT-2 core; this design ensures both efficient multi-task learning and explicit use of channel context for semantic reasoning.
4. Decoupled Multi-Task Decoder: Employs distinct heads for localization, trajectory prediction, and visual reconstruction, mitigating interference between dense (reconstruction) and sparse (localization, forecasting) tasks.

Channel-Awareness and Semantic Gating

A central innovation of LM-SCIP is treating channel state as a first-class citizen for perception fusion. CASM leverages physical layer indicators to generate a Channel Prompt, conditioning the semantic fusion by modulating the degree to which external radar is trusted. At low SNR, the gating mechanism suppresses the contribution of the radar stream, resulting in a fallback to vision dominance. Conversely, as SNR improves, the radar features are reweighted upwards, yielding a synergistic fusion regime. This approach extends prior works by explicitly coupling perception confidence to real-time wireless environment variations.

Multi-Task Learning with H-MoE

The token-level routing via H-MoE enables efficient specialization, crucial given the disparity in requirements across tasks: precise regression for localization, long-horizon temporal modeling for trajectory forecasting, and dense generative synthesis for reconstruction. The LoRA-tuned GPT-2 LLM component is pivotal for global reasoning, integrating task and channel context at the semantic layer. Explicit load-balancing losses mitigate MoE collapse and enable robust co-training.

Experimental Results and Numerical Highlights

Comprehensive evaluations are conducted on nuScenes and VIRAT datasets, under both ideal and adverse (occlusion, weather, low SNR) conditions. Key quantitative findings:

• Localization Performance: On VIRAT, LM-SCIP achieves an RMSE of 0.214 m under ideal conditions—a 42.3% reduction versus a vision-only baseline. On nuScenes, adding radar yields a 40.0% reduction in localization RMSE relative to vision-only, outperforming other fusion frameworks (e.g., CRAFT achieves a 32.3% reduction).
• Trajectory Forecasting: The model attains minFDE1 = 0.179 m on VIRAT, matching leading unimodal forecasters while delivering multi-modal robustness.
• Robustness under Degradation: With severe occlusion and low SNR (5 dB), the RMSE degradation remains contained ($\approx$13% relative increase with CASM ablation), and ADE increases by only $\approx$15%, confirming CASM’s critical role in maintaining reliability under adverse conditions.
• Task Decoupling and Specialization: Ablation demonstrates that eliminating the H-MoE drastically deteriorates all metrics (e.g., RMSE rises from 0.214 to 13.083 m), reflecting the necessity of expert-based specialization for multi-task robustness.

Quality-of-service metrics reinforce operational viability: in ideal scenarios, 95.97% of localizations are within 0.5 m, and over 85% of reconstructions exceed 20 dB PSNR, with only marginal drops under SNR/occlusion stress.

Implications and Future Directions

By explicitly integrating channel state information into the inference loop and arbitrating fusion at the semantic level, LM-SCIP addresses limitations of context-agnostic multimodal fusion and sets a new standard for robustness in cooperative autonomous driving perception. This paradigm not only improves resilience to wireless degradation, but also offers a model-agnostic pathway for incorporating richer V2X side information and for scaling to multi-agent coordination via prompt-level interactions.

Practically, deployment prospects for LM-SCIP are promising due to its edge-friendly inference pipeline, reliance on parameter-efficient LoRA-tuned LLMs, and compatibility with emerging V2X standards. Theoretically, this work frames semantic communication and reasoning in a unified architecture, bridging the gap between AI perception and wireless channel modeling.

Extensions include incorporating more granular or predictive link quality indicators, prompt-level multi-agent collaboration over dynamic topologies, and emulation with real-world V2X hardware. Furthermore, the approach is amenable to progressive scaling, both in model size and cooperative agent count, as LLM-based perception frameworks continue to evolve.

Conclusion

LM-SCIP exemplifies a channel-aware, LLM-centric multimodal framework for autonomous driving that elevates fusion strategies from fixed feature-level operations to dynamic, context-sensitive reasoning at the semantic layer. Empirical evidence demonstrates substantial accuracy gains and robustness in both typical and degraded settings, substantiating the utility of channel-informed LLM arbitration. This methodology lays the groundwork for future research at the intersection of cooperative perception, semantic communication, and multi-agent collaboration in autonomous systems.