EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Control

Abstract: Modern vision-LLMs (VLMs) for driving assistants typically treat vehicle dynamics as a black box, resulting in decisions that lack awareness of the vehicle's real-time electro-mechanical state. To bridge this gap, we introduce the Electro-Visual-Language Assistant (EVLA) -- a novel framework that combines multi-modal scene understanding with real-time perception of the electrified powertrain state (e.g., motor torque, battery SOC). Our approach features two key innovations: first, a Unified Co-State Encoder (UCSE) that fuses visual, textual, and vehicle-state inputs into a shared latent representation, augmented with an Energy-Efficiency Field to model spatial energy costs; and second, an Electro-aware Structured Reasoning Chain (ESRC), which replaces external chain-of-thought prompting with an internal, deterministic reasoning process grounded in physical constraints and optimization objectives. Trained end-to-end with a physics-guided joint loss, EVLA learns to generate context-aware and energy-optimal driving decisions. Extensive evaluations on a driving QA benchmark demonstrate that EVLA substantially outperforms strong fine-tuned VLM baselines, improving the final score by +0.0871 and accuracy by +5.6\%. Ablation studies validate the necessity of each component, and efficiency analyses show that EVLA achieves 36\% faster inference than multi-stage pipelines. This work underscores that integrating vehicle-state awareness and structured physical reasoning is crucial for developing next-generation, physically-grounded driving assistants.

• The paper introduces EVLA, which fuses visual, language, and vehicle state modalities to enable energy-aware autonomous driving with improved prediction and planning.
• The proposed Unified Co-State Encoder and Electro-aware Structured Reasoning Chain achieve faster convergence and superior accuracy, outperforming baselines by +5.6%.
• Experimental results show enhanced inference speed, robust performance across modules, and clear interpretability, supporting real-time physics-grounded driving assistance.

Electro-Aware Multimodal Reasoning for Physically-Grounded Driving Assistance: An Expert Overview of EVLA

Motivation and Limitations of Prior Approaches

Vision-LLMs (VLMs) have considerably advanced autonomous driving by fusing visual perception and natural language understanding, enabling competent scene analysis and behavioral reasoning. However, these architectures typically lack explicit modeling of the vehicle's electro-mechanical state, which is crucial for safe and energy-efficient control, especially with the proliferation of electrified powertrains. Existing benchmarks and pipeline integrations operate in a passive manner, disregarding real-time variables such as motor torque, battery SOC, and thermal constraints—factors intrinsic to energy-aware decision-making.

Figure 1: Motivation for EVLA—explicit integration of visual, language, and vehicle state modalities is necessary for energy- and physics-sensitive reasoning in driving.

This deficit leads to physically ungrounded, suboptimal planning, and is left unmitigated by post-hoc Chain-of-Thought prompt engineering strategies. As demonstrated in prior work, scene understanding without explicit powertrain state modeling is insufficient for high-fidelity prediction and planning, particularly in electrification scenarios.

EVLA Architecture: Unified Co-State Latents and Structured Reasoning

EVLA—the Electro-Visual-Language Assistant—represents a paradigmatic shift by explicitly embedding the complete vehicle state alongside visual and textual domains. Its core technical innovations reside in two architectural components:

1. Unified Co-State Encoder (UCSE): A modality-fusing transformer that projects multi-view visual scenes, language queries, and a dense real-time vector of powertrain state ($\tau_m$, $\omega_m$, $P_{batt}$, $SOC$, etc.) into a shared latent. Critically, this space structurally encodes an Energy-Efficiency Field (EEF), modeling spatial energy costs associated with decisions, thereby generalizing beyond heuristic depth maps.
2. Electro-aware Structured Reasoning Chain (ESRC): This deterministic, internal symbolic reasoning chain replaces externalized, unstructured CoT prompting. ESRC sequentially parses the co-state latent into scene and powertrain factors, formalizes constraints and optimization objectives, and executes symbolic feasibility and deduction—yielding both interpretable traces and actionable control recommendations.

These modules are trained jointly under a multi-term physics-guided loss, including language, state prediction, control consistency, and EEF map fidelity, ensuring end-to-end grounding in vehicle dynamics.

Figure 2: EVLA system architecture—multi-modal encoding, co-state fusion, and structured reasoning yield physically plausible, interpretable actions.

Experimental Results and Ablations

Training Dynamics and Performance

Comprehensive evaluation on DriveLM-nuScenes demonstrates that EVLA convincingly outperforms LoRA/DoRA-LLaVA baselines. The architecture converges faster with lower training and validation losses, exhibits superior generalization, and attains a final aggregate score of 0.8548 (+0.0871 over baseline) and an accuracy improvement of +5.6%. Integration of powertrain state and structured reasoning is most consequential in Prediction and Planning tasks, confirming the value of explicit physical modeling.

Figure 3: EVLA trains more rapidly, generalizes better, and achieves a markedly higher validation score than the most competitive baseline.

Figure 4: Consistent performance improvements for EVLA, with especially strong gains in tasks requiring prediction and planning.

Role of Core Components

Ablation experiments reveal strict necessity of both the UCSE and ESRC. Removal of either (or powertrain state fusion in total) regresses performance to strong but ultimately inadequate visual-language compliance. The full EVLA configuration shows super-additive synergy: improvements in complex reasoning tasks exceed the sum of gains from integrating these modules separately.

Figure 5: Visualized results of ablation analysis—every core component (UCSE, ESRC, physics-trained objectives) adds substantial net value to all task categories.

Physics-Guided Optimization

Incrementally activating the state, control, and EEF losses in the joint training objective yields monotonic gains in final score. Each additional component translates into tangible improvements, underscoring the systemic advantage of direct physics domain knowledge injection.

Figure 6: Each loss component in the physics-guided training regime boosts EVLA's overall quality, confirming importance of multi-task objectives.

Pipeline Efficiency and Hyperparameter Robustness

Owing to its unified latent and internalized reasoning, EVLA’s inference is 1.6× faster than legacy multi-stage pipelines while providing end-to-end interpretability. Hyperparameter variations (loss coefficients and LoRA ranks) confirm the framework’s robustness across settings.

Figure 7: End-to-end reasoning yields significant inference speedups versus multi-stage baselines.

Figure 8: EVLA exhibits stable performance across a variety of loss weights and LoRA ranks, facilitating efficient hyperparameter selection.

Theoretical and Practical Implications

EVLA establishes that holistic, physically grounded driving assistance mandates real-time integration of powertrain state with perception and language. The architecture’s deterministic, rule-augmented symbolic reasoning chain ensures responses are not only data-driven but provably constrained by vehicle physics and safety rules. This internalizes formalisms long known in graph reliability and system diagnosability, adapting them into modern end-to-end neural reasoning systems. On the practical front, the architecture facilitates real-time, energy-aware autonomous driving policies, with interpretability suitable for both operational verification and regulatory compliance.

Key limitations lie in the current reliance on simulated and synthetic powertrain data for training; future extensions should address real-world deployment and validation, and generalization across more diverse vehicle platforms and long-horizon control regimes.

Conclusion

EVLA provides a new technical direction for vision-language driving assistants through integrated vehicle-state reasoning and structured, physically-constrained deduction. This approach yields significant empirical gains and architectural efficiencies, demonstrating that the next generation of autonomous driving systems requires fusion of semantics, language, and concrete vehicle physics in a unified, end-to-end framework. Further progress will require scaling to broader operational domains, robust handling of long-tailed real-world state transitions, and verified reliability in diverse conditions.

---

Reference:

"EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Control" (2606.28938)