Optimizing Tracking Accuracy in Energy-Constrained Multimodal ISAC via Lyapunov-Driven Heterogeneous Mixture-of-Experts

Abstract: The integration of multimodal sensing and millimeter-wave (mmWave) communications is a key enabler for highly mobile vehicle-to-infrastructure (V2I) networks. However, continuous high-resolution visual sensing incurs prohibitive computational energy, while delayed sensing information causes severe beam misalignment. This paper establishes a physics-aware multimodal integrated sensing and communication (M-ISAC) framework that mathematically bridges network-layer queuing delays with physical-layer spatial uncertainty via the semantic age of information (AoI). Guided by this relationship, we aim to strike an optimal trade-off between the tracking posterior Cramer-Rao bound (PCRB) and system energy budgets, we formulate a stochastic mixed-integer non-linear programming (MINLP) problem. Addressing the coupled challenges of temporal computing congestion and non-convex constant modulus constraints, we propose a reinforcement learning (RL) framework empowered by a Lyapunov-driven heterogeneous mixture-of-experts (LD-H-MoE) architecture. By strictly decoupling temporal scheduling and spatial phase mapping into specialized subnetworks, the LD-H-MoE circumvents gradient conflicts prevalent in monolithic multi-task learning. Simulations demonstrate that the proposed LD-H-MoE achieves a highly-effective event-triggered sensing policy, yielding superior tracking accuracy and radio-frequency (RF) resilience while guaranteeing edge computing queue stability and long-term energy budgets.

• The paper presents a novel LD-H-MoE architecture that optimizes tracking accuracy and energy constraints using Lyapunov-driven reinforcement learning.
• It integrates physics-aware AoI with PCRB and distinct expert networks to manage beam misalignment and ensure edge computing queue stability in V2I networks.
• Experimental results demonstrate near-optimal PCRB performance and robust energy budget compliance under varying SNR and harsh channel conditions.

Lyapunov-Driven Multimodal ISAC Optimization in Energy-Constrained V2I Networks

Introduction

This paper addresses the optimization of tracking accuracy in multimodal Integrated Sensing and Communication (ISAC) systems for highly mobile Vehicle-to-Infrastructure (V2I) networks operating at mmWave frequencies. The core challenge lies in sustaining accurate beam tracking and robust connectivity under stringent computational energy constraints. The work introduces a physics-aware cross-layer framework that mathematically couples network-layer queuing delays to physical-layer spatial uncertainties via the Age of Information (AoI). The optimization objective integrates the posterior Cramér-Rao Bound (PCRB) for tracking accuracy with long-term energy budgets, resulting in a mixed-integer non-linear programming (MINLP) problem. The authors tackle this using a Lyapunov-driven heterogeneous mixture-of-experts (LD-H-MoE) architecture within a reinforcement learning framework, leveraging strict gradient decoupling to address the temporal scheduling and spatial phase mapping separately.

Figure 1: Illustration of the multimodal ISAC system in a V2I scenario.

System Model and Mathematical Foundations

The model considers a single base station (BS) with a large antenna array and heterogeneous sensors (RGB camera and mmWave radar) serving multiple high-mobility vehicles. Beamforming in mmWave bands is modeled with geometric multipath channels as multivariate functions of vehicle position and angle of arrival (AoA), focusing primarily on macroscopic beam misalignment. The edge computing queue dynamics reflect the computational burden of visual sensing and radar observations, with energy consumption driven by both computation (modeled using DVFS) and RF beam recovery penalties due to misalignment.

The novelty arises from the vectorized and physics-aware AoI: the radial AoI is continually refreshed by radar, while the tangential AoI is governed by the completion of queued camera processing tasks. The spatial uncertainty at the BS is characterized as a 2D Gaussian distribution whose covariance is coupled with AoI and vehicle velocity components.

Figure 2: Hierarchical execution architecture of the proposed LD-H-MoE, illustrating the interplay between the inner heterogeneous policy networks and the outer Lyapunov optimization loop.

Beam misalignment is rigorously quantified by a Q-function mapping the Gaussian angular error statistics to beam failure probability. This is essential for controlling system energy by adjusting recovery efforts in response to deteriorating spatial uncertainty.

The reliability metric is the AoI-coupled PCRB, which combines the Bayesian Fisher information from real-time multimodal observations and historical state estimates. The angular PCRB is prioritized due to its dominant role in mmWave link robustness.

Optimization Framework

Directly solving the long-term, time-coupled MINLP problem is intractable. The paper applies Lyapunov stochastic optimization by introducing a virtual energy deficit queue, enabling the transformation of the original problem into a sequence of deterministic single-slot surrogate objectives (drift-plus-penalty approach). This decouples queue stability and energy adherence requirements, allowing opportunistic minimization at each time slot.

The action space in the Markov Decision Process (MDP) encompasses discrete sensor scheduling, continuous CPU frequency allocations, and phase-constrained high-dimensional beamforming vectors. The reward function is derived directly from the drift-plus-penalty objective, ensuring compliance with both tracking error minimization and energy constraints.

LD-H-MoE Architecture

The LD-H-MoE employs two heterogeneous experts: a temporal LSTM subnetwork for event-triggered scheduling (resource allocation and sensor activation) and a spatial MLP subnetwork for complex beamforming policy under phase constraints. Crucially, gradient isolation prevents negative transfer between objectives and enforces the non-convex constant modulus hardware constraint in mmWave arrays. This separation yields stronger learning memory for scheduling without compromising spatial refinement.

Experimental Results

Tracking Accuracy

LD-H-MoE achieves time-averaged PCRB performance nearly matching the upper bound of continuous vision-based sensing, vastly outperforming radar-only and standard DRL baselines, especially under harsh mobility and channel conditions.

Figure 3: Time-averaged PCRB tracking performance of different strategies under 10 dB SNR.

Energy Budget Compliance

The architecture strictly maintains the system's long-term energy consumption below the predefined sustainable budget by dynamically modulating sensor activation and resource allocation in response to energy deficit queue states. Other approaches, such as vision-only and monolithic DRL, fail to stay within the budget due to relentless pursuit of accuracy at high computational cost.

Figure 4: Time-averaged system energy consumption against the predefined energy budget constraint ($E_{budget}$).

Queue Stability

LD-H-MoE consistently stabilizes edge computing queues, preventing backlog divergence that would severely increase AoI and jeopardize spatial certainty. Vision-only policies cause unbounded queue growth, violating the strong stability criterion.

Figure 5: Evolution of the average edge computing queue length over time, demonstrating the strong stability of the LD-H-MoE.

Robustness to RF Impairments

Under varying SNRs, LD-H-MoE dynamically adapts modality reliance. It depends on radar when the RF channel is favorable, throttling visual sensing to conserve energy, and reverts to vision when radar is corrupted by noise, ensuring low PCRB in all regimes.

Figure 6: Steady-state PCRB performance versus varying RF Signal-to-Noise Ratios (SNRs).

Implications and Future Directions

Practically, the LD-H-MoE architecture is positioned as a scalable solution for energy-limited ISAC deployments in 6G V2I contexts. Its event-driven scheduling and robustness to hardware phase constraints enable deployment in real-world systems with unstable resource availability and hostile RF environments. Theoretically, the explicit coupling between AoI, PCRB, and Lyapunov optimization opens avenues for studying more general stochastic control problems with heterogeneous constraints (e.g., latency, reliability, energy, and security). The use of specialized expert subnetworks with strict gradient isolation provides a template for future multi-task learning in control-oriented ISAC and edge AI.

Advancements in ISAC system design could extend this methodology by integrating more granular semantic representation learning, cross-modal sensor fusion, and federated reinforcement learning across distributed vehicular clusters. Investigating higher-order AoI models and adapting PCRB constraints to other uncertainty quantification metrics (e.g., Information Bottleneck, mutual information flow) represents a significant theoretical progression in real-time multimodal decision-making for wireless networks.

Conclusion

This paper presents a cross-layer Lyapunov-driven optimization for multimodal ISAC in highly mobile V2I networks. Through heterogeneous mixture-of-experts, strict gradient isolation, and physics-aware AoI modeling, the proposed system achieves superior tracking accuracy, robust energy budget compliance, queue stability, and RF resilience. The results delineate a rigorous foundation for adaptive, resource-efficient multimodal sensing-communication integration in next-generation vehicular networks.