Multimodal Large Language Model Enabled Robust Beamforming for HAP Downlink Communications

Abstract: Small changes in high altitude platform (HAP) attitude can cause significant deviations in HAP downlink beam directions, thereby severely degrading HAP downlink communication performance. In this paper, we develop a multimodal LLM enabled beamforming framework to achieve robust HAP downlink communications.Specifically, we design a vision-language LLM (VL-LLM) that learns from multivariate flight telemetry to forecast short-term HAP attitudes under platform shaking and support delay-aware proactive beam steering.We design an offline forecast-error calibration procedure to obtain upper bounds on forecast errors and improve the reliability of proactive analog beam steering.Based on the attitude forecasts, we proactively update the analog beamformer and propose a QoS-driven beamforming and admission method with a lightweight feasibility-enforcement step to satisfy instantaneous transmit-power and QoS requirements.Simulation results indicate that the designed VL-LLM can accurately capture changes in the HAP attitude and the proposed beamforming method achieves a 22.1% higher user service ratio and a 12.5% higher sum-rate than representative baselines.The measured mean and p99 total latencies are 36.24 ms and 40.13 ms, respectively, supporting practical delay-aware deployment.

• The paper introduces a novel VL-LLM framework that forecasts HAP attitude for proactive, hybrid beamforming under wind-induced perturbations.
• It employs offline calibration and a KKT-guided digital beamforming protocol to enforce robust main-lobe performance and strict QoS constraints.
• Simulations show marked improvements in user service ratio, sum-rate, and latency when compared to reactive and conventional beam management methods.

Multimodal LLM-Enabled Robust Beamforming for HAP Downlink Communications

Introduction and Motivation

High-Altitude Platforms (HAPs) are integral to beyond-5G/6G networks, providing quasi-stationary, wide-area coverage from the lower stratosphere. However, maintaining reliable mmWave downlink communication from HAPs is fundamentally challenged by wind-induced platform shaking—small attitude changes substantially impact beam alignment, sharply degrading array gain and user throughput. Conventional beam management, including iterative tracking and classical learning-based methods, remains largely reactive. They lag in providing both stringent robustness guarantees and low-latency, multiuser quality-of-service (QoS) provisioning under real-world actuation delays.

This paper introduces a multimodal LLM-enabled beamforming framework to address robust HAP downlink communications under platform shaking scenarios. The core innovation is a vision-language LLM (VL-LLM) that leverages multivariate flight telemetry for short-term attitude forecasting, enabling delay-aware, proactive hybrid beamforming integrated with a robust, feasibility-enforced digital beamforming protocol. The study also proposes an offline calibration to empirically bound forecast-induced uncertainty, thus certifying analog beamforming actions with explicit main-lobe gain loss thresholds. Simulation demonstrates the system’s superiority in forecasting accuracy, user service ratio, sum-rate, and end-to-end inference latency.

System Model and Problem Formulation

The HAP downlink system uses a $M_x \times M_y$ Uniform Planar Array (UPA) to serve $K$ ground users. Due to persistent wind and turbulence, actual beam directions deviate from ideal, degrading the main-lobe gain (Figure 1).

Figure 1: Downlink beam misalignment for a HAP under wind-induced attitude shaking.

The system is slotted in time. For each slot, a sequence of short-term attitude forecasts is generated; only post-latency horizons ($h > d$) are actionable for beam alignment due to inherent actuation and computation delays. Hybrid precoding is employed: analog beamformers are proactively updated per forecast, while digital beamforming and user admission are optimized online using the effective post-analog channel.

The error resulting from attitude forecast ($\Delta \boldsymbol{\omega}_\tau$) is modeled as a rotation difference in $\mathrm{SO}(3)$, and its effect on array gain is quantified via a quadratic loss in the direction-cosine domain. Forecast errors are statistically or deterministically bounded using an independent calibration set, then translated to robust beamforming constraints. Admissibility under power and instantaneous QoS constraints is enforced via a two-stage optimization: maximizing the admission ratio, then refining energy efficiency on the admitted set.

VL-LLM for Multimodal Attitude Forecasting

Forecasting short-term HAP attitude is intrinsically a multivariate time-series problem subject to noise, nonlinearity, and nonstationarity. Existing methods, whether LLM-based (e.g., token-sequence reprogramming) or vision-based (e.g., temporal convolutions), are limited by modality mismatch and lack of structured prior injection.

This framework blends:

• Vision Features: Telemetry is rendered into pseudo-images capturing local, periodic, and frequency-domain patterns, processed via a vision backbone.
• Cross-Variable Adaptation: Dedicated adapter tokens summarize cross-variable dependencies via multi-head attention.
• Language Conditioning: Structured prompts inject domain/task instructions and window-level statistics (trend, periodicity, dispersion) into the LLM, leveraging its powerful sequence modeling and interpretability advantages.

The multimodal representation fed into a frozen LLM backbone yields a multi-horizon attitude forecast, targeting only the actionable windows impacted by system delay.

Figure 2: Attitude trajectories and forecasting deviations for yaw, pitch, and roll.

Training emphasizes direct multi-step outputs, geometry-aware losses, and explicit smoothness constraints. Offline calibration—by empirical tail-quantile extraction on the forecast errors—yields robust main-lobe gain loss guarantees.

Robustness Calibration and Beamforming

Forecast-induced uncertainty is explicitly mapped to array gain loss through quadratic approximations. Offline, the empirical $(1-\rho)$-quantile of target-window errors yields a deterministic robustness radius $\delta_\omega$. The robustness constraint is then $L_k^2 \delta_\omega^2 \leq \epsilon$, ensuring the analog steering’s main-lobe loss does not exceed $\epsilon$ uniformly in the forecast window with probability at least $1-(\rho+\rho_s)$. A moment-based (mean/covariance) alternative supports statistical certification.

QoS-driven digital beamforming is addressed by:

• Deploying a learning-assisted forecaster that outputs both soft admission scores and KKT-aligned auxiliary variables.
• Relying on closed-form, KKT-guided digital beamforming reconstruction, with bisection on the dual variable for the power constraint.
• Employing a strict feasibility repair loop based on worst-normalized QoS violation, ensuring that all admitted users meet instantaneous QoS and power constraints after each update.

This procedure eliminates the need for full-scale iterative optimization per slot, yielding a low-complexity, repairable, and certifiably feasible solution path.

Numerical Results

Attitude Forecasting: VL-LLM yields significant improvements over vision-only and LLM-only time-series baselines in MAE and RMSE, with robust performance maintained even under imposed IMU noise and latency constraints. At forecast horizon $K$0, 97–99% of errors are within $K$1 for each axis.

Beamforming Gains: Forecast-based compensation outperforms both reactive and “no compensation” modes in QoS admission ratio (QAR) and sum-rate, with a 22.1% higher user service ratio and 12.5% higher sum-rate compared with baselines. Mean and p99 full model latencies are 36.24 ms and 40.13 ms, respectively, verifying practical deployment. The system also displays resilience to NLoS conditions or challenging user distributions.

Solver Architecture: The KKT-guided repair approach achieves near-unity feasibility and superior QAR and sum-rate compared to diffusion-based, RL-based, or classical solvers. Offline calibration yields direct control of the conservativeness versus coverage trade-off, without degrading core forecast accuracy.

Implications and Future Directions

Practically, coupling high-fidelity, low-latency multimodal attitude forecasting with robust, learning-assisted beamforming delivers certifiable, real-time gains in volatile, dynamic HAP links. The separation of the offline robustness calibration from online per-slot optimization ensures computational scalability and resilience under system delay or environmental uncertainty. The approach exemplifies robust AI/ML system design where statistical error controls are rigorously propagated into safety-critical decisions.

Theoretically, the results open avenues for integrated forecasting-beamforming frameworks leveraging foundation models, domain-structured prompting, and composable robustness certificates. The modular calibration architecture is extensible to broader aerial, satellite, or cooperative networked systems requiring online uncertainty quantification and tractable robustness enforcement.

Conclusion

This work demonstrates the first robust forecast-then-beamform system for HAP downlink under real-world platform shaking. The fusion of VL-LLM-based attitude forecasting, offline-calibrated main-lobe loss guarantees, and learning-assisted digital beamforming with strict feasibility enforcement achieves both strong numerical gains and certifiable constraints on QoS and power in dynamic channels. Future research directions include handling more severe NLoS scenarios, time-varying sensing uncertainty, and extension to multi-HAP or cooperative network environments.

---

Reference: "Multimodal LLM Enabled Robust Beamforming for HAP Downlink Communications" (2604.09017)