When Adaptive Rewards Hurt: Causal Probing and the Switching-Stability Dilemma in LLM-Guided LEO Satellite Scheduling

Abstract: Adaptive reward design for deep reinforcement learning (DRL) in multi-beam LEO satellite scheduling is motivated by the intuition that regime-aware reward weights should outperform static ones. We systematically test this intuition and uncover a switching-stability dilemma: near-constant reward weights (342.1 Mbps) outperform carefully-tuned dynamic weights (103.3+/-96.8 Mbps) because PPO requires a quasistationary reward signal for value function convergence. Weight adaptation-regardless of quality-degrades performance by repeatedly restarting convergence. To understand why specific weights matter, we introduce a single-variable causal probing method that independently perturbs each reward term by +/-20% and measures PPO response after 50k steps. Probing reveals counterintuitive leverage: a +20% increase in the switching penalty yields +157 Mbps for polar handover and +130 Mbps for hot-cold regimes-findings inaccessible to human experts or trained MLPs without systematic probing. We evaluate four MDP architect variants (fixed, rule-based, learned MLP, finetuned LLM) across known and novel traffic regimes. The MLP achieves 357.9 Mbps on known regimes and 325.2 Mbps on novel regimes, while the fine-tuned LLM collapses to 45.3+/-43.0 Mbps due to weight oscillation rather than lack of domain knowledge-output consistency, not knowledge, is the binding constraint. Our findings provide an empirically-grounded roadmap for LLM-DRL integration in communication systems, identifying where LLMs add irreplaceable value (natural language intent understanding) versus where simpler methods suffice.

• The paper demonstrates that frequently changing adaptive rewards impede PPO convergence by introducing non-stationarity in LEO satellite scheduling.
• It shows that near-constant reward weights via MLPs yield significantly higher throughput and stability compared to dynamic LLM-based strategies.
• The study advocates separating semantic intent parsing from numerical reward mapping to ensure reliable, efficient DRL performance in satellite networks.

Causal Probing and the Switching-Stability Dilemma in LLM-Guided LEO Satellite Scheduling

Introduction and Motivation

Adaptive reward design in reinforcement learning for LEO satellite scheduling is often motivated by the intuition that regime-aware, dynamically-updated reward weights should outperform static alternatives. However, this study systematically refutes such intuition by demonstrating that the non-stationarity induced by adaptive rewards fundamentally impedes convergence for PPO-based DRL agents. Empirically, the paper establishes that near-constant reward weights result in significantly higher sum rates compared to even carefully-optimized, dynamically-adjusted weights. The adaptive control of reward weights by an MDP architect—implemented via fixed rules, MLP, or LLM—was rigorously interrogated, with a particular focus on generalization to novel traffic regimes and the dynamic stability of reward generation mechanisms.

Architecture: Three-Timescale Adaptive Scheduling

The system decomposes LEO satellite resource scheduling into three hierarchically-resolved timescales:

• Strategic (LLM): High-level parsing of operator intent (natural language), which occurs rarely (hours).
• Tactical (MLP): Fast mapping of observed KPIs to numerical reward weights, reactive to regime changes on the scale of minutes.
• Operational (DRL): Second-to-second beam scheduling via PPO, contingent on a fixed reward function.

  Figure 1: The three-timescale adaptive architecture decouples intent inference, reward adaptation, and real-time DRL beam scheduling.

The key insight is that LLMs are leveraged solely where necessary, i.e., in semantically rich tasks (intent parsing), while MLPs are used for high-frequency, numerically-stable reward weight mapping. This division enforces architectural stability and efficiently exploits the respective strengths of each model class.

Experimental Methodology

Simulations were conducted on a Ka-band LEO satellite model with 19 beams and realistic channel effects. Four canonical and three novel traffic regimes were considered, with regime change detection via CUSUM on traffic KPIs. Four variants of MDP architects were assessed: Fixed, Rule-based, MLP, and Fine-tuned LLM (Qwen3-4B with LoRA). The LLMs generated structured numeric outputs from natural language and KPI text descriptions using domain-specific RAG augmentation.

Empirical Analysis and Performance Comparison

Reward Switching-Stability Dilemma

A central contribution is the empirical demonstration of a "switching-stability dilemma": PPO requires quasi-stationary reward signals for value function convergence. When reward weights are adapted frequently—even if optimally—the convergence process is repeatedly destabilized, resulting in inferior overall sum rates. Stationary reward functions, even if sub-optimal for a specific regime, consistently yield higher aggregate throughput (342.1 Mbps with near-constant weights versus 103.3 Mbps with dynamic probe-optimal switching).

Figure 2: Mean sum rate comparison shows that the MLP architect achieves highest throughput and stable generalization, while FT-LLM collapses on novel regimes due to oscillatory weight generation.

Architect Variant Comparison

• MLP Architect: Achieved the best results in both aggregate throughput and stability, generalizing even to novel regimes at 325.2 Mbps.
• Rule-based Architect: Close to MLP on aggregate but lacks flexibility for unseen regimes.
• Fine-tuned LLM Architect: Underperforms dramatically, collapsing to 45.3 Mbps on novel regimes due to high output variance and oscillation in numerical weight proposals that PPO cannot absorb.
• Fixed Baselines: Outperform dynamic LLM-based adaptation unless reward weights are extremely well-matched to the regime, highlighting the penalty incurred by non-stationarity.

Single-variable causal probing revealed unexpectedly high leverage for the switching penalty weight ($w_s$): a $+20\%$ perturbation in $w_s$ delivered $+157$ Mbps and $+130$ Mbps on polar handover and hot-cold regimes, respectively—sensitivities not recoverable by human expertise or naively trained MLPs.

Weight Generation Consistency

The analysis of reward weight trajectories reveals high consistency and narrow-band variation for MLP-generated weights, enabling stable PPO learning. By contrast, the LLM architect demonstrates bimodal and highly-oscillatory behavior (CV $>$ 4.6 times MLP), with reward weights swinging between extremes each time the LLM is queried—causing repeated value function restarts and drastic reductions in achieved sum rates.

Figure 3: Outage weight time series illustrate that MLP outputs are stable, while fine-tuned LLM outputs undergo high-variance, bimodal oscillations after regime transitions, undermining PPO convergence.

Theoretical and Practical Implications

The findings rigorously delimit the space where LLMs are practically beneficial in communication system RL architectures:

• LLMs excel at natural language and semantic parsing but are unreliable as sources of numerically-stable, high-frequency reward weights.
• MLPs should be favored for real-time numerical weight mapping due to their output determinism, computational efficiency (< 1ms per inference), and superior generalization given adequate training diversity.
• Adaptive reward design is not free of cost; its additional non-stationarity can actively degrade DRL performance beyond that of static policies unless architectures provide isolation or stable re-initialization mechanisms for value estimators.

Approaches such as value function isolation, per-regime meta-learning, or tightly-constrained historical RAG anchoring are necessary to bridge the adaptation-convergence divide.

Deployment and Future Developments

For deployment, latency and hardware resource constraints further reinforce the strategic separation: LLM intent parsing can be executed off-line or in the cloud at low frequency, while lightweight MLPs enable on-device, low-latency adaptation. Engineering attention must focus on reward sign conventions and ensuring robust fallback to rule-based or static methods when reward weight proposals are erratic.

Promising future directions include:

• Performance-grounded RAG anchoring: Using historical KPI-performance tuples to constrain LLM or LLM-MLP hybrids, enforcing output consistency.
• Meta-learning and value function modularization: Retaining separate value estimators for each regime or initializing value predictors upon reward switches.
• Scalability to multi-satellite and intent negotiation: Delegating semantic intent parsing to LLMs while keeping individual satellite control loops numerically stable.

Conclusion

The study provides a technically rigorous and empirically-founded roadmap for LLM-DRL integration in satellite scheduling, unambiguously identifying both the critical strengths and the binding limitations of LLM output consistency. The three-timescale hybrid—LLM for semantic intent, MLP for fast, stable weight mapping, and DRL for real-time control—represents a practical solution. Resolving the switching-stability dilemma is now recognized as the fundamental challenge in adaptive RL system design for communication networks.