Robust Multi-Agent Reinforcement Learning for Small UAS Separation Assurance under GPS Degradation and Spoofing

Abstract: We address robust separation assurance for small Unmanned Aircraft Systems (sUAS) under GPS degradation and spoofing via Multi-Agent Reinforcement Learning (MARL). In cooperative surveillance, each aircraft (or agent) broadcasts its GPS-derived position; when such position broadcasts are corrupted, the entire observed air traffic state becomes unreliable. We cast this state observation corruption as a zero-sum game between the agents and an adversary: with probability R, the adversary perturbs the observed state to maximally degrade each agent's safety performance. We derive a closed-form expression for this adversarial perturbation, bypassing adversarial training entirely and enabling linear-time evaluation in the state dimension. We show that this expression approximates the true worst-case adversarial perturbation with second-order accuracy. We further bound the safety performance gap between clean and corrupted observations, showing that it degrades at most linearly with the corruption probability under Kullback-Leibler regularization. Finally, we integrate the closed-form adversarial policy into a MARL policy gradient algorithm to obtain a robust counter-policy for the agents. In a high-density sUAS simulation, we observe near-zero collision rates under corruption levels up to 35%, outperforming a baseline policy trained without adversarial perturbations.

• The paper introduces a closed-form, value-gradient adversarial perturbation for bounded GPS corruption, with a second-order error bound that enables efficient robust MARL training without iterative attack optimization.
• The robust PPO policy combines perturbation-invariance and teacher-anchor KL regularization, preserving nominal performance while providing a theoretical performance-degradation bound that grows linearly with corruption probability.
• Simulation results show near-zero NMACs up to approximately 35% corrupted observations and about 5 NMACs per episode at 95% corruption versus roughly 18 for nominal PPO, while ablations show both regularizers are necessary.

Problem and motivation

This paper addresses autonomous separation assurance for small Unmanned Aircraft Systems (sUAS) operating under corrupted GPS-derived state information. The authors' central observation is that sUAS cooperative surveillance differs structurally from commercial aviation surveillance: because Remote ID broadcasts are themselves derived from onboard GPS receivers, a single corruption source can perturb the entire joint traffic observation—the ownship's state and every intruder's broadcast simultaneously. They formalize this as an $R$-contamination model over full-state observations: with probability $R$, the agent's observation of the next true state is replaced by an adversarially chosen point within a componentwise bounded uncertainty set $\Omega(\bm S)$ calibrated to empirically reported urban GNSS errors (multipath errors of 10–60 m in urban canyons; field-demonstrated spoofing success rates of 5–40%). The paper's positioning against prior work is explicit: existing robust RL handles single-agent observation attacks [zhang_robust_rl_state] or $R$-contaminated transitions [wang2022policygradientmethodrobust], while CBF-based multi-agent safety methods assume known dynamics or true-state access—assumptions violated precisely when GPS is unreliable.

Closed-form adversarial perturbation

The core technical device is a first-order characterization of the inner minimization in the robust value objective, $$\min_{\bm\Xi \in \Omega(\bm S_{t+1})} V^\pi(\bm\Xi)$$. Under differentiability of the value function (Assumption 1), linearizing $V^\pi$ around the true next state reduces the adversary to a box-constrained linear program whose solution is componentwise sign-based:

$$\bm\Xi^{\star,\mathrm{FO}} = \bm S_{t+1} - \bm\kappa \odot \mathrm{sign}\bigl(\nabla V^\pi(\bm S_{t+1})\bigr).$$

Because this is closed form, it bypasses iterative adversarial training (e.g., PGD-style inner loops) entirely and evaluates in linear time in the state dimension—a practical advantage for per-transition adversary construction during MARL rollouts. The corresponding worst-case value drop is approximately $\|\bm\kappa \odot \nabla V^\pi(\bm S_{t+1})\|_1$ (Corollary 1).

The approximation quality is quantified in Theorem 2: assuming an $L_V$-Lipschitz gradient on the value function near $\bm S_{t+1}$, the gap between the exact worst-case value and the first-order estimate is bounded by $R$0, i.e., second-order in the corruption radius. This is a local guarantee only—it presumes smooth neural value approximators on bounded neighborhoods—and the bound scales quadratically with the uncertainty magnitude, so its tightness degrades for large $R$1. The authors do not empirically measure this gap, leaving open how accurate the surrogate is at the largest tested corruption levels.

Robust PPO with KL regularization

The adversary is integrated into a two-phase PPO pipeline with centralized training and decentralized execution over a shared actor-critic network (ownship/intruder encoders aggregated via multi-head attention). Phase 1 pretrains a nominal teacher policy under clean observations ($R$2); phase 2 trains a new policy on trajectories where observations are probabilistically replaced by the first-order adversarial perturbation computed from the frozen teacher critic $R$3. Freezing the teacher prevents the adversary from co-adapting with the policy being trained.

Two complementary KL regularizers are added to the clipped PPO objective: a perturbation-invariance term penalizing divergence between the current policy's outputs on clean versus adversarial observations, and a teacher-anchor term penalizing drift from the pretrained nominal policy on clean observations. The theoretical payoff is Proposition 3: if the expected invariance KL is bounded by $R$4 and Q-values are bounded by $R$5, then the expected one-step value loss from acting on a corrupted observation satisfies

$R$6

via Pinsker's inequality and Jensen's inequality. Corollary 4 extends this over the horizon: total discounted degradation is bounded by $R$7, i.e., performance loss grows at most linearly in the corruption probability $R$8. The regularization weight $R$9 thus directly controls the robustness–performance tradeoff through the budget $\Omega(\bm S)$0. Note that these bounds concern one-step decision degradation propagated through discounting; they do not account for compounding distribution shift across long horizons, which the paper does not address.

Experimental results

Evaluation uses BlueSky with a structured en-route airspace (~10 km routes crossing and merging at waypoints), Poisson arrivals with 30 s minimum headway, Amazon MK30-like vehicles (20 m/s cruise, [7.5, 36] m/s admissible range), discrete ±5-knot speed actions, NMAC threshold 100 m, detection radius 500 m, and corruption bounds $\Omega(\bm S)$1 m, $\Omega(\bm S)$2 m/s, $\Omega(\bm S)$3. Training spans $\Omega(\bm S)$4 steps with a curriculum on $\Omega(\bm S)$5; results average 100 episodes.

The headline result is that the robust policy maintains near-zero NMACs up to $\Omega(\bm S)$6, while the nominal policy holds only to $\Omega(\bm S)$7 before deteriorating sharply—reaching roughly 18 NMACs per episode at $\Omega(\bm S)$8 versus about 5 for the robust policy. Minimum separation distances remain consistently larger for the robust policy, with the gap widening as $\Omega(\bm S)$9 grows. At $R$0 both policies perform comparably, indicating no nominal-capability sacrifice from robust training.

The ablation study yields a notable negative finding: invariance regularization without anchoring destabilizes training outright, producing high NMAC counts even at $R$1—enforcing output consistency without a stable reference collapses the learned representation. Anchoring alone preserves nominal competence but degrades sharply beyond $R$2. Only the combination achieves consistent performance, so neither regularizer is individually sufficient.

Limitations and open questions

Several limitations are stated or evident. The guarantees rest on smoothness assumptions (differentiable, Lipschitz-gradient value functions) that may fail for the attention-based architecture used in practice, and the second-order accuracy claim is unverified empirically. The corruption model assumes independent per-step contamination with a shared bounded perturbation; real GPS spoofing exhibits temporally correlated, structured biases that the elementwise sign attack does not represent. Rewards depend on true states during offline training—an assumption the authors justify but which would not hold in online learning. Evaluation is confined to homogeneous agents, a single structured airspace, and simulation only; generalization to heterogeneous fleets, wind variability beyond the modeled components, and hardware-in-the-loop validation remain untested. Finally, the horizon-level bound in Corollary 4 ignores trajectory-level distribution shift, so the gap between the proven bound and observed behavior at high $R$3 (where both policies degrade) is not characterized.

Conclusion

The paper contributes a tractable, theoretically grounded route to observation-robust MARL for sUAS separation assurance: a closed-form, second-order-accurate worst-case perturbation derived from the value gradient, combined with dual KL regularizers that yield a provable linear-in-$R$4 bound on performance degradation. Simulation results support the approach, showing near-zero collision rates under corruption up to 35% and graceful degradation thereafter, with the ablation clarifying that anchoring is essential to make invariance regularization trainable. The main open questions concern empirical validation of the approximation bounds, correlated and structured spoofing models, and transfer beyond simulated, homogeneous traffic.