Decentralized End-to-End Multi-AAV Pursuit Using Predictive Spatio-Temporal Observation via Deep Reinforcement Learning

Abstract: Decentralized cooperative pursuit in cluttered environments is challenging for autonomous aerial swarms, especially under partial and noisy perception. Existing methods often rely on abstracted geometric features or privileged ground-truth states, and therefore sidestep perceptual uncertainty in real-world settings. We propose a decentralized end-to-end multi-agent reinforcement learning (MARL) framework that maps raw LiDAR observations directly to continuous control commands. Central to the framework is the Predictive Spatio-Temporal Observation (PSTO), an egocentric grid representation that aligns obstacle geometry with predictive adversarial intent and teammate motion in a unified, fixed-resolution projection. Built on PSTO, a single decentralized policy enables agents to navigate static obstacles, intercept dynamic targets, and maintain cooperative encirclement. Simulations demonstrate that the proposed method achieves superior capture efficiency and competitive success rates compared to state-of-the-art learning-based approaches relying on privileged obstacle information. Furthermore, the unified policy scales seamlessly across different team sizes without retraining. Finally, fully autonomous outdoor experiments validate the framework on a quadrotor swarm relying on only onboard sensing and computing.

• The paper introduces PSTO, a predictive grid that aligns LiDAR obstacles with evader intent and teammate motion for decentralized end-to-end control using MAPPO.
• The learned policy maintains over 94% success across tested evader speeds and obstacle densities, captures targets 10–15 steps faster than the privileged OPEN baseline, and scales zero-shot from 2-vs-1 to 4-vs-1 teams.
• The system runs at about 3.87 ms latency and achieves coordinated outdoor quadrotor encirclement, but still relies on shared state estimates, a virtual boundary, and planar pursuit scenarios.

Overview

This paper presents a decentralized end-to-end multi-agent reinforcement learning (MARL) framework for multi-AAV pursuit–evasion (P–E) that maps raw LiDAR point clouds directly to continuous velocity commands. Its central contribution is the Predictive Spatio-Temporal Observation (PSTO), a fixed-resolution egocentric grid that spatially aligns dense obstacle geometry with predicted adversarial intent and teammate motion. The framework is validated in simulation against heuristic and learning-based baselines, and in fully autonomous outdoor flight on a quadrotor swarm using only onboard sensing and computation (2603.24238).

Motivation and related work

The authors position the work against two established lines of pursuit research. Classical methods—differential game theory, Voronoi partitions, time-to-intercept schemes, artificial potential fields (APF), and bio-inspired collectives—presuppose precise, noise-free state knowledge, and rigorous HJI-based solvers suffer from the curse of dimensionality while APF is prone to local minima. Modern MARL approaches, including the state-of-the-art OPEN method (2603.24238), largely adopt a state-to-control paradigm: policies consume low-dimensional state vectors, hand-crafted relative features, or privileged ground-truth obstacle geometry, implicitly assuming an upstream perception module delivers clean, fully observable states. End-to-end perception-to-control learning has been demonstrated for single-agent agile flight, but, according to the authors, no prior multi-AAV framework learns a decentralized P–E policy directly from raw LiDAR data. The paper claims this as a first, and the claim is plausible given the baselines surveyed.

The PSTO representation

The task is formulated as a Dec-POMDP with parameter-shared policies trained under the CTDE paradigm using MAPPO. Each agent's local observation comprises a 12-dimensional proprioceptive vector (attitude quaternion, body-frame linear and angular velocity, previous action) and a two-channel PSTO tensor of resolution $120 \times 6$ pixels, covering $360^\circ$ horizontal and $30^\circ$ vertical field of view via spherical projection.

The two channels are constructed as follows:

• Obstacle proximity map ($M_{\text{LiDAR}}$): raw current-frame LiDAR points are binned into grid cells with minimum-range aggregation and inverted relative to the maximum sensing range, so higher intensity denotes higher collision threat. The vertical resolution maps agents at different altitudes to distinct rows, preserving 3D separation.
• Intent heatmap ($H_{\text{intent}}$): an LSTM trajectory predictor forecasts the evader's future waypoints, defaulting to the prior prediction under occlusion; predicted points receive attraction values decaying over distance and time ($\lambda_e$ decay). Teammate states, shared over a communication network, are linearly extrapolated and assigned repulsive values with analogous temporal decay ($\lambda_t$). The two components use maximum and minimum aggregation respectively and are summed into a single channel.

The policy backbone is dual-stream: standard convolution and pooling for the dense LiDAR channel; dilated convolutions with Squeeze-and-Excitation channel attention for the sparse intent channel. The actor outputs parameters of a Beta distribution for continuous actions. The reward combines dense shaping (projected closing velocity, Gaussian coordination potential, angular-separation formation term, log-barrier obstacle penalty, constant time penalty) with sparse terminal signals (capture, collision, escape, timeout).

A notable architectural assumption deserves emphasis: to enable cooperative maneuvers, pursuers explicitly share local kinematic states over a communication network. The authors state this is a deliberate, hardware-aligned simplification that isolates cooperative strategy validation from onboard relative-perception errors. Evader tracking likewise relies on shared state estimates rather than fully onboard detection.

Training uses a progressive curriculum in NVIDIA Isaac Sim with the OmniDrones platform across 1024 parallel environments: evader speed and obstacle density increase, capture radius shrinks, and team scale expands from 2-vs-1 to 4-vs-1. Domain randomization injects Gaussian noise into proprioceptive and relative observations to support sim-to-real transfer.

Simulation results

Experiments use a 9.0 m circular arena with randomly placed static obstacles, a specialized multi-modal APF evader with tangential wall-following, and metrics averaged over 150 rollouts per condition. The comparison spans three heuristics (APF, Angelani, Janosov), the SOTA OPEN baseline, and four learning-based ablations (1-vs-1, purely reactive MLP, velocity-aware MLP, separated-input CNN).

The headline findings are:

• Robustness across conditions: PSTO maintains success rates above 94% across all tested evader speeds (0.8–1.6 m/s) and obstacle counts (0–9), without retuning. Heuristic baselines degrade sharply with speed—for example, at 1.4 m/s with no obstacles, APF and Angelani achieve 0% success. The authors note, candidly, that rising baseline success rates in dense clutter are an environmental artifact caused by obstacles trapping the evader.
• Capture efficiency versus privileged SOTA: despite OPEN's use of ground-truth obstacle states, PSTO consistently reduces capture time by 10–15 steps. The authors attribute this to PSTO's projection-level coupling of intent and geometry, which lets agents perceive traversable corridors and take efficient shortcuts, whereas OPEN's disjoint semantic tokens obscure gap feasibility and induce conservative margins. This is the paper's strongest and most notable claim: a purely onboard-sensing policy outperforms a privileged-state method on efficiency, if not uniformly on success rate (OPEN retains slightly higher SR in several dense-clutter conditions, e.g., 98.98% vs. 94.44% at 1.4 m/s, 9 obstacles).
• Ablation evidence: the purely reactive baseline trails intent-aware variants by over 15% in dense clutter, and the separated-input ablation shows that disjoint encoding of intent and geometry costs roughly 16.3% in capture time relative to PSTO, supporting the spatial-alignment hypothesis.

Scalability

Because PSTO is input-invariant with respect to team size, a single policy trained through the curriculum (final stage: 4-vs-1) is evaluated zero-shot across all team configurations. At an evader speed of 2.0 m/s—above the pursuers' own training-constrained maximum—success rates remain above 89% for 2-, 3-, and 4-vs-1 with 0, 3, and 6 obstacles. Capture time decreases monotonically with team size (e.g., 129.50 → 110.20 → 107.90 steps with no obstacles), and re-testing on 2-vs-1 after 4-vs-1 training shows no catastrophic forgetting. Trajectory analysis indicates that agents adaptively widen attack angles as team size grows, forming multi-angle encirclements that compress the evader's feasible space.

Physical experiments

The policy is deployed zero-shot on custom quadrotors (Livox Mid-360 LiDAR, Intel NUC 13) in an unstructured outdoor environment, with state estimation via Swarm-LIO. The full perception-to-control pipeline runs at approximately 3.87 ms average latency (2.61 ms encoding, 1.26 ms inference), well within the 100 ms control cycle. Two experimental accommodations are disclosed: a virtual circular boundary is superimposed as artificial point cloud wall to replicate the training arena, and, when the evader is observable, its relative state is taken directly from shared Swarm-LIO estimates as a proxy for onboard detection. The swarm executed a coordinated pincer maneuver culminating in collision-free encirclement, which the authors corroborate quantitatively with telemetry.

Limitations and open questions

The paper is explicit about its boundaries. Validation is confined to planar pursuit; extension to volumetric 3D P–E is asserted as feasible for the representation but untested. Evader tracking depends on shared state estimates from the Swarm-LIO network rather than fully onboard detection, and the communication-based sharing of teammate kinematic states is an assumption that sidesteps onboard relative-perception error—the authors themselves frame it as isolating cooperative validation from perception error rather than solving it. The physical experiment's virtual boundary and evader-state proxy similarly narrow the gap between the demonstrated system and a fully self-contained deployment. Whether the LSTM-based evader predictor and linear teammate extrapolation remain effective against more agile or evasive adversarial policies than the specialized APF evader used here is left open, as is performance under real communication degradation beyond the filtering and predictive delay compensation mechanisms described.

Conclusion

This paper contributes a unified egocentric predictive representation, PSTO, that enables a single decentralized policy to perform obstacle avoidance, interception, and cooperative encirclement from raw LiDAR data. The evidence supports three substantive claims: competitive or superior success rates against heuristics and learning baselines without privileged information, faster capture than a SOTA method that does use privileged obstacle states, and zero-shot scalability across team sizes under one policy. The outdoor quadrotor experiments demonstrate that the representation transfers to physical hardware with onboard compute. The principal caveats—planar operation, shared-state assumptions for evader and teammate information, and a virtual arena boundary in the field tests—define the remaining distance between this demonstration and fully autonomous, unrestricted 3D swarm pursuit.