Multi-AUV Cooperative Target Tracking Based on Supervised Diffusion-Aided Multi-Agent Reinforcement Learning

Abstract: In recent years, advances in underwater networking and multi-agent reinforcement learning (MARL) have significantly expanded multi-autonomous underwater vehicle (AUV) applications in marine exploration and target tracking. However, current MARL-driven cooperative tracking faces three critical challenges: 1) non-stationarity in decentralized coordination, where local policy updates destabilize teammates' observation spaces, preventing convergence; 2) sparse-reward exploration inefficiency from limited underwater visibility and constrained sensor ranges, causing high-variance learning; and 3) water disturbance fragility combined with handcrafted reward dependency that degrades real-world robustness under unmodeled hydrodynamic conditions. To address these challenges, this paper proposes a hierarchical MARL architecture comprising four layers: global training scheduling, multi-agent coordination, local decision-making, and real-time execution. This architecture optimizes task allocation and inter-AUV coordination through hierarchical decomposition. Building on this foundation, we propose the Supervised Diffusion-Aided MARL (SDA-MARL) algorithm featuring three innovations: 1) a dual-decision architecture with segregated experience pools mitigating nonstationarity through structured experience replay; 2) a supervised learning mechanism guiding the diffusion model's reverse denoising process to generate high-fidelity training samples that accelerate convergence; and 3) disturbance-robust policy learning incorporating behavioral cloning loss to guide the Deep Deterministic Policy Gradient network update using high-quality replay actions, eliminating handcrafted reward dependency. The tracking algorithm based on SDA-MARL proposed in this paper achieves superior precision compared to state-of-the-art methods in comprehensive underwater simulations.

• The paper presents a hierarchical MARL architecture that integrates supervised diffusion models with dual-decision mechanisms, enhancing sample efficiency and convergence in multi-AUV tracking.
• It combines DDPG-based exploitation with generative diffusion exploration and supervised behavioral cloning to robustly address non-stationarity and sparse reward challenges under hydrodynamic disturbances.
• Empirical evaluations demonstrate significant improvements in tracking accuracy, energy efficiency, and system stability, setting a new benchmark for decentralized multi-agent underwater tracking.

Supervised Diffusion-Aided Multi-Agent Reinforcement Learning for Multi-AUV Cooperative Target Tracking

Introduction

The paper "Multi-AUV Cooperative Target Tracking Based on Supervised Diffusion-Aided Multi-Agent Reinforcement Learning" (2603.29426) addresses key challenges in the application of multi-agent reinforcement learning (MARL) to cooperative target tracking in autonomous underwater vehicle (AUV) swarms. It targets the principal problems of non-stationarity in decentralized learning, sparse-reward induced inefficiency, and policy fragility under unmodeled hydrodynamic disturbances. The authors propose a hierarchical MARL architecture incorporating a supervised diffusion model for generative policy learning, yielding the SDA-MARL algorithm. The method integrates a dual-decision framework (DDPG and diffusion policy), experience quality filtering, and supervised behavioral cloning. These innovations are empirically validated over competitive baselines in multiple multi-AUV tracking regimes.

Hierarchical MARL Architecture with Supervised Diffusion

The approach is built upon a four-layer hierarchical framework, introducing clear separation of concerns between global scheduling, agent coordination, local policy synthesis, and real-time execution. System-level task decomposition and inter-agent communication are coordinated via a central Unmanned Surface Vessel-based Global Controller (USV-GC), with dedicated coordinators for spatial task allocation and aggregation of local policies. At the lowest layer, each AUV leverages a dual-decision mechanism, integrating a generative diffusion policy for exploratory action proposals and a DDPG network for exploitation, regularized via behavioral cloning.

Figure 1: Architecture of the hierarchical multi-AUV MARL system, showing the flow across global, coordination, local policy, and execution layers with dual-decision (diffusion and DDPG) on each AUV.

This architectural decomposition supports centralized training with decentralized execution, allowing system scalability and robust knowledge transfer. The replay buffer is partitioned into experience sub-pools, isolating synthetic (diffusion-generated) and real (environment-interaction) samples to mitigate experience pollution and prevent non-stationarity from destabilizing MARL training.

Supervised Diffusion-Aided MARL Algorithm Design

The SDA-MARL algorithm leverages a diffusion model to support robust generative policy learning under sparse and noisy reward feedback. Synthetic samples are generated via forward and reverse diffusion in action space, supervised by a quality-labeled experience buffer. Behavioral criteria based on trajectory displacement, target proximity, and directionality filter high-value samples through cosine similarity and convergence checks.

Figure 2: Workflow of the proposed generative MARL algorithm, covering dual decision structure, sample generation, supervised filtering, and network training.

The diffusion policy network is trained with a noise prediction loss regularized by sample quality, while the DDPG actor is updated with a weighted combination of behavioral cloning and Q-learning objectives:

• Behavior cloning loss imposes action distribution consistency between DDPG and diffusion-generated high-quality behaviors.
• Q-learning loss ensures convergence toward high-return policy modes, with a dual Q-network ensemble providing value estimates and backup targets.

This combined optimization effectively regularizes policy updates, accelerating convergence in data-limited, high-variance regimes. The employment of supervised sample filtering (instead of generic experience replay) ensures that only high-impact behaviors propagate through the learning pipeline, further stabilizing decentralized training under non-stationary dynamics.

Multi-AUV Cooperative Tracking Formulation

The problem is formulated as a multi-agent MDP, with each AUV observing a rich state representation (egomotion, local sonar observations, joint target and agent positions) and producing real-valued thruster commands via a $\tanh$-bounded policy. The action-value update and behavioral evaluation explicitly account for physical dynamics, including hydrodynamic disturbances (modeled via extended Navier-Stokes and associated drag/lift forces) and collision avoidance.

A composite reward design penalizes target misalignment, inter-AUV proximity, and environmental constraint violation. This encourages strategic, energy-efficient, and collision-free behaviors that are robust to disturbances and partial observability.

Figure 3: Tracking architecture, illustrating sensor modeling, hierarchical task decomposition, and interfacing between generative and deterministic control for robust underwater pursuit.

Empirical Evaluation and Results

Extensive experiments are conducted in high-fidelity simulation under varying swarm and target configurations. All relevant state-of-the-art algorithms (DSBM, MA-A3C, MAPPO, MASAC, MAAC, MATD3, MADDPG) are used as baselines. Evaluation metrics include convergence speed, tracking accuracy, inter-agent velocity difference (mean and stability), and path efficiency.

Figure 4: Evaluation of convergence speed across multiple multi-AUV tracking scenarios, highlighting the superior early and stable convergence of SDA-MARL compared to baselines.

Figure 5: Impact of different diffusion time steps on convergence, demonstrating the non-linear relationship between generative sample quality and training speed.

SDA-MARL consistently outperforms alternatives, exhibiting:

• Statistically significant reductions in path length and energy dissipation.
• Lower and more stable velocity differences, minimizing oscillatory corrections and power consumption.
• Improved tracking accuracy (66–68% vs. 10–47% for non-diffusion MARL).
• Robustness across swarm size and complexity, demonstrating efficient sample generation and policy generalization.

Ablation studies confirm that the removal of the diffusion model, supervised action filtering, or behavioral cloning loss degrades convergence and stability. The algorithm’s design also enhances sample efficiency, as shown by the rapid reward improvement and convergence compared to both on-policy and off-policy centralized and decentralized baselines.

Figure 6: Ablation evaluation demonstrating the necessity of the diffusion model, supervised learning, and behavioral cloning loss for achieving high and stable rewards.

Figure 7: Availability evaluation in 3D simulation—tracking sequences across system phases illustrate successful pursuit and coherent cooperation.

Theoretical and Practical Implications

The integration of supervised diffusion models into the MARL paradigm addresses core open issues in non-stationary multi-agent policy optimization, particularly under the severe exploration constraints typical of real-world AUV deployments. The approach’s explicit sample quality filtering and decoupled dual-decision structure reduce policy churning and catastrophic forgetting, yielding higher sample efficiency and convergence resilience. By eliminating hand-crafted reward shaping and robustifying to disturbance, this architecture promises improved transferability from simulation to deployment, making it suitable for energy- and safety-critical marine operations where environmental variability is high.

The proposed design also opens potential for meta-RL and transfer learning extensions, as the generative experience component could be further diversified with cross-domain skill distillation, adaptive replay prioritization, and self-supervised auxiliary tasks. The formal separation of policy generation and execution layers supports modular adaptation to new mission profiles or operational contingencies.

Conclusion

The paper presents a structured, technically rigorous approach to robust multi-AUV cooperative tracking, combining generative diffusion-based policy learning with supervised experience filtering and dual-actor optimization. The SDA-MARL framework achieves state-of-the-art results in convergence speed, energy efficiency, and multi-target tracking accuracy under realistic simulation conditions, setting a new benchmark for decentralized multi-robot learning in stochastic and partially observable domains. Its hierarchical architectural principles and sample-centric training pipeline offer a generalizable advancement for MARL system design in AI-driven autonomy.