MARL-AUV Benchmark Platform

• MARL-AUV Platform is an open-source benchmark integrating DI-engine with a 6-DOF simulator for persistent underwater target tracking under acoustic constraints.
• It employs a Semantic Task Graph-enhanced MARL algorithm (STG-MAPPO) that uses velocity-level action abstraction and decentralized policy execution for robust AUV swarm coordination.
• Experimental results show low tracking errors and near-zero target loss under stress tests, offering a controlled, reproducible testbed for underwater MARL research.

MARL-AUV Platform denotes an open-source benchmarking and research platform that integrates DI-engine with a six-degree-of-freedom underwater AUV target-tracking simulator for persistent underwater target tracking under narrow-band, intermittent acoustic communication, limited sensing range and partial observability, and dynamic communication topology (Zhu et al., 15 May 2026). It was introduced together with STG-MAPPO, a Semantic Task Graph-enhanced variant of Multi-Agent Proximal Policy Optimization, and provides a unified experimental protocol for fair training, testing, and comparison of representative RL and MARL algorithms on physically modeled AUV swarm tasks (Zhu et al., 15 May 2026). This suggests that the term functions both as the name of a specific benchmark and as a reference point for a broader underwater MARL design pattern spanning physics-based simulation, communication-constrained coordination, and decentralized policy execution.

1. Definition and research setting

The platform is motivated by persistent underwater target tracking in AUV swarms, where each node must sense, communicate, process local data, and make decisions under severe acoustic constraints. The task setting combines nonlinear 6-DOF AUV dynamics, narrow-band and intermittent acoustic communication, limited sensing range and partial observability, and dynamic communication topology as AUVs and targets move (Zhu et al., 15 May 2026). Within that setting, multi-agent reinforcement learning is treated as a natural candidate for distributed tracking, but existing studies are described as lacking a unified open-source platform for evaluating different MARL algorithms under six-degree-of-freedom AUV dynamics (Zhu et al., 15 May 2026). The work therefore positions MARL-AUV as, to the best of its authors’ knowledge, the first open platform that connects a public MARL training framework with physically modeled AUV swarm-based tasks (Zhu et al., 15 May 2026).

The benchmark emerges within a wider underwater autonomy literature rather than in isolation. Earlier underwater MARL work includes grid-based adaptive sampling with multiple autonomous vehicles using a shared Q-table and a central server (Rahmati et al., 2019). A distinct imitation-learning line evaluates multi-AUV formation control and obstacle avoidance on a Gazebo platform with an AUV simulator, and reports that AUVs trained via MAGAISIL can surpass provided sub-optimal expert demonstrations and reach performance close to or even better than MAGAIL with optimal demonstrations (Fang et al., 2024). These adjacent efforts indicate that underwater multi-agent learning had already diversified across adaptive sampling, formation control, and obstacle avoidance, but the DI-engine-based MARL-AUV benchmark makes physically modeled swarm target tracking the organizing abstraction (Zhu et al., 15 May 2026).

2. Platform architecture and physical modeling

The platform consists of four principal layers: a 6-DOF AUV simulator, a DI-engine integration layer, a MARL algorithm suite, and experiment-management infrastructure (Zhu et al., 15 May 2026). The simulator models the vehicle’s full pose, velocities, forces, and moments, while the integration layer wraps the simulator as a multi-agent environment in Dec-POMDP style and supports centralized critic and decentralized actor training. Configuration files expose scenario type, dynamics parameters, noise, and communication settings, and the experimental stack includes logging and evaluation tools for mean tracking error, target lost rate, action saturation, control cost, and communication quality (Zhu et al., 15 May 2026).

The vehicle model follows standard 6-DOF underwater dynamics. Pose in the inertial frame is represented as $\eta = [x, y, z, \phi, \theta, \psi]^T$, body-frame velocity as $v = [u, v, w, p, q, r]^T$, and generalized input as $\tau = [X, Y, Z, K, M, N]^T$. The kinematics are

$\dot{\eta} = J(\eta) v,$

and the dynamics are

$M \dot{v} + C(v)v + D(v)v + g(\eta) = \tau + w,$

where $M$ is the mass plus added-mass matrix, $C(v)$ captures Coriolis and centripetal effects, $D(v)$ is hydrodynamic damping, $g(\eta)$ collects gravity, buoyancy, and restoring forces, and $w$ denotes environment disturbances such as currents and noise (Zhu et al., 15 May 2026). The RL policy does not alter these equations; rather, the environment maps learned actions to control inputs.

Sensing and communication are encoded as first-class components of the environment. Each AUV is a local sensing node that observes target-relative state within sensing range and neighbor-relative information within communication range. Observation confidence is distance-modulated,

$v = [u, v, w, p, q, r]^T$0

and the communication graph is written as $v = [u, v, w, p, q, r]^T$1, with a basic distance-based link quality

$v = [u, v, w, p, q, r]^T$2

An extended link-quality model additionally incorporates attenuation, packet loss, and age of information (Zhu et al., 15 May 2026). The benchmark configuration is symmetric across agents, with parameter sharing in STG-MAPPO, and the canonical task instantiates 4 AUVs and 1 moving target (Zhu et al., 15 May 2026).

3. Task formulation and semantic representation

Swarm target tracking is formulated as a Decentralized Partially Observable Markov Decision Process,

$v = [u, v, w, p, q, r]^T$3

with centralized training and decentralized execution. During training, the critic can access global or concatenated states; during execution, each actor uses only local observation, including a semantic summary of neighborhood conditions (Zhu et al., 15 May 2026). The joint policy is factorized as

$v = [u, v, w, p, q, r]^T$4

and the return objective is the usual discounted cumulative reward (Zhu et al., 15 May 2026).

A central feature of the platform is the replacement of raw geometric state with semantic policy input. For each agent, the observation includes raw physical state, tracking diagnostics, task phase and observation-quality semantics, and a Semantic Task Graph summary. Tracking diagnostics include distance to target, tracking error, improvement trend, closing tendency, moving-average error, observation confidence, loss indicator, and normalized consecutive loss duration. Task phases are discretized into $v = [u, v, w, p, q, r]^T$5, and observation quality into $v = [u, v, w, p, q, r]^T$6, both one-hot encoded (Zhu et al., 15 May 2026).

The final semantic observation is

$v = [u, v, w, p, q, r]^T$7

Here $v = [u, v, w, p, q, r]^T$8 is a compact local graph summary over current neighbors. The platform therefore does not treat graph structure as merely a communication artifact; it converts task phase, observation reliability, link quality, neighbor tracking quality, and local role advantage into explicit policy input (Zhu et al., 15 May 2026). This design choice is important because the platform’s motivating claim is that raw geometric states and low-level force actions often struggle to represent task phases, observation reliability, link quality, and local cooperation roles (Zhu et al., 15 May 2026).

4. STG-MAPPO and the learning interface

STG-MAPPO is the platform’s reference algorithm. It is described as MAPPO augmented with a Semantic Task Graph, a velocity-level action abstraction, and a semantic, task-oriented reward (Zhu et al., 15 May 2026). Rather than using a full graph neural network, the method compresses network state into a local semantic vector,

$v = [u, v, w, p, q, r]^T$9

where $\tau = [X, Y, Z, K, M, N]^T$0 is an effective neighbor ratio, link qualities are aggregated by mean and max, $\tau = [X, Y, Z, K, M, N]^T$1 measures neighbor tracking error, $\tau = [X, Y, Z, K, M, N]^T$2 is local tracking advantage relative to neighbors, and $\tau = [X, Y, Z, K, M, N]^T$3 indicates whether the agent acts as a primary tracker or helper (Zhu et al., 15 May 2026). The local advantage term is

$\tau = [X, Y, Z, K, M, N]^T$4

The action interface is equally significant. Baselines often act directly in 6-D force/torque space, but STG-MAPPO instead adopts a 3D desired-velocity action,

$\tau = [X, Y, Z, K, M, N]^T$5

with the environment mapping this high-level velocity command to executable 6-DOF control inputs through a velocity-tracking controller and an attitude-stabilization moment controller (Zhu et al., 15 May 2026). Actions are also smoothed by rate limiting. The stated rationale is that learning directly in raw force/torque space is difficult because of strong coupling and saturation, whereas the velocity abstraction preserves full 6-DOF dynamics while simplifying the decision space (Zhu et al., 15 May 2026).

The learning core retains standard PPO/MAPPO machinery. The actor is an MLP $\tau = [X, Y, Z, K, M, N]^T$6 over semantic input, and the critic is a centralized value network over concatenated semantic states. The algorithm suite exposed through DI-engine includes cooperative on-policy methods such as MAPPO and HAPPO, off-policy actor-critic methods such as MASAC, MADDPG, and MATD3, and a value-based MADQN baseline (Zhu et al., 15 May 2026). Because all methods share the same environment, training budget, and metric suite, the platform is intended as a controlled benchmark rather than an algorithm-specific simulator (Zhu et al., 15 May 2026).

5. Experimental protocol and reported behavior

The unified benchmark uses a standard scenario of 4 AUVs and 1 target with 500-step episodes, typically trained for $\tau = [X, Y, Z, K, M, N]^T$7 environment steps and evaluated every 5000 steps (Zhu et al., 15 May 2026). Difficulty is partitioned into medium and hard settings, and the platform defines additional stress tests involving faster targets, farther initial AUV-target separations, limited sensing range, degraded communication, and a combined-stress condition (Zhu et al., 15 May 2026). Standard metrics are evaluation return, mean tracking error in kilometers, tail mean target distance in kilometers in the final portion of the episode, target lost rate, action saturation rate, control cost, and communication-quality diagnostics (Zhu et al., 15 May 2026).

Within that protocol, STG-MAPPO is reported to achieve very low tail mean target distance, approximately $\tau = [X, Y, Z, K, M, N]^T$8–$\tau = [X, Y, Z, K, M, N]^T$9 km, and near-zero target lost rate, often exactly $\dot{\eta} = J(\eta) v,$0, in both medium and hard scenarios and under combined stress (Zhu et al., 15 May 2026). The same section reports higher evaluation returns, lower variance, and smoother convergence than representative baselines. MAPPO and MADDPG can perform reasonably in nominal conditions, but degrade under stress, retain non-zero lost rates, and exhibit larger variance across seeds; off-policy baselines such as MASAC, MATD3, and MADQN struggle more severely in tracking accuracy and target-loss metrics (Zhu et al., 15 May 2026).

The ablation results attribute the gain to the combination of semantic state, STG neighbor summaries, semantic reward, and velocity abstraction. A velocity-only variant stabilizes low-level control and virtually eliminates target loss, but still leaves tail distance significantly higher, around $\dot{\eta} = J(\eta) v,$1 km. A semantic-state variant improves over velocity-only input, yet remains clearly below the full model. The full configuration, denoted STG-MAPPO-full, yields the highest return and near-zero tail distance and lost rate (Zhu et al., 15 May 2026). The reported interpretation is that velocity-level abstraction alone helps stabilize control but does not yield precise tracking, while semantic state alone helps but is insufficient without explicit network-aware summarization and reward design (Zhu et al., 15 May 2026).

6. Practical scope, codebase, and stated limitations

The released codebase includes environment definitions for a REMUS-like 6-DOF dynamics model, target motion models and scenario definitions, acoustic communication and sensing models, DI-engine wrappers, STG-MAPPO implementation, baseline scripts and configurations, metric computation, convergence plotting, stress-test experiments, and GEBCO-based visualization of swarm trajectories over realistic bathymetry (Zhu et al., 15 May 2026). The repository is public at https://github.com/dasjsaj/MARL-AUV (Zhu et al., 15 May 2026). The platform is also described as one-command and reproducible, which places it closer to a benchmark infrastructure than to a single simulation script (Zhu et al., 15 May 2026).

The platform’s stated limitations are narrow but explicit. Team size is fixed in the paper at 4 AUVs; dynamic population adaptation is not fully addressed in the released code. The acoustic channel models range, attenuation, loss, and information aging, but full ocean acoustics, including multipath and Doppler, are abstracted. The evaluation is simulation only, with no direct hardware integration, and the primary task is target tracking, so other underwater missions may require custom reward and semantic-state design (Zhu et al., 15 May 2026). This suggests that MARL-AUV is best understood as a controlled experimental substrate for underwater swarm learning rather than as a complete field-deployment stack.

7. Relation to adjacent underwater MARL platforms and stacks

The broader literature shows several neighboring architectures that illuminate what a MARL-AUV platform can mean in practice.

Work	Core substrate	Relation to MARL-AUV
UW-MARL (Rahmati et al., 2019)	Shared-Q adaptive sampling on a grid	Early underwater MARL framework with central server and communication protocol
FATHOM-Net / DSBM (Wang et al., 2024)	SDN-style HSARL with USV-GC, LC-AUVs, ET-AUVs	Hierarchical control-plane/data-plane decomposition for multi-target tracking
Scene-adaptive EI (Tian et al., 28 Mar 2026)	Three-layer embodied-intelligence architecture	Beacon-based communication-control model and dual-path critic SA-MARL
SDA-MARL (Ma et al., 31 Mar 2026)	Four-layer hierarchy with diffusion and behavioral cloning	Dual-decision architecture and segregated experience pools for robust tracking
SafeDiver (Deng et al., 15 Sep 2025)	AUV-USV-diver relay system	Application-driven CTDE stack combining MAPPO movement and MATD3 relay selection
CoUGARs / MeCO (Durrant et al., 11 Nov 2025, Widhalm et al., 13 Mar 2025)	Open hardware with ROS 2, Docker, simulator integration	Fieldable low-cost or medium-cost AUV substrates for future MARL deployment

Within this surrounding ecosystem, MARL-AUV occupies the role of a public, physically modeled benchmark, whereas several related systems emphasize hierarchical networking, hardware modularity, or mission-specific control. FATHOM-Net frames the swarm as an underwater ad-hoc network under a hierarchical software-defined reinforcement-learning architecture and adds Dynamic-Switching Attention and Dynamic-Switching Resampling to accelerate convergence and avoid local optima (Wang et al., 2024). Scene-adaptive embodied intelligence further formalizes a three-layer perception-decision-execution framework with beacon-based communication and a dual-path critic that fuses scene-specific and general safety objectives (Tian et al., 28 Mar 2026). SDA-MARL extends the hierarchy to four layers and introduces supervised diffusion-aided sample generation and behavioral cloning to mitigate non-stationarity, sparse rewards, and disturbance fragility (Ma et al., 31 Mar 2026).

Other works emphasize application specialization or hardware grounding rather than an open benchmark interface. SafeDiver defines a concrete AUV-USV-diver system with MAPPO for cooperative AUV movement and MATD3 for dynamic relay selection, but is described as application-driven rather than a general platform API (Deng et al., 15 Sep 2025). CoUGARs introduces a low-cost, configurable AUV fleet with ROS 2, Docker, acoustic cooperative localization software, and tight HoloOcean integration, while MeCO provides an open-source AUV with ROS 2, micro-ROS, Docker, and a Unity-based simulator; both are strong physical substrates for MARL yet are not themselves presented as MARL benchmarks (Durrant et al., 11 Nov 2025, Widhalm et al., 13 Mar 2025). Non-MARL but architecturally relevant counterparts include a five-tiered multi-AUV route planner for stochastic ocean environments (Zhang et al., 2023) and autonomous marine-debris mapping with a modified BlueROV2 Heavy using stereo imaging and underwater hyperspectral sensing (Fossum et al., 2022). Taken together, these systems indicate that the contemporary MARL-AUV landscape spans benchmark platforms, hierarchical coordination stacks, imitation-learning environments, and open hardware, with MARL-AUV (Zhu et al., 15 May 2026) serving as the most explicit public benchmark centered on 6-DOF AUV swarm target tracking.