---
title: MARL-AUV Benchmark Platform
url: https://www.emergentmind.com/topics/marl-auv-platform
type: topic
---

# MARL-AUV Benchmark Platform

MARL-AUV Platform denotes an open-source benchmarking and research platform that integrates DI-engine with a six-degree-of-freedom underwater AUV target-tracking simulator for persistent underwater target tracking under narrow-band, intermittent acoustic communication, limited sensing range and partial observability, and dynamic communication topology [2605.15528]. It was introduced together with STG-MAPPO, a Semantic Task Graph-enhanced variant of Multi-Agent Proximal Policy Optimization, and provides a unified experimental protocol for fair training, testing, and comparison of representative RL and MARL algorithms on physically modeled AUV swarm tasks [2605.15528]. This suggests that the term functions both as the name of a specific benchmark and as a reference point for a broader underwater MARL design pattern spanning physics-based simulation, communication-constrained coordination, and decentralized policy execution.

## 1. Definition and research setting

The platform is motivated by persistent underwater target tracking in AUV swarms, where each node must sense, communicate, process local data, and make decisions under severe acoustic constraints. The task setting combines nonlinear 6-DOF AUV dynamics, narrow-band and intermittent acoustic communication, limited sensing range and partial observability, and dynamic communication topology as AUVs and targets move [2605.15528]. Within that setting, multi-agent reinforcement learning is treated as a natural candidate for distributed tracking, but existing studies are described as lacking a unified open-source platform for evaluating different MARL algorithms under six-degree-of-freedom AUV dynamics [2605.15528]. The work therefore positions MARL-AUV as, to the best of its authors’ knowledge, the first open platform that connects a public MARL training framework with physically modeled AUV swarm-based tasks [2605.15528].

The benchmark emerges within a wider underwater autonomy literature rather than in isolation. Earlier underwater MARL work includes grid-based adaptive sampling with multiple autonomous vehicles using a shared Q-table and a central server [1911.04096]. A distinct imitation-learning line evaluates multi-AUV formation control and obstacle avoidance on a Gazebo platform with an AUV simulator, and reports that AUVs trained via MAGAISIL can surpass provided sub-optimal expert demonstrations and reach performance close to or even better than MAGAIL with optimal demonstrations [2401.11378]. These adjacent efforts indicate that underwater multi-agent learning had already diversified across adaptive sampling, formation control, and obstacle avoidance, but the DI-engine-based MARL-AUV benchmark makes physically modeled swarm target tracking the organizing abstraction [2605.15528].

## 2. Platform architecture and physical modeling

The platform consists of four principal layers: a 6-DOF AUV simulator, a DI-engine integration layer, a MARL algorithm suite, and experiment-management infrastructure [2605.15528]. The simulator models the vehicle’s full pose, velocities, forces, and moments, while the integration layer wraps the simulator as a multi-agent environment in Dec-POMDP style and supports centralized critic and decentralized actor training. Configuration files expose scenario type, dynamics parameters, noise, and communication settings, and the experimental stack includes logging and evaluation tools for mean tracking error, target lost rate, action saturation, control cost, and communication quality [2605.15528].

The vehicle model follows standard 6-DOF underwater dynamics. Pose in the inertial frame is represented as $\eta = [x, y, z, \phi, \theta, \psi]^T$, body-frame velocity as $v = [u, v, w, p, q, r]^T$, and generalized input as $\tau = [X, Y, Z, K, M, N]^T$. The kinematics are
$$
\dot{\eta} = J(\eta) v,
$$
and the dynamics are
$$
M \dot{v} + C(v)v + D(v)v + g(\eta) = \tau + w,
$$
where $M$ is the mass plus added-mass matrix, $C(v)$ captures Coriolis and centripetal effects, $D(v)$ is hydrodynamic damping, $g(\eta)$ collects gravity, buoyancy, and restoring forces, and $w$ denotes environment disturbances such as currents and noise [2605.15528]. The RL policy does not alter these equations; rather, the environment maps learned actions to control inputs.

Sensing and communication are encoded as first-class components of the environment. Each AUV is a local sensing node that observes target-relative state within sensing range and neighbor-relative information within communication range. Observation confidence is distance-modulated,
$$
c_i^t = \text{clip}\left(1 - \frac{d_i^t}{R_s}, 0, 1\right),
$$
and the communication graph is written as $G_t = (\mathcal{A}, E_t)$, with a basic distance-based link quality
$$
q_{ij} = \text{clip}\left(1 - \frac{\|p_i - p_j\|_2}{R_c}, 0, 1 \right).
$$
An extended link-quality model additionally incorporates attenuation, packet loss, and age of information [2605.15528]. The benchmark configuration is symmetric across agents, with parameter sharing in STG-MAPPO, and the canonical task instantiates 4 AUVs and 1 moving target [2605.15528].

## 3. Task formulation and semantic representation

Swarm target tracking is formulated as a Decentralized Partially Observable Markov Decision Process,
$$
\mathcal{M} = \{ S, \{O_i\}_{i=1}^N, \{A_i\}_{i=1}^N, P, R, \gamma, N\},
$$
with centralized training and decentralized execution. During training, the critic can access global or concatenated states; during execution, each actor uses only local observation, including a semantic summary of neighborhood conditions [2605.15528]. The joint policy is factorized as
$$
\pi_\theta(a_t \mid o_t) = \prod_{i=1}^N \pi_\theta(a_i^t \mid o_i^t),
$$
and the return objective is the usual discounted cumulative reward [2605.15528].

A central feature of the platform is the replacement of raw geometric state with semantic policy input. For each agent, the observation includes raw physical state, tracking diagnostics, task phase and observation-quality semantics, and a Semantic Task Graph summary. Tracking diagnostics include distance to target, tracking error, improvement trend, closing tendency, moving-average error, observation confidence, loss indicator, and normalized consecutive loss duration. Task phases are discretized into $\{\text{search}, \text{approach}, \text{stable}, \text{lost}\}$, and observation quality into $\{\text{high}, \text{medium}, \text{low}, \text{unavailable}\}$, both one-hot encoded [2605.15528].

The final semantic observation is
$$
\text{STG}_i^t = [\, o_{i,\text{raw}}^t,\; z_{\text{track},i}^t,\; z_{\text{sem},i}^t,\; g_i^t \,].
$$
Here $g_i^t$ is a compact local graph summary over current neighbors. The platform therefore does not treat graph structure as merely a communication artifact; it converts task phase, observation reliability, link quality, neighbor tracking quality, and local role advantage into explicit policy input [2605.15528]. This design choice is important because the platform’s motivating claim is that raw geometric states and low-level force actions often struggle to represent task phases, observation reliability, link quality, and local cooperation roles [2605.15528].

## 4. STG-MAPPO and the learning interface

STG-MAPPO is the platform’s reference algorithm. It is described as MAPPO augmented with a Semantic Task Graph, a velocity-level action abstraction, and a semantic, task-oriented reward [2605.15528]. Rather than using a full graph neural network, the method compresses network state into a local semantic vector,
$$
g_i^t = \Big[
\omega_i^t,\;
\text{mean}_{j \in \mathcal{N}_i^t} q_{ij}^t,\;
\max_{j \in \mathcal{N}_i^t} q_{ij}^t,\;
\text{mean}_{j \in \mathcal{N}_i^t} e_j^t,\;
\text{adv}_i^t,\;
\text{role}_i^t
\Big],
$$
where $\omega_i^t$ is an effective neighbor ratio, link qualities are aggregated by mean and max, $\text{mean}(e_j^t)$ measures neighbor tracking error, $\text{adv}_i^t$ is local tracking advantage relative to neighbors, and $\text{role}_i^t$ indicates whether the agent acts as a primary tracker or helper [2605.15528]. The local advantage term is
$$
\text{adv}_i^t = \text{clip}\left(\text{mean}_{j \in \mathcal{N}_i^t} e_j^t - e_i^t,\,-1,1\right).
$$

The action interface is equally significant. Baselines often act directly in 6-D force/torque space, but STG-MAPPO instead adopts a 3D desired-velocity action,
$$
a_i^t = [a_{x,i}^t, a_{y,i}^t, a_{z,i}^t] \in [-1,1]^3,\qquad
v_{\text{des},i}^t = S_v a_i^t,
$$
with the environment mapping this high-level velocity command to executable 6-DOF control inputs through a velocity-tracking controller and an attitude-stabilization moment controller [2605.15528]. Actions are also smoothed by rate limiting. The stated rationale is that learning directly in raw force/torque space is difficult because of strong coupling and saturation, whereas the velocity abstraction preserves full 6-DOF dynamics while simplifying the decision space [2605.15528].

The learning core retains standard PPO/MAPPO machinery. The actor is an MLP $\pi_\theta(a_i^t \mid \text{STG}_i^t)$ over semantic input, and the critic is a centralized value network over concatenated semantic states. The algorithm suite exposed through DI-engine includes cooperative on-policy methods such as MAPPO and HAPPO, off-policy actor-critic methods such as MASAC, MADDPG, and MATD3, and a value-based MADQN baseline [2605.15528]. Because all methods share the same environment, training budget, and metric suite, the platform is intended as a controlled benchmark rather than an algorithm-specific simulator [2605.15528].

## 5. Experimental protocol and reported behavior

The unified benchmark uses a standard scenario of 4 AUVs and 1 target with 500-step episodes, typically trained for $2 \times 10^6$ environment steps and evaluated every 5000 steps [2605.15528]. Difficulty is partitioned into medium and hard settings, and the platform defines additional stress tests involving faster targets, farther initial AUV-target separations, limited sensing range, degraded communication, and a combined-stress condition [2605.15528]. Standard metrics are evaluation return, mean tracking error in kilometers, tail mean target distance in kilometers in the final portion of the episode, target lost rate, action saturation rate, control cost, and communication-quality diagnostics [2605.15528].

Within that protocol, STG-MAPPO is reported to achieve very low tail mean target distance, approximately $0.012$–$0.013$ km, and near-zero target lost rate, often exactly $0$, in both medium and hard scenarios and under combined stress [2605.15528]. The same section reports higher evaluation returns, lower variance, and smoother convergence than representative baselines. MAPPO and MADDPG can perform reasonably in nominal conditions, but degrade under stress, retain non-zero lost rates, and exhibit larger variance across seeds; off-policy baselines such as MASAC, MATD3, and MADQN struggle more severely in tracking accuracy and target-loss metrics [2605.15528].

The ablation results attribute the gain to the combination of semantic state, STG neighbor summaries, semantic reward, and velocity abstraction. A velocity-only variant stabilizes low-level control and virtually eliminates target loss, but still leaves tail distance significantly higher, around $0.2$ km. A semantic-state variant improves over velocity-only input, yet remains clearly below the full model. The full configuration, denoted STG-MAPPO-full, yields the highest return and near-zero tail distance and lost rate [2605.15528]. The reported interpretation is that velocity-level abstraction alone helps stabilize control but does not yield precise tracking, while semantic state alone helps but is insufficient without explicit network-aware summarization and reward design [2605.15528].

## 6. Practical scope, codebase, and stated limitations

The released codebase includes environment definitions for a REMUS-like 6-DOF dynamics model, target motion models and scenario definitions, acoustic communication and sensing models, DI-engine wrappers, STG-MAPPO implementation, baseline scripts and configurations, metric computation, convergence plotting, stress-test experiments, and GEBCO-based visualization of swarm trajectories over realistic bathymetry [2605.15528]. The repository is public at `https://github.com/dasjsaj/MARL-AUV` [2605.15528]. The platform is also described as one-command and reproducible, which places it closer to a benchmark infrastructure than to a single simulation script [2605.15528].

The platform’s stated limitations are narrow but explicit. Team size is fixed in the paper at 4 AUVs; dynamic population adaptation is not fully addressed in the released code. The acoustic channel models range, attenuation, loss, and information aging, but full ocean acoustics, including multipath and Doppler, are abstracted. The evaluation is simulation only, with no direct hardware integration, and the primary task is target tracking, so other underwater missions may require custom reward and semantic-state design [2605.15528]. This suggests that MARL-AUV is best understood as a controlled experimental substrate for underwater swarm learning rather than as a complete field-deployment stack.

## 7. Relation to adjacent underwater MARL platforms and stacks

The broader literature shows several neighboring architectures that illuminate what a MARL-AUV platform can mean in practice.

| Work | Core substrate | Relation to MARL-AUV |
|---|---|---|
| UW-MARL [1911.04096] | Shared-Q adaptive sampling on a grid | Early underwater MARL framework with central server and communication protocol |
| FATHOM-Net / DSBM [2404.13654] | SDN-style HSARL with USV-GC, LC-AUVs, ET-AUVs | Hierarchical control-plane/data-plane decomposition for multi-target tracking |
| Scene-adaptive EI [2603.27194] | Three-layer embodied-intelligence architecture | Beacon-based communication-control model and dual-path critic SA-MARL |
| SDA-MARL [2603.29426] | Four-layer hierarchy with diffusion and behavioral cloning | Dual-decision architecture and segregated experience pools for robust tracking |
| SafeDiver [2509.11508] | AUV-USV-diver relay system | Application-driven CTDE stack combining MAPPO movement and MATD3 relay selection |
| CoUGARs / MeCO [2511.08822], [2503.10928] | Open hardware with ROS 2, Docker, simulator integration | Fieldable low-cost or medium-cost AUV substrates for future MARL deployment |

Within this surrounding ecosystem, MARL-AUV occupies the role of a public, physically modeled benchmark, whereas several related systems emphasize hierarchical networking, hardware modularity, or mission-specific control. FATHOM-Net frames the swarm as an underwater ad-hoc network under a hierarchical software-defined reinforcement-learning architecture and adds Dynamic-Switching Attention and Dynamic-Switching Resampling to accelerate convergence and avoid local optima [2404.13654]. Scene-adaptive embodied intelligence further formalizes a three-layer perception-decision-execution framework with beacon-based communication and a dual-path critic that fuses scene-specific and general safety objectives [2603.27194]. SDA-MARL extends the hierarchy to four layers and introduces supervised diffusion-aided sample generation and behavioral cloning to mitigate non-stationarity, sparse rewards, and disturbance fragility [2603.29426].

Other works emphasize application specialization or hardware grounding rather than an open benchmark interface. SafeDiver defines a concrete AUV-USV-diver system with MAPPO for cooperative AUV movement and MATD3 for dynamic relay selection, but is described as application-driven rather than a general platform API [2509.11508]. CoUGARs introduces a low-cost, configurable AUV fleet with ROS 2, Docker, acoustic cooperative localization software, and tight HoloOcean integration, while MeCO provides an open-source AUV with ROS 2, micro-ROS, Docker, and a Unity-based simulator; both are strong physical substrates for MARL yet are not themselves presented as MARL benchmarks [2511.08822][2503.10928]. Non-MARL but architecturally relevant counterparts include a five-tiered multi-AUV route planner for stochastic ocean environments [2311.06579] and autonomous marine-debris mapping with a modified BlueROV2 Heavy using stereo imaging and underwater hyperspectral sensing [2208.00802]. Taken together, these systems indicate that the contemporary MARL-AUV landscape spans benchmark platforms, hierarchical coordination stacks, imitation-learning environments, and open hardware, with MARL-AUV [2605.15528] serving as the most explicit public benchmark centered on 6-DOF AUV swarm target tracking.

Source: https://www.emergentmind.com/topics/marl-auv-platform