MARL-MASS: Multi-Agent RL Framework

• The paper introduces MARL-MASS as a framework for structured multi-agent reinforcement learning that integrates explicit coordination primitives like consensus updates and dynamic communication.
• It categorizes various formal models—from networked decentralized MDPs to Dec-POMDPs and major-minor mean field control—to address task-specific challenges in safety, robustness, and coordination.
• Key scaling mechanisms include advanced simulator acceleration, infrastructure optimization, and large-scale curriculum design to enhance learning stability and performance in complex settings.

MARL-MASS can be understood as an umbrella label for work on multi-agent reinforcement learning in multi-agent systems where communication, consensus, organizational structure, safety, and scaling constraints are explicit parts of the problem formulation. Recent work under this broad reading includes decentralized actor-critic learning over connected time-varying networks with adversarial agents (Sarkar, 2023), dynamic communication topology under time-varying delay and packet loss (Xu et al., 18 Aug 2025), major-minor mean field control for many homogeneous agents and a few complex agents (Cui et al., 2023), distributed epigraph-form safe control (Zhang et al., 21 Apr 2025), large-scale LLM-based MARL infrastructure (Jiang et al., 10 Feb 2026), and application-driven studies in warehouse robotics, underwater acoustic tracking, multi-party ad hoc teamwork, and MAS organization engineering (Allman et al., 4 Dec 2025, Gallici et al., 13 May 2025, Zhang et al., 29 Oct 2025, Soulé et al., 5 Jun 2025).

1. Conceptual scope and formal models

Across this literature, MARL-MASS spans several formal models rather than a single canonical formulation. One line studies a networked multi-agent MDP with shared state space $\mathcal{S}$, agent-specific action spaces $\mathcal{A}^i$, joint action space $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$, heterogeneous local rewards $$\mathcal{R}^i:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$$, and a collaborative objective that maximizes the long-run average of the global average reward $$\overline r_{t+1}=\frac{1}{N}\sum_{i\in\mathcal N} r^i_{t+1}$$ (Sarkar, 2023). Another line uses the Dec-POMDP template $\langle N,S,U,P,R,O,\gamma\rangle$ for connected vehicle platoons, or $$\mathcal{G}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{O},T,R,\Omega,\gamma\rangle$$ for multi-party ad hoc teamwork, with local observation histories and decentralized policies under partial observability (Xu et al., 18 Aug 2025, Zhang et al., 29 Oct 2025). A third line replaces explicit finite-agent state tuples by a major-minor mean field state $(x^0,\mu)$, where $x^0$ is the major agent state and $\mu$ is the empirical or limiting distribution of minor-agent states (Cui et al., 2023).

This diversity is visible in the application layer as well. Warehouse robotics is modeled as a Dec-POMDP with local observations, discrete actions, and sparse shared rewards; organizational engineering work treats the environment as a Dec-POMDP but maps joint histories into structural, functional, and deontic specifications of $\mathcal{A}^i$0 (Allman et al., 4 Dec 2025, Soulé et al., 5 Jun 2025). A plausible implication is that MARL-MASS is less a single benchmark family than a family of structurally enriched MARL problems.

Stratum	Formal object	Representative work
Networked decentralized MARL	Shared state, heterogeneous local rewards, consensus critic	(Sarkar, 2023)
Communication-aware Dec-POMDP	Dynamic graph, delayed and lossy observations	(Xu et al., 18 Aug 2025)
Major-minor mean field MARL	State $\mathcal{A}^i$1 and mean-field action $\mathcal{A}^i$2	(Cui et al., 2023)
Multi-party ad hoc teamwork	Controlled agents plus multiple unfamiliar uncontrolled groups	(Zhang et al., 29 Oct 2025)

2. Coordination primitives: consensus, communication, relations, and organization

A central theme is that coordination is implemented through explicit structural primitives rather than implicit joint-policy coupling. In decentralized consensus-based actor-critic MARL, each agent maintains its own critic parameter vector $\mathcal{A}^i$3 and actor parameter $\mathcal{A}^i$4, exchanges only critic parameters, and applies a consensus update

$\mathcal{A}^i$5

with Metropolis weights as one concrete design. The vulnerability identified in this setting is that these consensus updates are linear and trust all neighbors equally up to fixed weights, so an adversarial agent can bias the critic estimate propagated through the network (Sarkar, 2023).

Communication-aware platoon control pushes this structural layer into the policy itself. DCT-MARL augments each agent state as

$\mathcal{A}^i$6

and learns a dynamic communication topology by a multi-key gated communication mechanism. The topology is encoded by an adjacency trajectory matrix, while edge activation is produced by Gumbel-Softmax gates and refined by distance-aware weighting (Xu et al., 18 Aug 2025). In multi-party ad hoc teamwork, MARS replaces a fully connected interaction graph by a sparse skeleton graph: intra-group edges are fully connected, whereas inter-group edges are randomly sampled and processed by a Relational Forward Model, yielding relational cooperation embeddings for controlled agents (Zhang et al., 29 Oct 2025).

A different structural mechanism appears in organization-oriented MARL. AOMEA and PRAHOM couple a MARL process with $\mathcal{A}^i$7, using action and observation masking to constrain policy space during training and using joint histories to infer roles, links, missions, permissions, and obligations after training (Soulé et al., 5 Jun 2025). This suggests a recurring pattern: coordination is being moved into explicit layers of consensus, topology selection, relation sampling, or organizational specification.

3. Safety, robustness, and hard constraints

Safety and robustness in MARL-MASS appear in at least three distinct forms: adversarial communication robustness, hard state constraints, and robustness to non-ideal channels. In adversary-aware decentralized MARL, the threat model assumes that rewards and environment data are not corrupted, but adversarial agents do not follow the consensus updates. The adversarial set is assumed to be $\mathcal{A}^i$8-local, and the robust consensus rule removes the highest and lowest $\mathcal{A}^i$9 neighbor states before averaging: $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$0 The local RL updates remain unchanged; only the aggregation layer is modified (Sarkar, 2023).

Hard safety is formalized more directly in distributed epigraph-form MARL. The multi-agent safe optimal control problem is posed as

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$1

where $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$2 is cumulative task cost and

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$3

is the worst future constraint violation across agents. The epigraph reformulation introduces a scalar $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$4 and an inner objective

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$5

together with augmented dynamics

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$6

The resulting Def-MARL algorithm uses centralized training and distributed execution, solves the outer problem in distributed fashion, and is reported to achieve the best overall performance, satisfy safety constraints, and maintain stable training across eight tasks and real-world Crazyflie experiments (Zhang et al., 21 Apr 2025).

Robustness to non-ideal communication is developed in platoon control. DCT-MARL models time-varying delays $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$7 and packet loss on a time-varying directed graph $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$8, includes delay explicitly in the state, and uses a reward with spacing, velocity, jerk, and safety terms. Simulation results show that DCT-MARL significantly outperforms state-of-the-art methods in terms of string stability and driving comfort (Xu et al., 18 Aug 2025). A plausible implication is that MARL-MASS safety is not a single notion: it ranges from Byzantine resilience in parameter exchange to zero-violation control and delay-aware robustness in cyber-physical networks.

4. Scaling mechanisms: populations, simulators, and training systems

Scaling is addressed at three different levels: population-level abstraction, simulator-level acceleration, and cluster-level systems design. In major-minor mean field MARL, a large finite system is replaced by a mean-field control problem with state $\mathcal{A}=\prod_{i=1}^N \mathcal{A}^i$9 and action $$\mathcal{R}^i:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$$0, where $$\mathcal{R}^i:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$$1 is a joint state-action distribution for the minor population. The theory provides a dynamic programming principle, existence of optimal stationary deterministic policies, propagation-of-chaos convergence, and an $$\mathcal{R}^i:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$$2 rate for finite $$\mathcal{R}^i:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$$3, while M3FMARL uses the finite system as a Monte Carlo simulator of the M3FC MDP and is shown to approximate the policy gradient of the underlying problem (Cui et al., 2023).

At the simulator level, underwater acoustic tracking uses iterative distillation from Gazebo’s LRAUV simulator into a simplified, GPU-accelerated JAX environment. The distilled environment preserves the coarse action-to-motion mapping at a 30-second control scale, enables fully GPU-resident vectorized simulation, and achieves up to a 30,000x speedup over Gazebo through parallelization. On top of this environment, TransfMAPPO uses a Transformer-based architecture that learns multi-agent policies invariant to the number of agents and targets, and large-scale curriculum learning is conducted entirely on GPU (Gallici et al., 13 May 2025).

At the infrastructure level, FlexMARL studies rollout-training co-design for large-scale LLM-based MARL. It introduces a rollout-training disaggregated architecture, a joint orchestrator, an experience store, a micro-batch driven asynchronous pipeline with strong consistency guarantees, hierarchical load balancing for skewed inter/intra-agent request patterns, and agent-centric resource allocation with unified and location-agnostic state swapping. On a large-scale production cluster, FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks (Jiang et al., 10 Feb 2026).

Sparse-reward warehouse robotics exposes a fourth scaling bottleneck: exploration. In that setting, QMIX’s value decomposition significantly outperforms independent learning approaches, but requires extensive hyperparameter tuning, particularly extended epsilon annealing (5M+ steps) for sparse reward discovery, and significant scaling challenges remain beyond small-scale deployments of 2–4 robots (Allman et al., 4 Dec 2025). This suggests that MARL-MASS scaling is simultaneously an abstraction problem, a simulation problem, a systems problem, and an exploration problem.

5. Evaluation, benchmark design, and methodological caution

A meta-analysis of 75 cooperative MARL papers from 2016 to 2022 identifies large, unexplained performance discrepancies for the same algorithm and environment, sparse and late adoption of ablation studies, inconsistent and often missing statistical methodology, heavy reliance on one or two benchmarks such as SMAC and MPE, and substantial environment overfitting. The proposed standardized protocol recommends 10 independent training runs, 2 million timesteps for off-policy algorithms, 20 million timesteps for on-policy algorithms, evaluation every 10,000 timesteps for 32 episodes, and a final “absolute” evaluation on 320 episodes, together with IQM, stratified bootstrap confidence intervals, optimality gap, probability of improvement, and performance profiles (Gorsane et al., 2022).

A complementary critique concerns what current benchmarks actually test. Cooperative MARL is usually formalized as a Dec-POMDP, but recurrent function approximators often succeed by learning brittle conventions rather than recovering a useful Markov signal. In the Prediction Game case study, blind agents achieve high returns during co-adaptation, mutual information between observations and actions and between hidden states and actions remains low, and performance collapses when trained agents are paired with fixed heuristic partners. The same recurrent models learn grounded policies when the task design necessitates it, leading to two design principles for future environments: behaviors grounded in observations and memory-based reasoning about other agents (Tessera et al., 24 Jul 2025).

Warehouse robotics provides an application-level example of the same methodological issue. Default hyperparameters produce “zero learning” in sparse-reward settings, and successful QMIX training depends on long horizons, large replay buffers, and extended exploration schedules (Allman et al., 4 Dec 2025). This suggests that MARL-MASS evaluation cannot be separated from benchmark construction, observability structure, and the reporting of system-level and optimization-level details.

6. Application domains and open problems

The application range is broad. Communication-aware platoon control studies leader-plus-followers under realistic C-V2X-style delay and packet loss, with evaluation in terms of string stability and driving comfort (Xu et al., 18 Aug 2025). Warehouse robotics studies cooperative package delivery in RWARE and Unity ML-Agents, with value decomposition, sparse rewards, and scaling limits (Allman et al., 4 Dec 2025). Underwater tracking controls fleets of AUVs for acoustic target tracking with realistic communication constraints and reports tracking errors below 5 meters over extended durations in Gazebo (Gallici et al., 13 May 2025). AOMEA targets MAS organization engineering in IoT and cyberdefense settings by translating learned joint histories into curated organizational specifications (Soulé et al., 5 Jun 2025). MAHT addresses collaboration with multiple mutually unfamiliar groups of uncontrolled teammates through sparse relation sampling (Zhang et al., 29 Oct 2025). FlexMARL targets merchant and category assistant workloads with multiple LLM agents on a 48-node cluster (Jiang et al., 10 Feb 2026). Major-minor mean field control covers traffic, logistics, swarms, formation, foraging, and potential-field tasks with many minor agents and a few major ones (Cui et al., 2023).

The open problems are equally diverse. Adversary-aware decentralized MARL explicitly leaves open a formal convergence argument for the robust consensus updates under the adversarial framework (Sarkar, 2023). DCT-MARL leaves analytical string stability under stochastic dynamic topology and broader communication–control co-design as future directions (Xu et al., 18 Aug 2025). Def-MARL points to communication assumptions, approximation error, and centralized-training scalability as remaining constraints (Zhang et al., 21 Apr 2025). Warehouse robotics highlights better exploration strategies, curriculum learning, hierarchical decomposition, and more advanced value factorization for larger deployments (Allman et al., 4 Dec 2025). MAHT identifies adaptive skeleton construction, online partner modeling, and group discovery as next steps (Zhang et al., 29 Oct 2025). FlexMARL notes that off-policy MARL with replay buffers spanning multiple versions and richer centralized critics are not deeply explored at the system level (Jiang et al., 10 Feb 2026). Underwater tracking leaves sim-to-real transfer, richer target behavior, and stronger safety guarantees unresolved (Gallici et al., 13 May 2025). A plausible implication is that MARL-MASS is converging on a common agenda: robust structure-aware coordination, scalable training, and stronger guarantees under partial observability, communication limits, and hard safety requirements.