Disaggregated MARL: Decentralized & Specialized RL

• Disaggregated MARL is a design paradigm that decomposes centralized control into decentralized agents using egocentric observations and local rewards.
• It leverages architectural decomposition and functional specialization to enhance scalability, coordination, and computational efficiency.
• The approach integrates internalized safety and distributional reward decomposition to improve robustness and reduce resource demands.

Disaggregated multi-agent reinforcement learning refers to MARL designs that replace monolithic control with decomposed learning, acting, safety, or deployment responsibilities across multiple agents or subsystems. In the cited literature, disaggregation appears as decentralized and self-supervised agents that act from egocentric observations without centralized training, global communication, or direct access to peers’ rewards (Xiang et al., 2023), local value and policy representations that depend only on one-hop neighbors or anonymous action counts (Wang et al., 2022, He et al., 2021), distributed actor-worker-learner infrastructures (Qi et al., 2022, Mehta et al., 2023), role- or task-specialized agents (Goel et al., 30 May 2025, Panitsas et al., 23 Sep 2025, Zhang et al., 23 Jun 2025), and internalized safety or resource-aware deployment mechanisms that move functionality from centralized modules into the agents themselves (Pan et al., 2 Oct 2025, Pavel et al., 8 Apr 2026). This suggests that disaggregated MARL is best understood as a design family rather than a single algorithm.

1. Conceptual scope and design space

In the cited literature, the term disaggregated is applied to several related decompositions. Some works emphasize decentralized learning under realistic information constraints, where each agent learns from its own local observations and rewards rather than from globally shared signals (Xiang et al., 2023). Other works use disaggregation to denote architectural decoupling, as in actor-worker-learner systems that separate environment interaction from model iteration (Qi et al., 2022). A third usage centers on functional specialization: agents can be assigned to slice types, contour points, attackers, defenders, or learned roles (Panitsas et al., 23 Sep 2025, Zhang et al., 23 Jun 2025, Pan et al., 2 Oct 2025). A fourth usage is deployment-oriented, in which a centralized expert is distilled into lightweight decentralized students with heterogeneous model capacities (Pavel et al., 8 Apr 2026).

Axis of disaggregation	Mechanism	Representative papers
Information structure	Egocentric observations, local rewards, no global communication	(Xiang et al., 2023)
Representation	One-hop neighborhoods, action configurations, permutation invariance	(Wang et al., 2022, He et al., 2021)
Systems architecture	Actor-worker, actor-worker-learner, asynchronous distributed training	(Qi et al., 2022, Mehta et al., 2023)
Functional specialization	Roles, slice-type agents, contour-point agents, attackers/defenders	(Goel et al., 30 May 2025, Panitsas et al., 23 Sep 2025, Zhang et al., 23 Jun 2025, Pan et al., 2 Oct 2025)
Deployment	Centralized expert distilled into lightweight decentralized students	(Pavel et al., 8 Apr 2026)

This plurality matters because different disaggregation choices address different bottlenecks. Local-information methods target realism and scalability under partial observability. Architectural decomposition targets sample collection speed, replay efficiency, and GPU utilization. Functional specialization targets coordination and division of labor. Deployment-oriented disaggregation targets FLOPs, memory, and inference time.

2. Local information, partial observability, and scalable representations

A central theme in disaggregated MARL is the rejection of unrealistic centralized assumptions. In realistic multi-agent settings, autonomous agents rely on egocentric observations; they do not receive global state or access to non-visible agent data, and several decentralized methods operate strictly locally, with rewards and predictions based only on what agents can observe, typically through a $15\times 15$ egocentric window (Xiang et al., 2023). The same realism motivation appears in one-hop-neighbor methods, where each agent’s policy and value functions depend only on the agent’s own state and its one-hop neighbors, reducing complexity while preserving local coordination structure (Wang et al., 2022).

Scalability under many-agent partial observability has also been approached through action anonymity and permutation invariance. When transition and reward functions depend only on the counts of actions taken rather than on agent identities, the joint action can be summarized by an action configuration,

$$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$

which is polynomial rather than exponential in the number of agents (He et al., 2021). Configuration-based critics for MADDPG and IA2C are lossless under anonymity and learn the true optimal policy regardless of interaction topology, whereas mean-field methods are exact only in fully connected settings or purely local pairwise interactions and otherwise can converge to suboptimal solutions (He et al., 2021). Mean-field methods nonetheless typically converge faster, by about $43\%$ fewer episodes in the reported experiments, which makes the trade-off between fidelity and efficiency explicit (He et al., 2021).

Partial observability can also be addressed through decentralized state estimation rather than through direct access to global state. A recent MARL+SL framework alternates one step of adaptive social learning with one step of MARL, enabling concurrent state estimation and policy optimization without a two-timescale inner loop (Zhaikhan et al., 8 Aug 2025). Simulation results verify that the performance of this methodology can approach that of reinforcement learning when the true state is known (Zhaikhan et al., 8 Aug 2025). A plausible implication is that disaggregation in MARL increasingly includes not only decentralized acting, but also decentralized inference over latent global state.

3. Disaggregated optimization and training architecture

At the algorithmic level, disaggregation often means that policy and value updates are computed locally, sometimes with only limited consensus. In decentralized deterministic actor-critic for continuous action spaces, each agent learns a local deterministic policy while exchanging only value parameters with neighbors during the consensus step. The local deterministic policy gradient is

$$\nabla_{\theta^i} J(\theta) = \mathbb{E}_{s \sim d^\theta} \left[ \nabla_{\theta^i}\mu^i_{\theta^i}(s)\, \nabla_{a^i}Q_\theta(s,a^{-i},a^i)\big|_{a^i=\mu^i_{\theta^i}(s)} \right],$$

and the method provides almost sure convergence for linearly approximated value functions under the stated assumptions (Grosnit et al., 2021). This line of work makes explicit that disaggregation can coexist with formal convergence analysis rather than being only an engineering heuristic.

A distinct strand focuses on systems-level decomposition. The Actor-Worker framework and the extended Actor-Worker-Learner framework decouple environment interaction from model iteration, deploy multiple asynchronous environment interaction modules, use a centralized experience replay pool, and communicate parameters through shared memory (Qi et al., 2022). In SMAC scenarios, AW-Qmix is about $1.6$–$2.1\times$ faster than Qmix in sample collection speed, while AWL-Qmix is about $6.6$–$7.6\times$ faster than Qmix (Qi et al., 2022). The significance of these results is architectural rather than algorithmic: disaggregation here reduces waiting time, increases sample diversity, and improves GPU utilization.

The same architectural perspective is visible in software frameworks. marl-jax supports single-threaded, synchronous distributed, IMPALA-style asynchronous distributed, and Sebulba-style inference-server training architectures, and is designed for training populations of agents and evaluating social generalization in cooperative and competitive environments (Mehta et al., 2023). It therefore treats disaggregation as a combination of population-based training, decentralized policy updates, and distributed systems support.

Deployment-oriented disaggregation is represented by KD-MARL. A centralized expert with a centralized critic is first trained and frozen; lightweight decentralized students are then trained without a critic, using distilled advantage signals computed from the frozen teacher critic and structured policy supervision that preserves coordination, structure, and roles (Pavel et al., 8 Apr 2026). Across SMAC and MPE benchmarks, KD-MARL retains over $90\%$ of expert performance while reducing computational cost by up to $28.6\times$ FLOPs (Pavel et al., 8 Apr 2026). Here, disaggregation is not primarily about training realism; it is about making coordinated MARL executable on edge devices or embedded platforms.

4. Specialization, role discovery, and internalized functionality

Disaggregation often manifests as specialization. In mixed-motive environments, centralized reward-sharing methods surpass self-supervised methods because they produce sharper, more effective role specialization and greater competence in those roles (Xiang et al., 2023). The same study reports that this gap shrinks in pure-motive settings, which emphasizes mixed-motive environments as the critical testbed for realistic MARL (Xiang et al., 2023). The underlying point is not that decentralization is ineffective, but that decentralized populations currently underperform when coordination requires difficult division-of-labor solutions.

The emergence of specialization has been studied directly in multi-task environments. In a grid-world ant environment with two sequential tasks, increasing the number of agents makes specialization more probable, and entropy-regularized A2C yields smoother convergence than independent DDDQN with $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$0-greedy exploration (Gasparrini et al., 2019). The reported interpretation is that synchronized exploration schedules can destabilize independent learners by forcing them to become deterministic at the same time, whereas decentralized entropy-based exploration lets agents adapt at different rates (Gasparrini et al., 2019). This suggests that disaggregated learning benefits from exploration mechanisms that are themselves disaggregated.

Role-based MARL makes specialization explicit. R3DM maximizes the mutual information between agents’ roles, observed trajectories, and expected future behaviors, combining contrastive learning on past trajectories with a learned dynamics model to shape intrinsic rewards for future diversity (Goel et al., 30 May 2025). On SMAC and SMACv2, R3DM improves win rates by up to $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$1, and the paper’s framing is specifically future-oriented: a role should shape future behavior, not merely summarize past behavior (Goel et al., 30 May 2025). The technical significance is that role discovery becomes a predictive, not just descriptive, latent-variable problem.

Internalized safety extends the same logic to adversarial settings. AdvEvo-MARL moves away from centralized external guard mechanisms and instead embeds safety within each defender agent (Pan et al., 2 Oct 2025). The framework is formulated as a partially observable Markov game with attackers and defenders; attackers first undergo supervised fine-tuning on jailbreak strategies, after which attackers and defenders co-evolve through reinforcement learning (Pan et al., 2 Oct 2025). To stabilize learning within each role group, it introduces a public baseline for advantage estimation:

$$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$2

with advantages defined relative to the appropriate group baseline (Pan et al., 2 Oct 2025). Across representative attack scenarios, AdvEvo-MARL consistently keeps attack-success rate below $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$3, whereas baselines reach up to $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$4, while preserving and sometimes improving task accuracy, with up to $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$5 on reasoning tasks (Pan et al., 2 Oct 2025). The paper explicitly argues that external guards create a single point of failure and extra system overhead, while disaggregated internalized safety avoids that architecture (Pan et al., 2 Oct 2025).

5. Uncertainty, noise, and distributional decomposition

Disaggregation is also applied to uncertainty. In task offloading for mobile edge computing, perturbations in the reward signal induce a decrease in performance relative to learning with perfect rewards, and robust MADDPG remains more stable than standard DDPG under noisy rewards (Xu et al., 2021). The reported finding is not that decentralized MARL is inherently robust, but rather that practical deployment requires explicit robustness mechanisms when rewards and states are subject to noise and alteration (Xu et al., 2021).

A more structural response is reward-distribution decomposition. Noise Distribution Decomposition models the globally shared noisy reward with a Gaussian mixture model and decomposes it into individual distributional local rewards, after which each agent updates locally through distributional reinforcement learning (Geng et al., 2023). A diffusion model is used to generate additional reward samples for distribution learning, and distortion risk functions enable risk-sensitive policies (Geng et al., 2023). The paper further provides monotonicity results connecting local and global distorted return expectations, making the method closely related to value-decomposition ideas under noisy feedback (Geng et al., 2023).

Distributional factorization in cooperative MARL has also been generalized directly. DFAC extends VDN, QMIX, and QPLEX from expected value factorization to return-distribution factorization through a mean-shape decomposition and quantile mixture construction (Sun et al., 2023). In the framework,

$$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$6

so that the mean preserves the original factorization while the shape captures higher-order distributional structure (Sun et al., 2023). Experiments on Super Hard SMAC maps and custom Ultra Hard maps show that DFAC variants outperform non-distributional baselines (Sun et al., 2023). The broader significance is that disaggregation can occur not only over control and computation, but over uncertainty representations themselves.

6. Applications, empirical trade-offs, and unresolved tensions

Application-specific systems often make disaggregation concrete. MARL-MambaContour models each contour point as an autonomous agent in contour-based medical image segmentation, uses a contour-specific Soft Actor-Critic algorithm enhanced with an Entropy Regularization Adjustment Mechanism, and incorporates a Mamba-based policy network with a Bidirectional Cross-attention Hidden-state Fusion Mechanism (Zhang et al., 23 Jun 2025). The paper describes this as the first contour-based medical image segmentation framework based on MARL and reports state-of-the-art performance on five diverse medical imaging datasets (Zhang et al., 23 Jun 2025). Here, disaggregation is literal: a global contour is optimized through a cooperative system of pointwise agents.

In network orchestration, SlicePilot assigns each agent to manage slice requests of the same type, thereby reducing the state-action space each agent must explore (Panitsas et al., 23 Sep 2025). Experimental evaluation on a multi-cloud testbed shows a $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$7 speed-up compared to combinatorial optimization, with deployment costs within $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$8 of the optimal, but with up to $$C^\mathbf{a} = \langle \#a^1, \#a^2, \ldots, \#a^{|A|} \rangle,$$9 more QoS violations under high load (Panitsas et al., 23 Sep 2025). This is a clear example of the central trade-off in disaggregated MARL: faster decision-making and reduced computational complexity can come with some loss in global optimality or constraint satisfaction.

The literature repeatedly emphasizes that disaggregation is not a universal substitute for centralized structure. In mixed-motive environments, centralized reward-sharing populations still outperform self-supervised populations, and closing this gap is identified as essential for real-world readiness (Xiang et al., 2023). Under action anonymity, configuration-based methods remain optimal across graph structures, whereas mean-field methods are faster but can be suboptimal outside fully connected settings (He et al., 2021). In deterministic decentralized MARL, exploration remains a bottleneck because deterministic policies inherently lack exploration, even when convergence guarantees are available (Grosnit et al., 2021). In safety-critical LLM-based multi-agent systems, disaggregated internalized safety removes extra guard agents and their overhead, but it also requires adversarial co-evolutionary training and careful credit assignment (Pan et al., 2 Oct 2025).

Taken together, these results define the current research frontier. Improving self-supervised intrinsic social motivations, strengthening decentralized emergent specialization, preserving coordination under heterogeneous observations and resource constraints, and maintaining robustness to noisy rewards and adaptive adversaries are recurrent open problems across the cited work (Xiang et al., 2023, Pavel et al., 8 Apr 2026, Geng et al., 2023, Pan et al., 2 Oct 2025). The common thread is that disaggregation is valuable when decomposition aligns with the problem’s information structure, coordination pattern, and deployment constraints; when that alignment fails, centralized or more structured alternatives can still dominate.