Decentralized Multiagent Reinforcement Learning

• Decentralized MARL is a framework where agents learn policies based on local observations to make sequential decisions without centralized communication.
• It employs methods like independent learners, CTDE, and decentralized credit assignment to manage large-scale and noisy environments.
• The approach is validated through theoretical guarantees and empirical results in domains such as robotics, wireless networks, and smart grids.

Decentralized multiagent reinforcement learning (MARL) addresses the challenge of sequential decision making by multiple agents acting based solely on their local information, in environments with possibly partial and noisy observations, nonstationarity induced by concurrent learning, and large-scale action spaces. It provides mathematical, algorithmic, and empirical frameworks for the synthesis and analysis of decentralized policies, often under limited or no inter-agent communication, in domains ranging from wireless access, collaborative robotics, pursuit-evasion, networked control, and swarm systems. Decentralization presents several distinctive technical problems: scalable value function representation in large agent populations, robustness to environmental and intrinsic noise, efficient credit assignment, and provable convergence under partial observability and network-induced constraints.

1. Problem Formulation and Modeling Paradigms

Decentralized MARL is typically formalized as either a Decentralized Markov Decision Process (Dec-MDP), Decentralized Partially Observable MDP (Dec-POMDP), or as a networked (often graph-structured) MDP wherein agent coupling is explicit in the environment dynamics or reward structure. In these models, global state $s\in \mathcal S$ may not be fully accessible to any single agent; each agent $i\in\{1,\ldots,N\}$ observes a local observation $o^i_t$ and selects an action $a^i_t$, yielding a joint transition and reward signal $r^i_t$ or a shared team reward $r_t$. In the absence of centralization, agents have access only to their own observations and possibly local histories, and the joint policy factorizes, e.g., $\pi(a_1,\ldots,a_N|s) = \prod_i \pi^i(a^i | o^i)$ (Xu et al., 2021, Wang et al., 2020). Special attention is given to networked agents with restricted communication, with agent communication modeled as a graph $G=(V,E)$ and local neighborhoods $N(i)$. Modeling frameworks accommodate stochastic games, continuous state/action spaces, and graph-induced coupling with reward or transition locality (Bolliger et al., 9 Mar 2025, Hu et al., 2021, Gu et al., 2021).

2. Algorithmic Techniques in Decentralized MARL

A broad array of algorithms have been developed for decentralized multiagent RL, covering both model-free and model-based, value-based and policy gradient approaches:

• Independent learners treat the environment as stationary from each agent's perspective, running single-agent RL such as DDPG, TD3, or Q-learning on local observations (Jr et al., 2020, Xu et al., 2021).
• Centralized Training with Decentralized Execution (CTDE) frameworks maintain a centralized critic (or centralized policy/distilled via imitation) during training, but enforce decentralized, purely local policies at deployment (Li et al., 2021, Chen, 2019).
• Decentralized policy gradient and actor-critic schemes update local policies and value functions, with agent-specific critics operating with local (or neighborhood) information and sometimes leveraging consensus averaging over networked communication (Bolliger et al., 9 Mar 2025, Li et al., 2021, Zhang et al., 2019).
• Model-based hierarchical strategies integrate learned local dynamic models or self-supervised predictors for rollouts and decentralized planning (e.g., HPP for rendezvous with noisy LIDAR) (Wang et al., 2020).
• Distribution matching and imitation-based methods align local policy visitation with expert marginals from joint demonstrations, with theoretical convergence to the original joint policy under ergodicity and compatibility conditions (e.g., DM$^2$) (Wang et al., 2022, Lin et al., 2019).
• Decentralized credit assignment exploits game-theoretic constructions (e.g., Shapley or Banzhaf values) to disambiguate agent-specific contributions under global reward, with both model-free and model-based estimations complementing policy-gradient updates (Han et al., 2021).
• Scalable value function schemes (e.g., monotonic value function factorization as in ReMIX), exploit architectural properties to reduce representational complexity and enable decentralized greedy action selection (Mei et al., 2023).

3. Communication and Information Constraints

Decentralization inherently restricts information flow: agents may act with full, partial, or strictly local observability, and may be permitted limited communication over fixed or time-varying graphs. Various works address this through explicit communication protocols or by enforcing privacy and non-disclosure of policy/critic parameters (Bolliger et al., 9 Mar 2025, Li et al., 2021, Zhang et al., 2019). Analyses commonly rely on the structure of the underlying communication graph—diameter, clique cover, mixing time—to derive regret or sample complexity bounds as a function of the degree of information sharing, showing that increased communication across larger neighborhoods yields improved convergence rates, scaling from $i\in\{1,\ldots,N\}$0 to $i\in\{1,\ldots,N\}$1 in multiagent regret (Lidard et al., 2021).

Locality in value-function and policy updates further enables scalable deployments: exponential decay of the Q-function's dependence on distant agents' states/actions justifies $i\in\{1,\ldots,N\}$2-hop truncation and localized actor-critic updates. The complexity and sample efficiency depend crucially on the neighborhood size, with theoretical accuracy $i\in\{1,\ldots,N\}$3 under local coupling (Hu et al., 2021, Gu et al., 2021).

4. Robustness, Stability, and Uncertainty

Decentralized MARL in real systems faces noise in reward, state, and transition models. Empirical evidence shows naive independent learners exhibit high variance and instability under reward perturbations, while robust extensions with adversarial “nature” (e.g., robust MADDPG) maintain stable learning and low variance (Xu et al., 2021). Explicit constraints from control theory, such as Lyapunov-based stability, can be incorporated into policy optimization via Lagrangian penalties or hard constraints, ensuring the learned decentralized controllers are not only reward-optimal but also provably closed-loop stable (Zhang et al., 2020). The robustness of decentralized MARL to partial observability, sensing noise, and model mismatch is further improved by self-supervised model learning and flexible planning (e.g., HPP in model-based rendezvous) (Wang et al., 2020).

5. Theoretical Guarantees and Sample Complexity

Recent theoretical advances have improved the understanding of convergence and complexity in decentralized MARL:

• Convergence rates and regret bounds: Decentralized Q-learning with asynchronous message passing and local consensus achieves regret bound $i\in\{1,\ldots,N\}$4 in the tabular case, with communication radius and clique cover critically affecting scaling (Lidard et al., 2021).
• Equilibrium learning: Stage-based V-learning algorithms attain $i\in\{1,\ldots,N\}$5-approximate Nash, coarse correlated, and swap correlated equilibria, with sample complexity scaling only with the maximal local action cardinality $i\in\{1,\ldots,N\}$6, rather than the joint space size (Jin et al., 2021, Mao et al., 2021).
• Locality and decay: Theoretical results establish that Q- and value-functions in networked MARL decay exponentially with distance, thereby justifying scalable, local policy updates for large system sizes (Gu et al., 2021, Hu et al., 2021).
• Approximate optimality under partial information: Decentralized RL with partial-history sharing (PHS) leverages the common-information approach plus finite-state truncation, guaranteeing exponentially decaying error in team performance with truncation depth (Arabneydi et al., 2020).

6. Empirical Results and Real-World Deployment

Extensive experimental results validate decentralized MARL algorithms across cooperative navigation, pursuit-evasion, task offloading, resource management, and pandemic mitigation:

• Wireless task offloading: Robust MARL methods (RMADDPG) stabilize learning in wireless environments with reward noise, showing lower variance, faster convergence, and greater reliability than independent DDPG (Xu et al., 2021).
• Swarm control and multi-robot systems: Decentralized pursuit with shared-experience policies and curriculum learning outperforms classical omnidirectional control and matched DRL-based methods, with successful transfer to real quadcopters (Jr et al., 2020). Model-based HPP achieves reliable decentralized rendezvous with sim-to-real transfer (Wang et al., 2020).
• Networked pandemic mitigation and smart grids: Distributed actor-critic with reward machines enables local controllers to achieve $i\in\{1,\ldots,N\}$7 reward improvement over static policies in COVID-19 mitigation; contextual multi-agent coordination in power grids converges to within $i\in\{1,\ldots,N\}$8 of known global-optimal performance (Hu et al., 2021, Mishra et al., 2020).
• Adversarial, competitive, heterogeneous, and large-$i\in\{1,\ldots,N\}$9 setups: Consensus-based MADDPG, distribution-matching (DM$o^i_t$0), and imitation-based decentralization recover coordination at scale, with performance approaching centralized or expert baselines in complex StarCraft and MuJoCo locomotion tasks (Bolliger et al., 9 Mar 2025, Wang et al., 2022, Han et al., 2021, Chen, 2019).

7. Open Challenges and Future Directions

Despite advances, major challenges remain:

• Partial observability and decentralization: Achieving convergence guarantees and scalable sample-complexity in general Dec-POMDPs, especially with function approximation or deep RL, is still unresolved (Zhang et al., 2019).
• Robustness to model misspecification and adversarial attack: Theoretical understanding of decentralized learning under adversarial noise, heterogeneity, and Byzantine faults is limited (Xu et al., 2021, Zhang et al., 2019).
• Communication efficiency and network dynamics: Designing protocols that adaptively balance communication overhead and information value, accommodating time-varying or unreliable networks, is a key frontier (Li et al., 2021, Zhang et al., 2019).
• Multi-scale and hierarchical architectures: Further exploration is needed on layered control, where high-level decentralized agents coordinate teams of low-level learners (Zhang et al., 2019).
• Integration with formal methods and safety: Combining decentralized MARL with formal logic constraints (e.g., LTL, STL) to ensure required safety and performance in critical applications is a nascent but increasingly critical area (Zhang et al., 2019).

The field continues to advance towards provably efficient, robust, and scalable decentralized MARL algorithms capable of addressing the complexity of real-world multiagent systems under realistic information, communication, and dynamical constraints.