---
title: CTDE with MAPPO in Multi-Agent RL
url: https://www.emergentmind.com/topics/centralized-training-decentralized-execution-ctde-with-mappo
type: topic
---

# CTDE with MAPPO in Multi-Agent RL

Centralized Training with Decentralized Execution (CTDE) combined with Multi-Agent Proximal Policy Optimization (MAPPO) constitutes a canonical architecture in cooperative multi-agent reinforcement learning (MARL). CTDE enables the use of global or joint information (e.g., joint observations, actions, or true state) during the training process, while enforcing strictly decentralized control at execution: each agent only accesses its own local observations and policy. MAPPO operationalizes CTDE by employing decentralized actors—learned individual policies per agent—and a centralized, typically shared, critic used exclusively during training to estimate value functions and compute advantage signals. This approach stabilizes learning in highly nonstationary MARL environments, supports scalable execution, and achieves strong performance on cooperative tasks ranging from traffic control and server scheduling to public goods games and multi-UAV coordination [2405.06161, 2509.19079, 2512.17187, 2512.03835].

## 1. Fundamental Principles of CTDE and MAPPO

CTDE leverages full joint information in the training phase—such as access to the true global state or the set of all agents’ observations—while ensuring that each agent’s policy conditions only on its own local observation at test time. This paradigm is especially relevant to fully cooperative settings where agents share a joint team reward and face severe nonstationarity arising from updating individual learners in parallel.

MAPPO implements CTDE as follows [2405.06161]:
- **Decentralized Actors**: Each agent $i$ maintains a policy $\pi_\theta(a_i \mid o_i)$, parameterized (often with parameter sharing) to optimize local actions from local observations.
- **Centralized Shared Critic**: A value network $V_\phi(s)$ or $V_\phi(o_1,\ldots,o_n)$, parameterized by $\phi$, is trained to estimate the state-(or joint-observation-)value using complete environment knowledge during centralized training.
- **Training-Only Centralization**: During offline or on-policy training, the critic accesses full joint data; at execution/online deployment, only decentralized actors remain operative.

The CTDE setup using MAPPO provides a direct solution to the nonstationarity problem inherent to concurrent MARL, stabilizes advantage estimation, and avoids the computational intractability of globally coordinated execution.

## 2. MAPPO Optimization Objectives and Learning Pipeline

At the core of MAPPO is the clipped surrogate PPO objective, adapted for the multi-agent case with a centralized critic. A trajectory $\tau$ of length $T$ is generated by assembling local agent actions:
- **Probability Ratio**:
  $$
  r_t(\theta) = \frac{\pi_\theta(a_t \mid o_t)}{\pi_{\theta_{old}}(a_t \mid o_t)},
  $$
  where $a_t$ and $o_t$ are the joint action and joint observation at time $t$, and $\pi_\theta$ factorizes over agents.

- **Clipped PPO Surrogate**:
  $$
  L^{CLIP}(\theta) = \mathbb{E}_t [ \min ( r_t(\theta) \cdot \hat{A}_t, \;
     \mathrm{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \cdot \hat{A}_t ) ].
  $$
  $\epsilon$ is the PPO clipping hyperparameter (typically $0.1$–$0.2$).

- **Centralized Critic Loss**:
  $$
  L^V(\phi) = \mathbb{E}_t [ (V_\phi(s_t) - R_t)^2 ],
  $$
  where $R_t$ is the empirical return (e.g., discounted reward sum or TD target).

- **Advantage Estimation (GAE)**:
  $$
  \delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t), \qquad
  \hat{A}_t = \sum_{l=0}^{T-t-1} (\gamma \lambda)^l \delta_{t+l}.
  $$

Training alternates between collecting data via decentralized rollouts, advantage computation with the centralized critic, and parameter updates on both actor ($\theta$) and critic ($\phi$), as described in the algorithmic blueprint [2405.06161, 2509.19079].

## 3. Architectural Instantiations and Modifications

The baseline MAPPO architecture can be customized to fit various domains—traffic networks, server clusters, large spatial games—while preserving the CTDE principle. Selected instantiations include:

- **Parameter Sharing**: Actors are often parameter-shared, leveraging agent-index or context embeddings to break symmetry [2405.06161].
- **Critic Conditioning**: Critics may take the true state, joint observations, or concatenated local info, depending on environment access [2512.03835].
- **Graph-Based Critics and Personalization**: For tasks with non-IID agent contexts (e.g., intersections with diverse flows), MAPPO can be extended to multi-head critics and "hyper-action" blending over multiple value heads, with the actor producing a distribution over value heads, achieving efficient policy personalization [2503.07678].
- **Communication Mechanisms**: CTDE+MAPPO can be combined with attention-based inter-agent communication modules in the actor, as in MCGOPPO, to further mitigate nonstationarity and accelerate convergence in settings where observation sharing is possible during training [2305.17141].

| Architectural Variant         | Actor Input                | Critic Input          |
|------------------------------|----------------------------|-----------------------|
| Standard MAPPO               | $o_i$                      | $s$, $o_1,\ldots,o_n$ |
| Multi-Head MAPPO             | $o_i$, history, agent idx  | graph($o_1,\ldots,o_n$)|
| Communication-Augmented      | $o_i$, received messages   | $s$, $o_1,\ldots,o_n$ |

## 4. CTDE in Practical Domains and Empirical Insights

MAPPO under CTDE has shown broad applicability and empirical advantages across diverse MARL domains:
- **Edge Server Monitoring**: Decentralized dispatchers learn joint server query and job scheduling policies, with partial observability handled via AoI-augmented observation. Centralized critics lead to robust throughput-cost tradeoffs and good scalability as agent count increases [2509.19079].
- **Spatial Public Goods Games**: MAPPO-LCR extends MAPPO to massive agent populations (e.g., 40,000 agents), leveraging a centralized critic over the global state. Local cooperation rewards drive stable emergence of cooperation and reduce variance versus independent PPO [2512.17187].
- **UAV-Assisted 5G Slicing**: MAPPO with a centralized critic achieves best QoS-energy tradeoff under severe interference, outperforming MADDPG and DQN baselines in both urban and rural topologies for latency, SINR, and throughput [2512.03835].
- **Traffic Signal Control**: Multi-head CTDE MAPPO constructs enable personalized policy representations for intersections with distinct statistics, outperforming naïve parameter scaling [2503.07678].
- **360° Video Streaming**: MAPPO-CTDE integrates with spatial-temporal transformers to maximize QoE under partial observability, yielding significant improvements over diverse DRL and MPC baselines [2405.07759].
- **Multi-UAV LoRa Networks**: GLo-MAPPO optimizes UAV trajectories, resource allocation, and associations with energy efficiency, achieving strong performance and stability via CTDE [2509.17676].
- **Swarm Pursuit Avoidance**: Imitation learning, policy distillation, and alternative training atop CTDE-MAPPO yield scalable and communication-efficient decentralized controllers [2311.02912].

## 5. Decentralized Execution and Deployment Characteristics

After centralized training, all agents discard the centralized critic and execute solely using their local actor $\pi_\theta(a_i|o_i)$. Each agent acts independently, basing its actions strictly on its own observation stream. No additional coordination, synchronization, or reliance on globally available information is allowed at test/deployment time [2405.06161, 2509.19079]. CTDE makes MAPPO uniquely attractive for domains with real-time, communication-constrained, or privacy-preserving requirements.

## 6. Stability, Scalability, and Convergence Properties

The use of a global value estimator (centralized critic) and surrogate advantage estimation in CTDE suppresses the nonstationarity introduced by concurrent policy updates, facilitating stable learning and improved sample efficiency relative to purely decentralized or independent PPO approaches [2405.06161, 2512.17187]. Empirical results consistently show that MAPPO-CTDE stabilizes convergence, especially in environments with large population-level payoff coupling or dynamic state-action interactions.

Experiments in public goods games reveal that centralized critics enable sharp phase transitions and deterministic convergence, outperforming decentralized baselines both in mean behavior and in run-to-run variability [2512.17187]. Scalability to large agent populations is preserved as execution policies scale linearly in agent count and memory footprint, since each agent’s execution policy is compact and independent post-training [2509.19079].

## 7. Misconceptions, Limitations, and Open Questions

A common misconception is that CTDE-based MAPPO can be deployed with centralized information—this is not the case; full joint information is used strictly during training, with execution fully decentralized. MAPPO can appear similar to parameter-sharing PPO, but crucially differs in its use of a joint/global critic in learning [2405.06161]. Open challenges include scaling critic architectures to even larger and more heterogeneous agent sets, handling dynamically changing agent populations, more complex communication constraints, and extending CTDE-MAPPO to domains with sparse rewards or delayed credit assignment.

---
**Key References:**  
- "An Initial Introduction to Cooperative Multi-Agent Reinforcement Learning" [2405.06161]  
- "MAPPO for Edge Server Monitoring" [2509.19079]  
- "MAPPO-LCR: Multi-Agent Policy Optimization with Local Cooperation Reward in Spatial Public Goods Games" [2512.17187]  
- "Multi-Agent Deep Reinforcement Learning for UAV-Assisted 5G Network Slicing: A Comparative Study of MAPPO, MADDPG, and MADQN" [2512.03835]  
- "Using a single actor to output personalized policy for different intersections" [2503.07678]  
- "Research on Multi-Agent Communication and Collaborative Decision-Making Based on Deep Reinforcement Learning" [2305.17141]  
- "MADRL-Based Rate Adaptation for 360° Video Streaming with Multi-Viewpoint Prediction" [2405.07759]  
- "GLo-MAPPO: A Multi-Agent Proximal Policy Optimization for Energy Efficiency in UAV-Assisted LoRa Networks" [2509.17676]  
- "Imitation Learning based Alternative Multi-Agent Proximal Policy Optimization for Well-Formed Swarm-Oriented Pursuit Avoidance" [2311.02912]

Source: https://www.emergentmind.com/topics/centralized-training-decentralized-execution-ctde-with-mappo