---
title: 'CTDE: Centralized Training & Decentralized Execution'
url: https://www.emergentmind.com/topics/centralized-training-with-decentralized-execution-ctde
type: topic
---

# CTDE: Centralized Training & Decentralized Execution

Centralized Training with Decentralized Execution (CTDE) is a foundational paradigm in cooperative multi-agent reinforcement learning (MARL), where agents are jointly trained using additional global or shared information but are later deployed to act independently using only local observations. CTDE addresses the core challenges of nonstationarity, credit assignment, and sample inefficiency that arise when multiple adaptive agents interact, while enabling practical scalability and robustness in real-world multi-agent systems. This paradigm underlies a broad array of value function factorization, centralized critic, and policy distillation methods, and continues to inspire substantial research seeking to bridge the gap between joint policy learning and decentralized autonomy.

## 1. Core Principles and Formal Structure

CTDE separates the learning and execution phases: during centralized training, each agent has access to global signals (joint state, joint actions, other agents' policies, or communication) that are unavailable or restricted during decentralized execution, where each agent conditions its decisions strictly on individual observations or limited local context [2409.03052][2405.06161].

The canonical mathematical structure of CTDE is as follows. For a set of agents $\{1,\dots,N\}$ in a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), the global policy during training factors as
$$
\pi_\mathrm{joint}(\mathbf{a}|\mathbf{h}) = \prod_{i=1}^N \pi_i(a_i | h_i)
$$
where $h_i$ is the agent's local (possibly recurrent) history. Centralized critics—or mixing networks for value-based approaches—can leverage access to the joint state $s$, joint action $\mathbf{a}$, and joint history $\mathbf{h}$. During execution, every $\pi_i$ is constrained to rely solely on $h_i$.

Key advantages of CTDE include:
- Improved coordination via improved exploration and more accurate credit assignment (through centralized critics or mixing networks).
- Enhanced sample efficiency by exploiting global feedback channels, reducing the impact of nonstationarity prevalent in independent learning schemes.
- Scalability and practical deployment, as execution does not require synchronized global communication or central controllers [2409.03052][2405.06161].

## 2. Representative Algorithmic Families

Several influential algorithmic frameworks formalize the CTDE principle. These can be grouped into two principal categories:

### A. Value Function Factorization

These methods use a joint action-value function $Q_\mathrm{tot}$, factored into agent-level contributions, typically satisfying a monotonicity or individual-global-max (IGM) condition for decentralized optimality:
$$
Q_\mathrm{tot}(\tau, \mathbf{a}) = f(Q_1(\tau_1,a_1),\dots,Q_N(\tau_N,a_N),s)
$$
with the constraint $\partial Q_\mathrm{tot} / \partial Q_i \geq 0$ for all $i$ (as in QMIX), guaranteeing that the joint greedy action can be assembled from per-agent argmax choices [2409.03052][2405.06161].

Notable methods include:
- **VDN (Value Decomposition Networks)**: additive factorization $Q_\mathrm{tot}=\sum_i Q_i$.
- **QMIX**: monotonic mixing network leveraging global state for more expressive, but still factorizable, joint Q-values.
- **QTRAN, QPLEX, and TVDO**: relaxations or nonlinear generalizations of the monotonicity/IGM condition, the last using Tchebycheff-based aggregation for provably tight IGM satisfaction [2306.13979].

### B. Centralized Critic Methods

Actor-critic approaches (MADDPG, MAPPO, COMA) use centralized critics $Q^i(s,\mathbf{a})$ or $V(s)$ during training—evaluating joint state-action pairs to provide low-variance policy gradients—while directly learning decentralized actors $\pi_i(a_i|o_i)$. For example, in MADDPG, the policy gradient for agent $i$ is:
$$
\nabla_{\theta_i} J(\theta_i) = \mathbb{E}\left[ \nabla_{\theta_i} \mu_i(a_i|o_i) \nabla_{a_i} Q_i(o_1,\dots,o_N, a_1,\dots,a_N) \right]
$$
Only the critics require joint information during training; actors are fully decentralized at execution. [2404.12520][2409.03052].

A table summarizing these approaches:

| Approach        | Centralized Signal Used | Decentralization at Execution |
| --------------- | ---------------------- | ---------------------------- |
| VDN/QMIX        | Mixing network, global state | Fully decentralized         |
| MADDPG/COMA/MAPPO | Centralized critic(s)    | Fully decentralized         |
| QPLEX/TVDO      | Duplex/flexibly mixed Q  | Fully decentralized         |

## 3. Techniques for Bridging Global and Local Policies

CTDE research has developed specialized techniques to ensure that coordination and information learned using global context can be faithfully transferred to decentralized policies:

- **Policy Distillation:** Centralized policies are distilled into decentralized policies by minimizing the divergence between full-state policies and local policy networks, as in CTEDD [1910.09152].
- **Centralized Teacher / Decentralized Student:** A centralized teacher learns with global observations; decentralized students are optimized by matching their Q-values to the teacher's outputs via knowledge distillation, as in CTDS [2203.08412].
- **Personalization of Global Information:** Personalizing global state embeddings for each agent before distillation, as in PTDE [2210.08872], addresses the issue of irrelevant or redundant global features for locally specialized agents.
- **Centralized Advice & Self-Pruning:** CADP [2305.17352] employs explicit agent-to-agent advice exchange (via cross-attention) during training, then prunes communication so as to produce truly decentralized executing policies.

These techniques are mathematically characterized by dual-loss schemes, where the standard TD or policy loss is augmented by a knowledge distillation, imitation, or attention-based auxiliary loss.

## 4. Theoretical Guarantees and Consistency Conditions

A major challenge in CTDE is ensuring that the decentralized execution of learned policies can recover the optimal joint policy. The key consistency requirement is the Individual-Global-Max (IGM) condition, formalized as:
$$
\operatorname{argmax}_{\mathbf{a}} Q_{\mathrm{tot}}(\cdot,\mathbf{a}) = (\operatorname{argmax}_{a_1} Q_1(\cdot, a_1),\dots, \operatorname{argmax}_{a_N} Q_N(\cdot, a_N))
$$
TVDO [2306.13979] provides a nonlinear Tchebycheff aggregation guaranteeing both necessity and sufficiency for IGM without artificial affine or monotonicity constraints. Theoretical analyses further show that under certain factorization choices and network architectures, the learned decentralized policies can provably reproduce the optimal joint policy.

For centralized critics, convergence and variance properties of policy gradient estimators have been rigorously analyzed, with results indicating equivalence in expected gradient direction between centralized and independent critics when convergence is reached, but generally higher variance in centralized settings [2404.12520].

## 5. Scalability, Partial Observability, and Practical Limitations

CTDE techniques are motivated by their ability to mitigate the "curse of dimensionality" encountered in fully centralized methods:

- **Scalability:** By factorizing value functions or critics and limiting input dimensionality (for example, using local neighborhoods in a network-aware critic [2312.04371]), CTDE scales to tasks with dozens or even hundreds of agents.
- **Partial Observability and Robustness:** Approaches such as generative inference [1910.03058] and intrinsic reward shaping [2406.18152] extend CTDE to environments with limited or unreliable local observations and sparse rewards, by equipping agents with models for inferring other agents’ states or for aligning action tendencies.
- **Communication Constraints:** CTDE does not require online communication for execution, but communication during training or via compact message passing can be leveraged for credit assignment or coordination in more complex settings [2205.15023][2312.04371].

Limitations persist. Performance gaps between teacher and student exist in extreme partial observability [2203.08412]. Large-scale centralized critics or mixing networks can still impose significant computational or bandwidth burdens during training. There remains a risk of overfitting to global features that are absent at execution time, and research has shown that naively using identical global information for all agents can even impair learning [2210.08872]. Curriculum learning, smart initialization of policy structures, and progressive distillation of central information are areas of ongoing research [2412.15639].

## 6. Extensions, Applications, and Future Directions

CTDE has been adapted and extended for a broad range of domains:
- **Infrastructure Management:** DDMAC-CTDE enables multi-agent scalable decision-making in large, partially observable transportation infrastructure environments, outperforming traditional heuristics and rule-based optimization [2401.12455].
- **Smart Grids and Electric Vehicle Charging:** CTDE-DDPG frameworks yield improved cost efficiency, fairness, and demand-side management in large-scale, privacy-sensitive power grid scenarios. Notably, the local actor networks are deployable without communication at runtime [2404.12520].
- **Networked Control:** Network-aware truncation and localized aggregation of information in critics allow CTDE to scale to energy grid regulation tasks with hundreds of agents [2312.04371].

Recent innovations include model-based extensions using imaginary rollouts to improve sample efficiency (as in MAMBA [2205.15023]), advanced communication protocols for latent state sharing, and structured memory/knowledge graph systems for scalable, fully decentralized generative agents in open-world environments, which critique CTDE’s rigidity and centralization [2502.05453].

Open research directions noted across the literature include:
- Adaptive selection of centralized information during training.
- Robustness of value factorization in highly partially observable or non-stationary settings.
- The theoretical limits of policy distillation and teacher-student mismatch.
- Unification with consensus-based, fully decentralized, and permutation-equivariant alternatives [2508.11706], as well as hybrid approaches (MAGPO [2507.18059]) that seek to systematically transfer centralized advances to decentralized execution.

## 7. Summary and Outlook

Centralized Training with Decentralized Execution is established as a central paradigm for enabling high-performance, scalable coordination in cooperative MARL. The research landscape demonstrates rapid algorithmic evolution—from factorized Q-learning, actor-critic models, and policy distillation, to advanced intrinsic reward shaping, tacit learning frameworks, and LLM-powered decentralized generative agents—each seeking to overcome the limitations of partial observability, scalability, and rigid centralization. The continued integration of theoretical guarantees (such as IGM satisfaction and policy improvement monotonicity) with algorithmic innovation positions CTDE as both the practical and conceptual fulcrum for advancing distributed intelligence in multi-agent systems [2409.03052][2405.06161][2203.08412][2210.08872][2305.17352][2306.13979][2312.04371][2401.12455][2404.12520][1910.09152][1910.03058][2205.15023][2406.18152][2412.15639][2502.05453][2503.05970][2507.18059][2508.11706].

Source: https://www.emergentmind.com/topics/centralized-training-with-decentralized-execution-ctde