CTDE in Multi-Agent Reinforcement Learning

• CTDE is a framework in multi-agent reinforcement learning that uses centralized training with global state information to learn decentralized policies based solely on local observations.
• It employs techniques like value decomposition, centralized critics, and teacher-student distillation to balance global coordination with limited communication during execution.
• Recent architectures such as QMIX, CTDS, and PTDE demonstrate CTDE’s effectiveness across domains like SMAC, UAV swarms, and power grid control, improving sample efficiency and scalability.

Centralized Training with Decentralized Execution (CTDE) is the prevailing paradigm in cooperative multi-agent reinforcement learning (MARL), designed to harness the advantages of full information during training while enabling scalable, communication-free policies at execution. CTDE methods rely on centralized critics, value decomposition, or other forms of joint signal during optimization, yet strictly constrain each agent’s deployed policy to depend only on private observation histories. The field’s rapid evolution has produced theoretically grounded architectures, value- and actor-critic factorization schemes, sample-efficient extensions, and numerous specialized variants attuned to partial observability, communication bottlenecks, or large-scale agent systems (Zhao et al., 2022, Liu et al., 20 Dec 2024, Egorov et al., 2022, Li et al., 2023, Chen et al., 2022, Zhou et al., 2023, Xu et al., 2023, Shojaeighadikolaei et al., 18 Apr 2024, Zhang et al., 5 Feb 2025, Saifullah et al., 23 Jan 2024, Park et al., 2022, Li et al., 24 Jul 2025, Amato, 10 May 2024, Leroy et al., 2022, Yu et al., 2019, Marchesini et al., 2021, Amato, 4 Sep 2024).

1. Foundational Principles and Mathematical Formulation

Under CTDE, a Markov game, Decentralized POMDP, or networked MDP is defined, with $N$ agents indexed by $i=1..N$. Each agent observes local $o_i$ (possibly a partial observation or action-observation history $\tau_i$), samples action $a_i\in\mathcal{A}_i$, and receives a shared team reward $r(s, \mathbf{a})$. In training, the algorithm leverages centralized information—global state $s$, all local observations $\mathbf{o}$, joint action $\mathbf{a}$, possibly joint history $\boldsymbol{\tau}$—to optimize value functions, critics, or policy targets. However, decentralized execution mandates that each $\pi_i(a_i|\tau_i)$ accesses only local trajectory $\tau_i$ (Amato, 4 Sep 2024, Zhao et al., 2022, Amato, 10 May 2024).

Canonical instantiations include:

• Value-decomposition: Learn per-agent $Q_i(\tau_i, a_i)$ and aggregate them via a mixing network:

$$Q_{\rm tot}(\boldsymbol{\tau}, \mathbf{a}; \phi) = f(Q_1, ..., Q_n, s; \phi)$$

Examples: VDN ($f$ is sum), QMIX (monotonic $f$ with nonnegative hypernetwork weights), QPLEX (duplex-dueling factorization). The Individual-Global-Max (IGM) constraint ensures decentralized action selection is compatible with joint maximization (Zhao et al., 2022, Amato, 10 May 2024).

• Centralized-critic, decentralized-actor: Each actor $\pi_i(a_i|\tau_i)$ is updated using gradients from a centralized critic $Q_\phi$:

$$\nabla_{\theta_i}J_i \propto \mathbb{E}[\nabla_{\theta_i}\log \pi_i(a_i|\tau_i)\nabla_{a_i} Q_\phi(s,\mathbf{a})]$$

Examples: MADDPG, MAPPO, COMA (Amato, 10 May 2024, Chen et al., 2022, Amato, 4 Sep 2024).

These methods formally balance global coordination/credit assignment and ecological deployment constraints.

2. Major Architectural Variants and Methodological Developments

CTDE has evolved through a series of architectures that differentially exploit, factorize, or distill centralized knowledge:

Variant	Centralized Mechanism	Decentralized Execution
VDN/QMIX/QPLEX	Monotonic mixing/duplex-dueling network	Each agent uses $Q_i(\tau_i, a_i)$
Centralized Teacher–Decentralized Student (CTDS) (Zhao et al., 2022)	Global-observation teacher Q-mixing, per-agent student distillation	Online only student Q; teacher network discarded
Personalized Training with Distilled Execution (PTDE) (Chen et al., 2022)	Agent-personalized global embeddings, knowledge distillation	Per-agent distilled embedding; no access to global state
CADP (Zhou et al., 2023)	Centralized cross-attention for advising; progressive pruning	Communication/pruning disables all but self-attention
Network-Aware Critics (SNA) (Xu et al., 2023)	Critic truncated to $\kappa$-hop subgraphs/latents	Agent policies depend on local $o_i$ and possibly local network

Many approaches explicitly address the trade-off between expressivity and tractability, e.g., SNA limits critic input to local neighborhoods for $\mathcal{O}(1)$ communication/compute per agent, while PTDE and CTDS utilize distillation to minimize centralized information required at run time.

3. Sample Efficiency, Scalability, and Information Bottlenecks

Scaling CTDE requires addressing several key issues: high-dimensional joint spaces, information overload, partial observability, and credit assignment. Notable advances are:

• Model-based extensions (MAMBA (Egorov et al., 2022)): Each agent learns a per-agent world model with discrete communication, sustaining local latent representations and using short imaginary rollouts to reduce sample complexity.
• Adaptive information gating/selection (SICA (Liu et al., 20 Dec 2024)): Gating and selection blocks filter relevant local features, with supervised or self-supervised regeneration of global context for execution, supporting implicit/tacit coordination even under severe observation constraints.
• Truncated critics (SNA (Xu et al., 2023)): Graph-structured critics operate only on $\kappa$-hop neighborhoods, with $\mathcal{O}(\gamma^{\kappa+1})$ theoretical error. This enables stability at $N=114$ agents in inverter-grid control.

Empirically, approaches such as MAMBA demonstrate an order-of-magnitude reduction in environmental interactions on SMAC and Flatland, outperforming model-free CTDE baselines in both sample efficiency and scalability.

4. Addressing Partial Observability and Communication Limitations

CTDE methods explicitly target settings where direct inter-agent communication at deployment is impossible or severely limited. Solutions include:

• Teacher-student distillation (CTDS (Zhao et al., 2022), PTDE (Chen et al., 2022)), which achieves robust test-time performance even with narrow observation windows by transferring global knowledge through supervised regression.
• Explicit-to-tacit coordination (TACO (Li et al., 2023), SICA (Liu et al., 20 Dec 2024), CADP (Zhou et al., 2023)): Early-phase explicit communication or attention is replaced by learned, locally reconstructible representations, annealed through cosine or linear schedules without notable performance degradation at test time.
• Differentiable belief embedding (team regret minimization with particle filter (Yu et al., 2019)): Each agent maintains local beliefs over hidden state via a neural particle filter, supporting belief-based policy and value networks without test-time communication.

These frameworks demonstrate that gradual or distillation-driven removal of centralized cues during training produces decentralized policies with near-centralized performance.

5. Exploration, Optimization, and Theoretical Guarantees

CTDE research includes rigorous analysis of learning dynamics and exploration:

• Exploration-enhanced CTDE (OPT-QMIX (Zhang et al., 5 Feb 2025)): Introduces a separate optimistic network $f_i$ to bias $\epsilon$-greedy exploration towards rarely sampled, potentially optimal joint actions. The monotonic increment property and action-selection rules increase the sampling frequency of optima, mitigating underestimation in monotonic-mixing methods.
• Monotonic improvement and policy alignment (MAGPO (Li et al., 24 Jul 2025)): Adopts an auto-regressive guider for joint coordinated exploration, with KL-regularized alignment between centralized guider and decentralized learner policies. Monotonic improvement theorems guarantee that each update step does not reduce expected return.

Theoretical results reveal that once critics are converged, CTDE and fully independent actor-critic gradients are unbiased equivalents, but CTDE estimates exhibit higher variance; the trade-off is justified by improved stability and coordination (Shojaeighadikolaei et al., 18 Apr 2024). In model-based CTDE, reward structure and information decay in networked MDPs enable control of truncation errors, with formal bounds proven for SNA (Xu et al., 2023).

6. Empirical Benchmarks and Domain Applications

CTDE architectures consistently dominate collaborative MARL benchmarks:

• StarCraft Multi-Agent Challenge (SMAC): CTDE variants (CTDS, SICA-QMIX, CADP, PTDE-derived) set state of the art across hard and super-hard maps, with win-rate gains up to $+17\%$ over base QMIX/VDN/QPLEX (Zhao et al., 2022, Liu et al., 20 Dec 2024, Zhou et al., 2023, Chen et al., 2022).
• Google Research Football (GRF): Methods such as SICA, CADP, and PTDE demonstrate up to $+20\%$ reward/win-rate improvement, and maintain robustness across changing agent counts and agent role specialization (Liu et al., 20 Dec 2024, Chen et al., 2022).
• Infrastructure and Power Systems: DDMAC-CTDE reduces total management cost by $7.5{-}31\%$ over Condition-Based and VDOT policies in a 96-component network (Saifullah et al., 23 Jan 2024). SNA-MASAC stabilizes and outperforms standard CTDE in voltage regulation for $N=114$ distributed generators (Xu et al., 2023).
• Autonomous Agents/UAVs: CommNet-style CTDE achieves higher convergence and reward in multi-UAV swarms controlling mobile access points (Park et al., 2022).
• Competitive and Mixed Settings: Population-based CTDE training yields more robust, higher-Elo teams in symmetric two-team Markov games and coordinate-to-competition transitions (Leroy et al., 2022).

Ablations frequently confirm that plug-and-play additions such as per-agent global information personalization, selection+regeneration blocks, and progressive annealing schedules directly translate into superior empirical robustness, scalability, and minimal dependency on inter-agent communication at execution.

7. Limitations, Open Questions, and Research Directions

Limitations of classic CTDE include representational restrictions (e.g., monotonicity constraints in QMIX limit expressivity for non-monotonic value landscapes), variance in policy gradients for large agent counts, and the challenge of aligning global centralized signals with agent-specific utility. Recent advances with personalized embeddings, auto-regressive centralized policies, factored critics, and network-aware approximations address many of these, but several areas remain for further investigation:

• Minimal centralization: Determining the theoretical and practical minimal set of global statistics necessary for optimal credit assignment and coordination remains open (Amato, 10 May 2024).
• Beyond cooperation: Extending CTDE with robust, efficient mechanisms for mixed cooperative–competitive and adversarial settings.
• Provable convergence: Developing CTDE algorithms with global-optimality or well-characterized local optimality guarantees in general Dec-POMDPs.
• Sample efficiency at scale: Sustaining order-of-magnitude gains seen in model-based and network-truncated CTDE approaches across a broader set of MARL domains.
• Sim-to-real transfer: Scaling findings from simulated benchmarks to large-scale real-world systems—transportation, power grids, autonomous fleets—while retaining the theoretical performance and safety guarantees.

• CTDS: "CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning" (Zhao et al., 2022)
• SICA: "Tacit Learning with Adaptive Information Selection for Cooperative Multi-Agent Reinforcement Learning" (Liu et al., 20 Dec 2024)
• MAMBA: "Scalable Multi-Agent Model-Based Reinforcement Learning" (Egorov et al., 2022)
• TACO: "From Explicit Communication to Tacit Cooperation: A Novel Paradigm for Cooperative MARL" (Li et al., 2023)
• PTDE: "PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning" (Chen et al., 2022)
• CADP: "Is Centralized Training with Decentralized Execution Framework Centralized Enough for MARL?" (Zhou et al., 2023)
• SNA: "A Scalable Network-Aware Multi-Agent Reinforcement Learning Framework for Decentralized Inverter-based Voltage Control" (Xu et al., 2023)
• CTDE-DDPG: "Centralized vs. Decentralized Multi-Agent Reinforcement Learning for Enhanced Control of Electric Vehicle Charging Networks" (Shojaeighadikolaei et al., 18 Apr 2024)
• OPT-QMIX: "Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning" (Zhang et al., 5 Feb 2025)
• DDMAC-CTDE: "Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management" (Saifullah et al., 23 Jan 2024)
• CommNet CTDE: "Coordinated Multi-Agent Reinforcement Learning for Unmanned Aerial Vehicle Swarms in Autonomous Mobile Access Applications" (Park et al., 2022)
• MAGPO: "Multi-Agent Guided Policy Optimization" (Li et al., 24 Jul 2025)
• VRM: "Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning" (Yu et al., 2019)
• GDQ: "Centralizing State-Values in Dueling Networks for Multi-Robot Reinforcement Learning Mapless Navigation" (Marchesini et al., 2021)
• CTDE overviews: (Amato, 4 Sep 2024, Amato, 10 May 2024)
• Value-based CTDE in two-team Markov games: (Leroy et al., 2022)