---
title: Centralized-Critic Actor-Critic Methods
url: https://www.emergentmind.com/topics/centralized-critic-actor-critic-methods
type: topic
---

# Centralized-Critic Actor-Critic Methods

Centralized-critic actor-critic methods constitute a pivotal approach in multi-agent reinforcement learning (MARL), particularly under the Centralized Training with Decentralized Execution (CTDE) paradigm. In these frameworks, each agent’s policy (actor) is conditioned only on its own observation or history and operates independently at execution-time, while training leverages a centralized critic with access to global information, such as the true system state or the joint action-observation histories. This separation enables the exploitation of centralized information during training for accelerated and stabilized learning, while ensuring decentralized, scalable operation at test time. Despite their empirical popularity, recent theoretical and empirical findings indicate that centralized critics—especially those based solely on state information—can introduce nontrivial bias and variance in policy gradient estimates in partially observable environments, and their benefits are environment-dependent [2201.01221, 2408.14597, 2102.04402].

## 1. Formal Framework for Centralized-Critic Actor-Critic Methods

The standard setting for centralized-critic actor-critic methods in MARL is the Decentralized Partially Observable Markov Decision Process (Dec-POMDP). At each timestep $t$, the environment is in an unobserved global state $s_t \in \mathcal{S}$; each agent $i$ receives a private observation $o_{i,t}$ and executes action $a_{i,t}$ according to its decentralized policy $\pi_{\theta_i}(a_i \mid h_i)$, where $h_i$ is the agent's local action-observation history. Typically, the agents collaboratively optimize a shared, discounted cumulative reward:
$$
J(\theta) = \mathbb{E}\left[ \sum_{t=0}^\infty \gamma^t r_t \right]
$$
where $\theta$ collects all actor parameters.

During centralized training, the critic is parameterized as either a value function $V_\varphi(s)$ or a state-action value function $Q_\varphi(s, \vec a)$, having access to the full system state $s$ or joint histories. Actor update rules—such as those following the policy gradient theorem—use the critic to evaluate actions and compute advantages, often via estimators of the form:
$$
\nabla_{\theta_i} J(\theta) = \mathbb{E} \left[ \nabla_{\theta_i} \log \pi_{\theta_i}(a_i \mid h_i) \left(Q^\pi(s, \vec a) - b(s)\right) \right]
$$
where $b(s)$ is a baseline derived from the central critic [2201.01221, 2408.14597].

## 2. Critic Architectures and Variants

Centralized critics can leverage various forms of available information, leading to several architecture variants:

| Critic Variant        | Input Features                    | Example Usage                     |
|---------------------- |-----------------------------------|-----------------------------------|
| State-based (SC)      | Global state $s$                  | SMAC, particle worlds (deterministic or nearly fully observable) [2201.01221, 2408.14597]|
| History-based (HC)    | Joint histories $h$               | Dec-Tiger, Box-Pushing (partial observability dominates) [2201.01221, 2408.14597]|
| Joint-Observation     | $(o_1,...,o_{|I|})$               | Cooperative navigation [2201.01221, 2408.14597]|
| Hybrid (HSC)          | $(h, s, a)$                       | Combines information for improved bias/variance [2201.01221]|

Architecturally, SC typically employs an MLP embedding of $s$; HC applies RNNs to full histories; HSC concatenates RNN and global state embeddings to inform value estimation [2201.01221].

## 3. Bias and Variance Analysis in Centralized-Critic Gradients

Substantial bias and variance effects emerge in the use of state-based centralized critics under partial observability:

- **Bias**: When multiple distinct histories $h$ map onto the same state $s$, but those histories yield different value-to-go, the expectation over $Q^\pi(s, a)$—used in the policy gradient—does not match the correct history-based value $Q^\pi(h, a)$. As established by rigorous analysis [2201.01221, 2408.14597]:
  $$
  Q^\pi(h, a) \neq \mathbb{E}_{s \sim \rho(s|h)} [ Q^\pi(s, a) ]
  $$
  leading to policy gradient bias.

- **Variance**: Even when bias is eliminated (e.g., if histories in each state are value-equivalent), the state-based estimator's variance is provably greater or equal to the history-based estimator's variance due to the random mapping from histories to states [2201.01221]:
  $$
  \mathrm{Var}\left[ Q^\pi(s,a) \nabla \log \pi(a|h) \right] \ge \mathrm{Var}\left[ Q^\pi(h,a) \nabla \log \pi(a|h) \right]
  $$

- **Empirical effects**: Domains such as Dec-Tiger and Box-Pushing (with strong information-gathering requirements and deep history dependencies) expose detrimental bias and variance effects, often causing state-based critics to converge to suboptimal policies, whereas history- or hybrid-based critics yield better performance and stability [2201.01221, 2408.14597].

## 4. Empirical Evidence and Environment Properties

Empirical comparisons indicate that the architectural choice of the critic must match the environment's observability and history structure:

| Domain                            | Information Dependency    | SC (State Critic) | HC (History Critic) | HSC (Hybrid)   |
|----------------------------------- |--------------------------|-------------------|--------------------|---------------|
| SMAC (full/near-full obs)         | Low (reactive optimal)   | Best or tied      | Tied               | Tied          |
| Meeting-in-a-Grid                  | Low                      | Fastest, optimal  | Tied               | Tied          |
| Dec-Tiger, Box-Pushing, Cleaner    | High (history critical)  | Suboptimal        | Outperforms SC     | Best          |

On tasks like StarCraft Multi-Agent Challenge or cooperative navigation with minimal history dependence, state-based critics are effective and efficient. In contrast, in environments with necessary information-gathering or latent variable inference, only history- or hybrid-based critics avoid bias and attain optimal (or near-optimal) returns [2201.01221, 2408.14597].

## 5. Centralized Critics in Asynchronous and Structured Multi-Agent Learning

The centralized-critic methodology is extended to domains that require asynchronous execution, temporally extended actions, or structured multi-agent cooperation. In asynchronous MARL, e.g., Mac-IAICC [2209.10113], each agent executes macro-actions using its own decentralized actor but leverages a centralized, global-information critic at training time, with updates triggered by individual macro-action termination events. This design circumvents the need for agent synchronization and supports scalability in complex domains.

In structured domains, such as knowledge base editing (STACKFEED [2410.10584]), a multi-actor, centralized-critic schema assigns each "document" to a separate actor, while a centralized critic decomposes global feedback into targeted per-actor updates. This design ensures credit assignment and mitigates nonstationarity among independently learning agents.

## 6. Design Guidelines, Practical Recommendations, and Limitations

The fundamental trade-offs summarized in recent analyses [2201.01221, 2408.14597, 2102.04402] are as follows:

- **When to Favor State-based Critics (SC)**
  - Observability is strong (local observations are almost sufficient to reconstruct state).
  - Optimal policies are reactive (history does not provide additional value).
  - Accelerated representation learning via compact state representations.

- **When to Favor History-based Critics (HC) or Hybrid Critics (HSC)**
  - Strong partial observability, information-gathering, or long-horizon dependencies.
  - Environments with latent state, hidden variables, or necessity for belief-state tracking.

- **Hybrid Approaches**
  - Combining state and history encodings or interpolating between $V(s)$ and $V(h)$ can reduce both bias and variance.

- **Potential Pitfalls**
  - Centralized critics are not universally beneficial; in some benchmarks, excessive variance or systematic bias in partial observability leads to degraded sample efficiency or failure to converge to optimal coordination.
  - Scalability issues can arise in scenarios with large numbers of agents or high-dimensional state-action spaces.

It is empirically necessary to ablate critic varieties (SC, HC, HSC) on simplified instances of the target environment to estimate the degree to which histories beyond the current state impact optimal behavior and learning efficiency [2201.01221].

## 7. Open Challenges and Future Directions

Despite their centrality in current MARL practice, centralized-critic actor-critic architectures remain an area of active research:

- Systematic analysis of state-based critic bias and variance in large-scale, realistic partial observability scenarios is ongoing [2201.01221, 2408.14597].
- Advanced network architectures—e.g., attention mechanisms, structured critics (see [2507.22782])—are being developed to address the limitations of existing methods, including improved scalability, communication, and credit assignment.
- Application-specific design, e.g., asynchronously updated critics or knowledge-editing with centralized critics [2209.10113, 2410.10584], is expanding the operational capabilities and applicability of these methods.
- The development and empirical validation of hybrid critics, importance-sampling corrections, and variance-reduction techniques remain priority open problems to close the gap between theory and large-scale, high-dimensional MARL practice [2201.01221, 2408.14597].

Centralized-critic actor-critic methods are thus characterized by a nuanced set of trade-offs; their design and deployment require careful matching between agent architecture, critic structure, and the partial observability properties of the target domain.

Source: https://www.emergentmind.com/topics/centralized-critic-actor-critic-methods