Centralized Critic and Decentralized Policies Explained

Updated 12 May 2026

Centralized Critic and Decentralized Policies integrate a central evaluation element with individual agent autonomy, optimizing multi-agent collaboration.
This framework significantly enhances cooperative learning in multi-agent reinforcement learning by efficiently attributing credit and optimizing joint actions.
Research reveals improved decision-making, adaptability, and coordination in complex environments like traffic systems or robotic simulations.

Multi-Agent Counterfactual Effect Decomposition is a framework for quantifying, attributing, and explaining the effects of policy changes, interventions, or actions in systems composed of multiple interacting agents. These decompositions are fundamental for causal inference, credit assignment in reinforcement learning, game-theoretic counterfactual analysis, and the accountability of multi-agent deployments. They combine tools from structural causal modeling, cooperative game theory, and sequential decision-making to rigorously trace how counterfactual interventions propagate through agent policies, state transitions, and environment dynamics to final system outcomes.

1. Formalism and Core Notions

At the foundation of multi-agent counterfactual effect decomposition is the multi-agent Markov Decision Process (MMDP), typically formalized as an MMDP-SCM (Structural Causal Model). Consider $n$ agents, state space $S$ , joint action space $A_1 \times ... \times A_n$ , a transition kernel $T$ , horizon $h$ , and policies $\pi_i(a_i|s)$ for each agent. A trajectory $\tau$ is generated by sequential sampling, and the system outcome (e.g. reward or final state) is denoted $Y$ .

Counterfactual reasoning seeks to answer: For an intervention on an agent’s action (or policy, type, or treatment vector), what is the causal effect on $Y$ ? The total counterfactual effect (TCFE) for a given factual trajectory $\tau$ and an alternative action $S$ 0 (vs. the factual $S$ 1) is:

$S$ 2

This definition is grounded in SCM semantics and ties together potential-outcome and path-specific frameworks (Triantafyllou et al., 2024, Triantafyllou et al., 2023). In reinforcement learning applications, direct analogues arise in the assignment of agent-specific (counterfactual) advantages for policy updating.

2. Agent-Specific and Path-Specific Decomposition

The propagation of an intervention depends both on direct environmental effects (through altered state transitions) and indirect effects (through the induced behavioral changes of other agents responding to the intervention). The decomposition framework isolates these propagation channels:

Total Agent-Specific Effect (tot-ASE): Measures the effect of a change in agent $S$ 3’s action on $S$ 4 that propagates through all future agents' policy responses but not through altered state transitions.
Reverse State-Specific Effect (r-SSE): Quantifies effect components resulting exclusively from state-transition pathways, capturing how much of the counterfactual change is explained by altered state variable trajectories independent of other agents' policy adaptations (Triantafyllou et al., 2024).

This yields a decompositional identity:

$S$ 5

Agent-specific effect decomposition can be further refined by regarding each agent as a "player" in a causal game, assigning marginal effects through Shapley value allocation. For a coalition $S$ 6, the N-specific ASE is:

$S$ 7

Shapley value formulas uniquely distribute $S$ 8 among agents with desirable fairness and efficiency properties (Triantafyllou et al., 2024).

3. Counterfactual Credit Assignment in Multi-Agent Reinforcement Learning

In cooperative multi-agent reinforcement learning (MARL), the joint reward or team return must be ascribed to agent-specific contributions, a challenge known as multi-agent credit assignment. Counterfactual effect decomposition provides actionable estimators via advantage calculations that compare the observed global outcome under the actual joint action with hypothetical outcomes under ablated or resampled agent actions.

COMA (Counterfactual Multi-Agent Policy Gradients): For each agent $S$ 9, advantage is computed as

$A_1 \times ... \times A_n$ 0

where $A_1 \times ... \times A_n$ 1 is a centralized critic (Foerster et al., 2017).

Shapley Counterfactual Credit Assignment: Uses the formal Shapley value to account for all possible coalitions. The value function $A_1 \times ... \times A_n$ 2 evaluates the global value with only agents in $A_1 \times ... \times A_n$ 3 acting and others set to a baseline, and the Shapley value $A_1 \times ... \times A_n$ 4 aggregates agent $A_1 \times ... \times A_n$ 5's marginal contributions across all $A_1 \times ... \times A_n$ 6 (Li et al., 2021).
Counterfactual Group Relative Policy Advantage (CGRPA): Extends COMA with a KL-regularization to the group average policy and aligns credit assignment with curriculum learning in non-stationary environments (Jin et al., 9 Jun 2025).
Counterfactual Credit Policy Optimization (CCPO): Estimates each agent's marginal contribution through explicit counterfactual rollouts with agent $A_1 \times ... \times A_n$ 7 "removed" and forms role-sensitive advantages for improved regret bounds and stability in LLM-based collaborative systems (Li et al., 23 Mar 2026).

The table below contrasts several MARL frameworks utilizing counterfactual decomposition:

Method	Credit Calculation	Decomposition Mechanism
COMA (Foerster et al., 2017)	Counterfactual baseline	Marginalize agent’s action
Shapley Counterfactual Credit (Li et al., 2021)	Coalition-aware marginal effects	Shapley value over all subsets
CGRPA (Jin et al., 9 Jun 2025)	Counterfactual + group KL penalty	Marginalize + coordination bonus
CCPO (Li et al., 23 Mar 2026)	Agent-marginalized trajectory diff	Explicit counterfactual rollouts

Each method guarantees unbiasedness under policy sampling and provides variance reduction over raw return-attribution baselines.

4. Structural Causal Models and Identifiability

Multi-agent effect decomposition relies on formal SCMs for sequential decision processes. Variables include agent actions $A_1 \times ... \times A_n$ 8, states $A_1 \times ... \times A_n$ 9, and exogenous, independently distributed noise $T$ 0. Identification of agent-specific effects in multi-agent SCMs is intricate due to recanting witnesses and path-entanglement. Under noise-monotonicity and finite discrete domains, agent-specific and counterfactual agent-specific effects (cf-ASE) are identified from observational trajectories (Triantafyllou et al., 2023).

The sampling-based estimation proceeds by abduction (sampling noise consistent with observed trajectories), re-simulation under interventions for different agent subsets, and Monte Carlo evaluation of effect differences. The computational cost scales with the sampling horizon and the cardinality of the effect-agent subsets.

5. Continuous-Time and Dynamical Multi-Agent Systems

In networked dynamical systems and continuous-time domains, effect decomposition requires modeling both direct and spillover effects under interference, confounding, and dynamic treatments. The Counterfactual GraphODE (CF-GODE) framework implements a continuous-time latent-variable model parameterized by graph neural ODEs, domain adversarially balanced to remove treatment and interference bias (Jiang et al., 2023). Direct, indirect (spillover), and total effects are extracted by computing counterfactual trajectories under agent-specific or neighbor-specific intervention paths.

This continuous framework subsumes classical discrete-time latent confounder models and enables fine-grained decomposition of time-evolving interventions and feedbacks.

6. Applications and Empirical Evidence

Decomposition frameworks have been empirically validated across economic games, scene graph generation, traffic and robotic coordination, medical treatment planning, and collaborative LLM scenarios.

In LLM-assisted Gridworld and clinical sepsis simulators, agent and state Shapley attributions cleanly isolate primary causal contributors (e.g., the "active" agent or critical state transitions) (Triantafyllou et al., 2024).
In airline market entry, policy effect decomposition can approximate full equilibrium predictions when equilibrium selection rules remain invariant; these tractable decompositions can closely predict realized empirical outcomes (Canen et al., 2020).
In reinforcement learning benchmarks such as StarCraft-II micromanagement, methods using Shapley or COMA-style counterfactual credits achieve decisive improvements in coordination and learning speed (Li et al., 2021, Foerster et al., 2017).
In complex multi-agent basketball scenarios, graph-structured RNN decompositions recover both individual and spillover counterfactual impacts of pass decisions on scoring probabilities (Fujii et al., 2022).

A plausible implication is that, subject to suitable identification and modeling assumptions, counterfactual decomposition approaches yield both interpretable and actionable attributions in multi-agent systems.

7. Limitations, Theoretical Guarantees, and Practical Guidance

Several core limitations and invariance requirements apply:

Invariance for validity: In policy or structural games, decomposition-based predictions coincide with equilibrium-based counterfactuals only when the set of equilibria and selection frequency remains invariant under intervention (Canen et al., 2020).
Identifiability constraints: SCM-based effect modules require exogeneity, noise-monotonicity, and discrete state/action domains for nonparametric identification; continuous extensions remain nontrivial (Triantafyllou et al., 2023, Triantafyllou et al., 2024).
Sample complexity and computation: Monte Carlo estimators and Shapley-based decompositions are factorial in the number of agents unless approximation is used, affecting scalability (Li et al., 2021).
Variance–bias tradeoffs: KL-regularization, value-function approximation, and truncation to top-k actions (in practice) introduce bias but can stabilize learning (Jin et al., 9 Jun 2025).

Best practices involve verifying invariance properties empirically, using partial identification for robustness, and choosing between decomposition and equilibrium approaches according to domain structure and intervention nature (Canen et al., 2020). For SCM-driven environments, algorithmic approaches should match effect identification assumptions (e.g., use of noise-monotonicity or adversarial balancing).

In summary, Multi-Agent Counterfactual Effect Decomposition provides a rigorous suite of tools for tracing and attributing interventions in multi-agent settings, spanning pathwise causal analyses, multi-agent reinforcement learning credit assignment, and equilibrium modeling. The synthesis of causal modeling, cooperative game theory, and empirical RL credit-assignment underpins a foundational methodology for the diagnosis and control of complex agent-based systems (Triantafyllou et al., 2024, Triantafyllou et al., 2023, Foerster et al., 2017, Li et al., 2021, Canen et al., 2020, Li et al., 23 Mar 2026, Fujii et al., 2022, Jin et al., 9 Jun 2025, Jiang et al., 2023).