Centralized Critic Grouped Actor (CCGA)

• The paper introduces CCGA, a scalable extension of MADDPG that restricts each critic's input to an agent's k-nearest neighbors, thus keeping the critic's input size constant.
• CCGA uses Euclidean distance to form local neighborhoods, effectively balancing centralized critic objectives with decentralized actor learning.
• Empirical results in cooperative and adversarial Multi-Particle Environment tasks show CCGA achieves faster convergence and superior runtime scaling compared to standard MADDPG.

Centralized Critic Grouped Actor (CCGA) is the name assigned in "Scalable Neighborhood-Based Multi-Agent Actor-Critic" to MADDPG-K, a scalable extension to Multi-Agent Deep Deterministic Policy Gradient (MADDPG) that addresses the computational limitations of centralized critic approaches by restricting each agent’s critic to the observations and actions of the agent itself and its $k$ closest agents under a chosen metric, taken in the reported experiments to be Euclidean distance. In this formulation, the critic input remains constant in size as the total number of agents grows, while the method retains the centralized-critic objective of mitigating multi-agent non-stationarity. The reported study positions CCGA as a compromise between fully centralized critics and independent learners, with empirical validation on cooperative and adversarial Multi-Particle Environment tasks showing competitive or superior performance compared to MADDPG, faster convergence in cooperative settings, and better runtime scaling as the number of agents grows [2604.18190].

1. Conceptual basis and relation to centralized critics

Multi-agent actor-critic methods extend single-agent policy-gradient algorithms to shared environments in which multiple learners interact. In the reported formulation, each agent $i$ maintains a policy, or actor, $\mu_i(o_i;\phi_i)$ that maps its local observation $o_i$ to an action $a_i$, and a critic $Q_i(\cdot;\theta_i)$ that estimates expected return. A central difficulty is non-stationarity: as each agent updates its policy, the environment seen by any one agent changes. Centralized critics alleviate this by conditioning each critic on the joint observations and actions of all agents, as in
[
Q_i^{\rm MADDPG}(o_1,\dots,o_n,\;a_1,\dots,a_n)\,.
]
The limitation identified for this design is that the input dimension of the critic grows linearly with the number of agents $n$, and the cost of every forward and backward pass scales as $\mathcal O(n^2)$ per update because of matrix multiplications on an input of size $n(o+a)$ [2604.18190].

CCGA modifies only the critic’s scope of conditioning. Rather than ingesting the full joint observations and actions, each agent’s critic receives information from a fixed-size neighborhood. This preserves a centralized signal, but only locally. The actor remains decentralized in the usual sense that it maps the local observation $o_i$ to the action $a_i$. The method therefore targets the dominant scalability bottleneck of centralized critics without abandoning the actor-critic decomposition that MADDPG uses.

A possible misconception is that CCGA eliminates all quadratic dependence on the number of agents. The paper does not make that claim. Instead, it states that the quadratic cost it retains arises from cheap scalar distance computations rather than from the expensive neural-network matrix multiplications that bottleneck standard MADDPG. This distinction is central to the method’s motivation [2604.18190].

2. Neighborhood construction and grouped critic structure

CCGA defines neighborhoods through Euclidean $k$-nearest neighbors. Let agent positions in a $d$-dimensional space be ${x_i}{i=1}^n$, with $x_i\in\mathbb R^d$. The distance metric is
[
\mathrm{dist}(i,j) = |x_i - x_j|_2\,.
]
For each agent $i$, the neighborhood index set is
[
\mathcal N_i^k := \arg\min{\substack{S\subset{1,\dots,n}\setminus{i}\|S|=k}} \sum_{j\in S}\mathrm{dist}(i,j),
]
that is, the $k$ distinct agents excluding $i$ with smallest Euclidean distance to $i$ [2604.18190].

The critic for agent $i$ is then written as
[
Q_i^{\rm CCGA}\bigl(o_i,a_i,\;o_{\mathcal N_i^k},a_{\mathcal N_i^k}\;;\,\theta_i\bigr),
]
where $o_{\mathcal N_i^k}=[o_{j_1},\dots,o_{j_k}]$ and $a_{\mathcal N_i^k}=[a_{j_1},\dots,a_{j_k}]$ are the stacked observations and actions of the $k$ nearest neighbors. The paper also gives the equivalent set-based notation
[
Q_i^{\rm CCGA}\bigl(o_i,a_i,\; {o_j,a_j}_{j\in\mathcal N_i^k}\;;\,\theta_i\bigr).
]

This grouped input structure is the defining architectural feature of CCGA. It keeps the critic input size constant as $n$ grows, since the critic sees only the agent itself and $k$ neighbors rather than all $n$ agents. The paper states that this preserves most of the relevant interaction information while drastically reducing the computation in the critic network [2604.18190]. A plausible implication is that the method encodes an explicit locality bias: interaction structure is assumed to be dominated by nearby agents, at least in the targeted spatial environments.

3. Optimization objective and update procedure

Each critic is trained by minimizing a Bellman-error loss over sampled transitions $(s,a,r,s')$, with $s=(o_1,\dots,o_n)$, $a=(a_1,\dots,a_n)$, and $r=(r_1,\dots,r_n)$. For agent $i$, the loss is
[
L(\theta_i)\;=\;\mathbb{E}_{(s,a,r,s')}!\Bigl[\bigl(Q_i^{\rm CCGA}(s,a;\theta_i)\;-\;y_i\bigr)^2\Bigr],
]
with one-step target
[
y_i \;=\; r_i \;+\;\gamma\, Q_i^{\rm CCGA}\bigl(s',\,a';\;\theta_i'\bigr)\big|\, a'_j=\mu_j(o'_j;\phi_j')\;\forall\,j\in{i}\cup\mathcal N_i^k,
]
where $\theta_i'$ and $\phi_j'$ are target-network parameters and $\gamma$ is the discount factor [2604.18190].

The actor update follows the deterministic policy gradient
[
\nabla_{\phi_i}J(\phi_i) \;=\; \mathbb{E}{s}\Bigl[\, \nabla{a_i}Q_i^{\rm CCGA}(s,a)\bigm|{a_i=\mu_i(o_i)}\, \nabla{\phi_i}\mu_i(o_i)\Bigr],
]
with the critic gradient back-propagated only through agent $i$’s action input. This maintains the standard decentralized-actor, centralized-critic pattern, but with the critic centralized only over the local neighborhood.

The procedural specification is explicit. At each environment step, the method observes positions $x_i$ and observations $o_i$ for all agents, selects actions $a_i=\mu_i(o_i;\phi_i)+\text{noise}$, executes the joint action, observes $r$ and $o'$, computes neighbor indices $\mathcal N_i^k = kNN(x_i,{x_j}_{j\ne i})$, and stores the transition $(o,a,r,o',{\mathcal N_i^k,\mathcal N_i^{k'}})$ in the replay buffer. Updates occur every $U$ steps; for each sampled batch and each agent, the procedure builds critic inputs using stored $\mathcal N_i^k$, computes the target $y_i$, updates the critic, updates the actor, and then updates the target networks via
[
\theta_i' \leftarrow \tau\theta_i + (1-\tau)\theta_i', \qquad
\phi_i' \leftarrow \tau\phi_i + (1-\tau)\phi_i'.
]
A key implementation point is that neighbor computation kNN(x_i,\cdot) is done once per environment step and stored in the buffer [2604.18190].

4. Complexity analysis and scalability claim

The complexity analysis is one of the paper’s main contributions. Let $n$ denote the number of agents, $d$ the state-space dimension, $k$ fixed, $b$ the batch size, $h$ the hidden-layer size, and $l$ the number of layers. In the reported experiments, $d=2$. For data collection, CCGA computes all $\binom n2$ pairwise distances in $\mathcal O(d\,n^2)$ time per step. The paper emphasizes that these are cheap scalar operations [2604.18190].

For training updates, each agent’s critic processes input of size $k(o+a)$, so the total critic cost is
[
\mathcal O\bigl(n\,b\,[\,k(o+a)\,h + l\,h^2\,]\bigr),
]
which is independent of $n$ in the critic-input term because $k$ is fixed. By contrast, standard MADDPG has $\mathcal O(n^2)$ matrix multiplies in the critic per update. The stated consequence is that CCGA shifts all remaining $n^2$ cost into inexpensive distance computations and achieves constant-size critic cost per agent [2604.18190].

The significance of this decomposition is methodological rather than merely asymptotic. In standard centralized-critic implementations, the computational bottleneck is not only that the input grows, but that growth occurs inside the critic network, where it amplifies the cost of forward and backward passes. CCGA moves the quadratic dependence outside the network into neighborhood construction. This suggests that the practical speedup depends not simply on asymptotic order, but on the relative cost of scalar distance computation versus neural-network matrix multiplication in the training system.

5. Empirical behavior in Multi-Particle Environment tasks

The reported empirical evaluation uses Multi-Particle Environment tasks, specifically Simple Spread and Simple Adversary, and compares CCGA with standard MADDPG under identical hyperparameters. Each learning curve is averaged over 10 seeds [2604.18190].

On Simple Spread with $K=2$, the final average episodic return is reported as follows:

N agents	MADDPG	CCGA (MADDPG-K)
3	$-9.8 \pm 0.4$	$-9.7 \pm 0.5$
5	$-15.2 \pm 0.8$	$-13.6 \pm 0.6$
7	$-21.7 \pm 1.0$	$-20.9 \pm 0.9$
9	$-28.0 \pm 1.2$	$-27.5 \pm 1.1$

The wall-clock time per 100 episodes on Simple Spread is reported as:

N agents	MADDPG (s)	CCGA (s)
10	120	110
30	360	250
60	720	460

The paper’s summary of these results is that Figure 1 shows CCGA learns faster and reaches equal or better returns as $N$ grows, especially in cooperative Simple Spread, and Figure 2 shows that beyond $N\approx 45$, CCGA is substantially faster per 100 episodes. In adversarial Simple Adversary, CCGA matches MADDPG in final reward and often converges more smoothly [2604.18190].

These results support two distinct claims. First, restricting the critic to local neighborhoods does not, in the reported tasks, produce an obvious performance penalty relative to the fully centralized baseline. Second, the runtime benefit becomes more pronounced as the number of agents increases. The combination is important because a purely computational improvement with degraded return would not support the stated objective of scalable centralized-critic training.

6. Design choices, limitations, and scope of applicability

The paper identifies three main benefits of CCGA: constant-size critic input decouples per-agent critic cost from total $n$; locality bias can speed learning by focusing on the most relevant other agents; and the remaining $n^2$ cost is a cheap distance computation that is easily parallelized [2604.18190]. These points delimit the intended operating regime: many-agent settings in which local interactions dominate and centralized critics are desirable but fully global conditioning is too expensive.

Two design choices are emphasized. First, Euclidean distance is described as natural in spatial environments, but any metric or learned embedding can be substituted. Second, the hyperparameter $k$ trades off information captured versus input size, and its optimum is environment-dependent. This indicates that CCGA is not tied in principle to Euclidean geometry, although the reported implementation uses Euclidean $k$-nearest neighbors.

The limitations are also explicit. Brute-force neighbor search is $\mathcal O(n^2)$ per step; the paper notes that this can be remedied by KD-Trees or Approximate Nearest Neighbors for large $n$. For highly global tasks where distant agents matter, small $k$ may omit critical signals. There is also additional complexity in buffering neighbor indices and sorted inputs [2604.18190].

A common interpretive error would be to treat CCGA as universally superior to full centralized critics. The reported evidence is narrower: the method is validated on Simple Spread and Simple Adversary, and the paper frames $k$ as environment-dependent. This suggests that CCGA is best viewed as a principled scalability mechanism whose effectiveness depends on whether the task’s interaction graph is sufficiently local. Where that assumption holds, the architecture offers a tractable middle ground between full joint conditioning and independent learning; where it fails, the locality constraint may exclude information that a fully centralized critic would capture.