---
title: 'CoLLM-CC: Centralized Critic for LLM Collaboration'
url: https://www.emergentmind.com/topics/collm-cc
type: topic
---

# CoLLM-CC: Centralized Critic for LLM Collaboration

Searching arXiv for the exact CoLLM-CC paper and related CoLLM disambiguation.
Searching arXiv for related work on decentralized LLM collaboration and actor-critic formulations.
CoLLM-CC is a multi-agent actor-critic method for decentralized large language model collaboration. It was introduced as the centralized-critic member of a pair of Multi-Agent Actor-Critic methods, alongside CoLLM-DC, for settings in which multiple LLM agents act in parallel, each with its own prompt and local history, while optimizing a shared episodic reward. Its defining property is centralized training with decentralized execution: during training, a single critic conditions on joint histories and optional global information, but at inference time the critic is discarded and each agent runs independently from its own local context [2601.21972].

## 1. Terminology, scope, and conceptual position

In the nomenclature of the method, “CC” denotes **Centralized Critic**. CoLLM-CC is therefore not a general label for collaborative LLM systems, but a specific MARL algorithm for decentralized LLM collaboration in which the actor policies remain local and the value function is centralized during training. The method is motivated by two limitations in prior work: hard-coded collaboration protocols that require centralized execution, and Monte Carlo policy-gradient approaches that wait until the end of an episode to compute returns and therefore suffer from high variance and poor sample efficiency, especially in long-horizon or sparse-reward tasks [2601.21972].

The method is explicitly designed for cooperative settings in which agents do not share full state information at inference time. Each agent produces a full textual response, such as a paragraph, a code function, or a game command, based only on its own prompt and history. The collaborative structure is therefore learned in training rather than enforced by a centralized inference-time controller. This suggests a deployment model in which parallelism and flexible system placement are preserved even though optimization uses centralized information.

A recurrent source of confusion is the reuse of the name “CoLLM” in unrelated literature. For example, the composed image retrieval framework “CoLLM” does not define any entity named “CoLLM-CC”; its paper states that the string “CoLLM-CC” does not appear anywhere in that work [2503.19910]. In the MARL context, by contrast, CoLLM-CC is a precise algorithmic designation.

## 2. Decentralized collaboration as a Dec-POMDP

The method formalizes decentralized LLM collaboration as a **Decentralized Partially Observable MDP** with tuple
\[
\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.
\]
Here, $\mathcal{I}$ is the set of LLM agents, $\mathcal{V}$ is the token vocabulary, $C$ is the context window size, $M$ is the maximum number of tokens an agent may output per turn, and $H$ is the episode horizon [2601.21972].

The state at turn $t$ is written as $s_t = (s_t^{\mathrm{sys}}, s_t^{\mathrm{usr}})$. The system-accessible component contains task state, environment state, or tool state, whereas the user-private component is not directly observable. Each agent receives a local observation $o_{i,t}$ in natural-language prompt form, representing a partial and noisy view of the underlying state. The action space is also textual: an agent action $a_{i,t}$ is a sequence of up to $M$ tokens. The joint reward is produced from the combined action and the system-accessible state, and transitions are governed by the environment dynamics.

Because agents are partially observed, the sufficient conditioning variable for each actor is its **local history**
\[
h_{i,t} = \{o_{i,0}, a_{i,0}, \dots, o_{i,t}\},
\]
while the critic may access the **joint history**
\[
\mathbf{h}_t = (h_{1,t}, \dots, h_{n,t}).
\]
Each actor implements a policy $\pi_i(a_{i,t}\mid h_{i,t})$, and the training objective is to maximize expected cumulative reward over the episode. The formulation is notable for treating a full generated response as the action, rather than individual tokens, which aligns the reinforcement-learning abstraction with the granularity of collaborative behavior.

## 3. Centralized-critic, decentralized-actor architecture

CoLLM-CC uses one LLM actor per agent and a separate centralized critic. The actors are not parameter-shared in the reported experiments; instead, the system uses heterogeneous role-specific models, such as paired Qwen variants for summarization or coding roles. Each actor maintains its own KV cache and local dialogue history. At each turn, the previous KV cache is concatenated with new prompt tokens, and only the latest $C$ KV pairs are retained [2601.21972].

A core design choice is to treat the **entire textual response as a macro-action**. For agent $i$, the policy is
\[
\pi_{\theta_i}(a_{i,t}\mid h_{i,t})
= \prod_{\mu=1}^{M}\pi_{\theta_i}(a_{i,t_\mu}\mid h_{i,t}, a_{i,t_{<\mu}}),
\]
and teacher-forcing forward passes are used to compute sequence log-probabilities efficiently in a single pass. This avoids token-level RL credit assignment inside each response and keeps the MARL layer at the level of agent turns.

The centralized critic estimates a joint value from the joint history, optionally augmented with global information such as turn index or progress:
\[
V_{\boldsymbol{\phi}}(\mathbf{h}_t).
\]
It is trained with temporal-difference learning. The TD error is
\[
\delta_t = r_t + \gamma V_{\boldsymbol{\phi}}(\mathbf{h}_{t+1}) - V_{\boldsymbol{\phi}}(\mathbf{h}_t),
\]
and the critic loss is
\[
\mathcal{L}(\boldsymbol{\phi}) =
\mathbb{E}_{\boldsymbol{\pi}}\left[
\sum_{t=0}^{H-1}
\left(
r_t + \gamma V_{\boldsymbol{\phi}}(\mathbf{h}_{t+1}) - V_{\boldsymbol{\phi}}(\mathbf{h}_t)
\right)^2
\right].
\]

Each actor is then updated using a centralized value-based advantage:
\[
\nabla_{\theta_i} J(\theta_i) =
\mathbb{E}_{\boldsymbol{\pi}}\left[
\sum_{t=0}^{H-1}
\rho_{i,t}\,
\nabla_{\theta_i}\log \pi_{\theta_i}(a_{i,t}\mid h_{i,t})\,
\delta_t
\right],
\]
with off-policy importance ratio
\[
\rho_{i,t} =
\frac{\pi_{\theta_i}(a_{i,t}\mid h_{i,t})}
{\pi_{\theta_i,\mathrm{old}}(a_{i,t}\mid h_{i,t})}.
\]
In implementation, the training loop uses a replay buffer and applies advantage clipping, with reported clipping values of $0.2$ in writing and coding and $0.05$ in Minecraft. The overall structure is therefore a textbook centralized-training, decentralized-execution design, but instantiated with full LLMs as both actors and critic.

## 4. Training loop and relation to Monte Carlo and decentralized critics

Training proceeds episodically. For a sampled task, the system initializes prompts for all agents, rolls out the episode by sampling one response from each actor per turn, records reward and next observations, stores the transition in a replay buffer, and then performs multiple gradient epochs over minibatches drawn from that buffer. For each stored action, teacher-forcing is used again during optimization to recompute the response probability under the current actor parameters, which yields stable sequence-level policy gradients [2601.21972].

The paper contrasts CoLLM-CC with two alternatives. The first is **Monte Carlo multi-agent policy gradients**, exemplified by MA-REINFORCE or MAGRPO, which estimate returns directly from rollout trees. Under the independence assumptions used in the paper’s analysis, the variance of the averaged estimator satisfies
\[
\mathrm{Var}(\bar{g}_{i,t}\mid h_{i,t}) = \frac{\sigma^2}{K^{H-t}},
\]
while the number of model calls grows as
\[
N_{\mathrm{call}}(n,K,H) = \frac{nK(K^H-1)}{K-1}.
\]
This is the central argument for why Monte Carlo methods become unattractive as horizon and branching factor increase.

The second alternative is **CoLLM-DC**, in which each agent has a decentralized critic $V_{\phi_i}(h_{i,t})$ depending only on local history. In principle, if these critics converge, the resulting gradients remain unbiased. In practice, the paper reports that local critics are highly exposed to non-stationarity because other agents’ policies are changing while each critic sees only a partial local view. The empirical claim is therefore conditional rather than absolute: Monte Carlo methods and CoLLM-DC are competitive in short-horizon, dense-reward settings, but they both underperform CoLLM-CC in long-horizon or sparse-reward regimes. This suggests that the benefit of centralization lies less in raw asymptotic expressivity than in variance reduction and critic convergence under partial observability.

## 5. Empirical profile across writing, coding, and game-playing

The reported experiments span three domains with increasing horizon and reward sparsity. In **writing collaboration**, the tasks are TLDR summarization and arXiv introduction expansion, both with horizon $H=1$. On TLDR, CoLLM-CC achieved time **1.8 s**, cost **181 tokens**, and score **95.2**, while CoLLM-DC achieved **95.4** and MAGRPO achieved **93.5**. On the arXiv task, CoLLM-CC achieved time **1.9 s**, cost **188**, and score **95.0**, compared with **94.1** for CoLLM-DC and **93.1** for MAGRPO [2601.21972]. These results support the paper’s more nuanced conclusion: in single-turn dense-reward writing, CoLLM-CC is strong and typically best among the MARL methods, but the gap to alternatives is modest.

The picture changes in **coding collaboration** on CoopHE, where horizon increases to $H=2$ and the reward depends heavily on structural integrity, syntax correctness, test pass rate, and cooperation quality. Here the key reported number is **Pass@1**. CoLLM-CC reached **75.2%**, MAGRPO **74.3%**, and CoLLM-DC **59.1%**. The associated training curves show that CoLLM-CC converged around **~2000 timesteps**, MAGRPO needed **~5000 timesteps** to stabilize, and CoLLM-DC oscillated and degraded in later stages. The coding domain therefore provides the clearest evidence for the claim that a centralized critic improves sample efficiency and stability.

In **Minecraft-style games**, the advantage of CoLLM-CC is larger still. In StrBuild, with horizon $H=4$, CoLLM-CC achieved **IoU 68.5** and **same-texture adjacency 7.3**, compared with **IoU 50.6** and **adjacency 13.3** for MAGRPO, and **IoU 44.6** and **adjacency 7.6** for CoLLM-DC. In HouseBuild, CoLLM-CC achieved **IoU 52.7** and **HP 86.4**, compared with **IoU 50.9** and **HP 80.2** for MAGRPO, and **IoU 46.8** and **HP 43.8** for CoLLM-DC [2601.21972]. The training curves reported for both Minecraft tasks indicate that CoLLM-CC steadily improves and converges, MAGRPO is slower and noisier, and CoLLM-DC fails to converge. A plausible implication is that critic centralization becomes increasingly valuable as delayed coordination structure dominates immediate local reward.

## 6. Implementation profile, operating regime, and limitations

The implementation uses full LLMs for both actors and critic rather than lightweight heads. In writing, the actors are **Qwen3-1.7B** models and the centralized critic is also **Qwen3-1.7B**. In coding, the actors are **Qwen2.5-Coder-3B** and **Qwen3-4B-Instruct**, with **Qwen2.5-Coder-3B** as critic. In Minecraft, the actors are **Qwen2.5-3B-Instruct** and **Qwen3-4B-Instruct**, and the critic is **Qwen3-4B-Instruct**. The reported experiments use **full fine-tuning**, not LoRA or other PEFT variants. Decoding temperatures are **0.7** for writing and **0.6** for coding and Minecraft, with maximum output lengths of **256** tokens for writing and coding and **256–512** tokens for Minecraft [2601.21972].

Training hyperparameters vary by domain. Writing uses horizon **1**, rollout buffer size **4**, **20** train epochs per episode, actor learning rate **$5\times10^{-6}$**, critic learning rate **$3\times10^{-6}$**, and advantage clip **0.2**. Coding uses horizon **2**, rollout buffer size **4**, **80** train epochs per episode, actor learning rate **$5\times10^{-6}$**, critic learning rate **$3\times10^{-6}$**, and advantage clip **0.2**. Minecraft uses horizon **4**, rollout buffer size **1**, **120** train epochs per episode, actor learning rates **$2.5\times10^{-6}$** for StrBuild and **$5\times10^{-6}$** for HouseBuild, critic learning rates **$1.5\times10^{-6}$** and **$3\times10^{-6}$**, and advantage clip **0.05**. These settings underscore that the method is not purely conceptual; it is a concrete training recipe for episodic multi-LLM systems.

The method’s principal practical trade-off is that training is more expensive even when it is more sample-efficient. On CoopHE, MAGRPO used **9640** samples, **603** policy updates, **4.5 hours**, and **93.8 GB** VRAM, whereas CoLLM-CC used **8438** samples, **2110** updates, **11.1 hours**, and **107.4 GB** VRAM [2601.21972]. The article’s own recommendation is correspondingly selective: CoLLM-CC is most attractive when horizon is **$H \ge 2$–4**, rewards are sparse or highly non-linear, and coordination quality matters enough to justify centralized training overhead. Monte Carlo methods remain acceptable when the task is very short-horizon and dense-reward, while CoLLM-DC is presented as a lower-cost alternative that is generally less reliable under partial observability and policy non-stationarity.

Two misconceptions are directly addressed by the results. First, CoLLM-CC does **not** imply centralized inference; the critic is discarded after training, and execution remains decentralized. Second, CoLLM-CC is **not** uniformly dominant across all regimes; the writing tasks show that short-horizon dense-reward settings can narrow the gap between centralized critics, decentralized critics, and Monte Carlo baselines. The main limitations identified are the scalability cost of a critic conditioned on full joint histories, the dependence on global information such as turn index or progress, the modest experimental scale in agent count and horizon, and the absence of inference-time communication between agents. Future directions named in the paper include better critic architectures for larger teams, more sophisticated RL algorithms, richer domains, and cheaper critic parameterizations [2601.21972].

Source: https://www.emergentmind.com/topics/collm-cc