CoLLM-CC: Centralized Critic for LLM Collaboration

• CoLLM-CC is a multi-agent actor-critic method for decentralized LLM collaboration that employs a centralized critic during training to reduce variance.
• It treats full textual responses as macro-actions and uses teacher-forcing for efficient sequence-level policy gradient computation.
• Empirical results in writing, coding, and gaming tasks demonstrate enhanced sample efficiency and stability compared to alternative approaches.

Searching arXiv for the exact CoLLM-CC paper and related CoLLM disambiguation. Searching arXiv for related work on decentralized LLM collaboration and actor-critic formulations. CoLLM-CC is a multi-agent actor-critic method for decentralized LLM collaboration. It was introduced as the centralized-critic member of a pair of Multi-Agent Actor-Critic methods, alongside CoLLM-DC, for settings in which multiple LLM agents act in parallel, each with its own prompt and local history, while optimizing a shared episodic reward. Its defining property is centralized training with decentralized execution: during training, a single critic conditions on joint histories and optional global information, but at inference time the critic is discarded and each agent runs independently from its own local context (Liu et al., 29 Jan 2026).

1. Terminology, scope, and conceptual position

In the nomenclature of the method, “CC” denotes Centralized Critic. CoLLM-CC is therefore not a general label for collaborative LLM systems, but a specific MARL algorithm for decentralized LLM collaboration in which the actor policies remain local and the value function is centralized during training. The method is motivated by two limitations in prior work: hard-coded collaboration protocols that require centralized execution, and Monte Carlo policy-gradient approaches that wait until the end of an episode to compute returns and therefore suffer from high variance and poor sample efficiency, especially in long-horizon or sparse-reward tasks (Liu et al., 29 Jan 2026).

The method is explicitly designed for cooperative settings in which agents do not share full state information at inference time. Each agent produces a full textual response, such as a paragraph, a code function, or a game command, based only on its own prompt and history. The collaborative structure is therefore learned in training rather than enforced by a centralized inference-time controller. This suggests a deployment model in which parallelism and flexible system placement are preserved even though optimization uses centralized information.

A recurrent source of confusion is the reuse of the name “CoLLM” in unrelated literature. For example, the composed image retrieval framework “CoLLM” does not define any entity named “CoLLM-CC”; its paper states that the string “CoLLM-CC” does not appear anywhere in that work (Huynh et al., 25 Mar 2025). In the MARL context, by contrast, CoLLM-CC is a precise algorithmic designation.

2. Decentralized collaboration as a Dec-POMDP

The method formalizes decentralized LLM collaboration as a Decentralized Partially Observable MDP with tuple

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$

Here, $\mathcal{I}$ is the set of LLM agents, $\mathcal{V}$ is the token vocabulary, $C$ is the context window size, $M$ is the maximum number of tokens an agent may output per turn, and $H$ is the episode horizon (Liu et al., 29 Jan 2026).

The state at turn $t$ is written as $s_t = (s_t^{\mathrm{sys}}, s_t^{\mathrm{usr}})$. The system-accessible component contains task state, environment state, or tool state, whereas the user-private component is not directly observable. Each agent receives a local observation $o_{i,t}$ in natural-language prompt form, representing a partial and noisy view of the underlying state. The action space is also textual: an agent action $a_{i,t}$ is a sequence of up to $\mathcal{I}$0 tokens. The joint reward is produced from the combined action and the system-accessible state, and transitions are governed by the environment dynamics.

Because agents are partially observed, the sufficient conditioning variable for each actor is its local history

$\mathcal{I}$1

while the critic may access the joint history

$\mathcal{I}$2

Each actor implements a policy $\mathcal{I}$3, and the training objective is to maximize expected cumulative reward over the episode. The formulation is notable for treating a full generated response as the action, rather than individual tokens, which aligns the reinforcement-learning abstraction with the granularity of collaborative behavior.

3. Centralized-critic, decentralized-actor architecture

CoLLM-CC uses one LLM actor per agent and a separate centralized critic. The actors are not parameter-shared in the reported experiments; instead, the system uses heterogeneous role-specific models, such as paired Qwen variants for summarization or coding roles. Each actor maintains its own KV cache and local dialogue history. At each turn, the previous KV cache is concatenated with new prompt tokens, and only the latest $\mathcal{I}$4 KV pairs are retained (Liu et al., 29 Jan 2026).

A core design choice is to treat the entire textual response as a macro-action. For agent $\mathcal{I}$5, the policy is

$\mathcal{I}$6

and teacher-forcing forward passes are used to compute sequence log-probabilities efficiently in a single pass. This avoids token-level RL credit assignment inside each response and keeps the MARL layer at the level of agent turns.

The centralized critic estimates a joint value from the joint history, optionally augmented with global information such as turn index or progress: $\mathcal{I}$7 It is trained with temporal-difference learning. The TD error is

$\mathcal{I}$8

and the critic loss is

$\mathcal{I}$9

Each actor is then updated using a centralized value-based advantage: $\mathcal{V}$0 with off-policy importance ratio

$\mathcal{V}$1

In implementation, the training loop uses a replay buffer and applies advantage clipping, with reported clipping values of $\mathcal{V}$2 in writing and coding and $\mathcal{V}$3 in Minecraft. The overall structure is therefore a textbook centralized-training, decentralized-execution design, but instantiated with full LLMs as both actors and critic.

4. Training loop and relation to Monte Carlo and decentralized critics

Training proceeds episodically. For a sampled task, the system initializes prompts for all agents, rolls out the episode by sampling one response from each actor per turn, records reward and next observations, stores the transition in a replay buffer, and then performs multiple gradient epochs over minibatches drawn from that buffer. For each stored action, teacher-forcing is used again during optimization to recompute the response probability under the current actor parameters, which yields stable sequence-level policy gradients (Liu et al., 29 Jan 2026).

The paper contrasts CoLLM-CC with two alternatives. The first is Monte Carlo multi-agent policy gradients, exemplified by MA-REINFORCE or MAGRPO, which estimate returns directly from rollout trees. Under the independence assumptions used in the paper’s analysis, the variance of the averaged estimator satisfies

$\mathcal{V}$4

while the number of model calls grows as

$\mathcal{V}$5

This is the central argument for why Monte Carlo methods become unattractive as horizon and branching factor increase.

The second alternative is CoLLM-DC, in which each agent has a decentralized critic $\mathcal{V}$6 depending only on local history. In principle, if these critics converge, the resulting gradients remain unbiased. In practice, the paper reports that local critics are highly exposed to non-stationarity because other agents’ policies are changing while each critic sees only a partial local view. The empirical claim is therefore conditional rather than absolute: Monte Carlo methods and CoLLM-DC are competitive in short-horizon, dense-reward settings, but they both underperform CoLLM-CC in long-horizon or sparse-reward regimes. This suggests that the benefit of centralization lies less in raw asymptotic expressivity than in variance reduction and critic convergence under partial observability.

5. Empirical profile across writing, coding, and game-playing

The reported experiments span three domains with increasing horizon and reward sparsity. In writing collaboration, the tasks are TLDR summarization and arXiv introduction expansion, both with horizon $\mathcal{V}$7. On TLDR, CoLLM-CC achieved time 1.8 s, cost 181 tokens, and score 95.2, while CoLLM-DC achieved 95.4 and MAGRPO achieved 93.5. On the arXiv task, CoLLM-CC achieved time 1.9 s, cost 188, and score 95.0, compared with 94.1 for CoLLM-DC and 93.1 for MAGRPO (Liu et al., 29 Jan 2026). These results support the paper’s more nuanced conclusion: in single-turn dense-reward writing, CoLLM-CC is strong and typically best among the MARL methods, but the gap to alternatives is modest.

The picture changes in coding collaboration on CoopHE, where horizon increases to $\mathcal{V}$8 and the reward depends heavily on structural integrity, syntax correctness, test pass rate, and cooperation quality. Here the key reported number is Pass@1. CoLLM-CC reached 75.2%, MAGRPO 74.3%, and CoLLM-DC 59.1%. The associated training curves show that CoLLM-CC converged around ~2000 timesteps, MAGRPO needed ~5000 timesteps to stabilize, and CoLLM-DC oscillated and degraded in later stages. The coding domain therefore provides the clearest evidence for the claim that a centralized critic improves sample efficiency and stability.

In Minecraft-style games, the advantage of CoLLM-CC is larger still. In StrBuild, with horizon $\mathcal{V}$9, CoLLM-CC achieved IoU 68.5 and same-texture adjacency 7.3, compared with IoU 50.6 and adjacency 13.3 for MAGRPO, and IoU 44.6 and adjacency 7.6 for CoLLM-DC. In HouseBuild, CoLLM-CC achieved IoU 52.7 and HP 86.4, compared with IoU 50.9 and HP 80.2 for MAGRPO, and IoU 46.8 and HP 43.8 for CoLLM-DC (Liu et al., 29 Jan 2026). The training curves reported for both Minecraft tasks indicate that CoLLM-CC steadily improves and converges, MAGRPO is slower and noisier, and CoLLM-DC fails to converge. A plausible implication is that critic centralization becomes increasingly valuable as delayed coordination structure dominates immediate local reward.

6. Implementation profile, operating regime, and limitations

The implementation uses full LLMs for both actors and critic rather than lightweight heads. In writing, the actors are Qwen3-1.7B models and the centralized critic is also Qwen3-1.7B. In coding, the actors are Qwen2.5-Coder-3B and Qwen3-4B-Instruct, with Qwen2.5-Coder-3B as critic. In Minecraft, the actors are Qwen2.5-3B-Instruct and Qwen3-4B-Instruct, and the critic is Qwen3-4B-Instruct. The reported experiments use full fine-tuning, not LoRA or other PEFT variants. Decoding temperatures are 0.7 for writing and 0.6 for coding and Minecraft, with maximum output lengths of 256 tokens for writing and coding and 256–512 tokens for Minecraft (Liu et al., 29 Jan 2026).

Training hyperparameters vary by domain. Writing uses horizon 1, rollout buffer size 4, 20 train epochs per episode, actor learning rate $C$0, critic learning rate $C$1, and advantage clip 0.2. Coding uses horizon 2, rollout buffer size 4, 80 train epochs per episode, actor learning rate $C$2, critic learning rate $C$3, and advantage clip 0.2. Minecraft uses horizon 4, rollout buffer size 1, 120 train epochs per episode, actor learning rates $C$4 for StrBuild and $C$5 for HouseBuild, critic learning rates $C$6 and $C$7, and advantage clip 0.05. These settings underscore that the method is not purely conceptual; it is a concrete training recipe for episodic multi-LLM systems.

The method’s principal practical trade-off is that training is more expensive even when it is more sample-efficient. On CoopHE, MAGRPO used 9640 samples, 603 policy updates, 4.5 hours, and 93.8 GB VRAM, whereas CoLLM-CC used 8438 samples, 2110 updates, 11.1 hours, and 107.4 GB VRAM (Liu et al., 29 Jan 2026). The article’s own recommendation is correspondingly selective: CoLLM-CC is most attractive when horizon is $C$8–4, rewards are sparse or highly non-linear, and coordination quality matters enough to justify centralized training overhead. Monte Carlo methods remain acceptable when the task is very short-horizon and dense-reward, while CoLLM-DC is presented as a lower-cost alternative that is generally less reliable under partial observability and policy non-stationarity.

Two misconceptions are directly addressed by the results. First, CoLLM-CC does not imply centralized inference; the critic is discarded after training, and execution remains decentralized. Second, CoLLM-CC is not uniformly dominant across all regimes; the writing tasks show that short-horizon dense-reward settings can narrow the gap between centralized critics, decentralized critics, and Monte Carlo baselines. The main limitations identified are the scalability cost of a critic conditioned on full joint histories, the dependence on global information such as turn index or progress, the modest experimental scale in agent count and horizon, and the absence of inference-time communication between agents. Future directions named in the paper include better critic architectures for larger teams, more sophisticated RL algorithms, richer domains, and cheaper critic parameterizations (Liu et al., 29 Jan 2026).