CoLLM-DC: Decentralized Collaboration for LLMs

• CoLLM-DC is a decentralized multi-agent actor-critic method that uses local critics and temporal-difference learning for independent LLM collaboration.
• It enables scalable, privacy-friendly deployment by allowing agents to operate concurrently with local histories, ideal for short-horizon, dense-reward scenarios.
• Empirical results indicate competitive performance in writing tasks, though the method underperforms in long-horizon or sparse-reward domains compared to centralized approaches.

CoLLM-DC is a decentralized-critic multi-agent actor-critic method for decentralized LLM collaboration, introduced alongside CoLLM-CC in “Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic” (Liu et al., 29 Jan 2026). It targets settings in which multiple LLM agents act concurrently, maintain only local histories, and must be trainable and deployable without centralized execution. In CoLLM-DC, each agent has its own actor $\pi_{\theta_i}(\cdot \mid h_{i,t})$ and its own critic $V_{\phi_i}(h_{i,t})$, so policy improvement is driven by local temporal-difference estimates rather than a centralized value function. The method is positioned as the fully decentralized actor-critic analog of collaborative LLM fine-tuning: it preserves decentralized execution and flexible deployment, but its empirical behavior is strongly task-dependent, being competitive mainly in short-horizon, dense-reward settings and substantially weaker than CoLLM-CC in long-horizon or sparse-reward regimes (Liu et al., 29 Jan 2026).

1. Origins, motivation, and decentralization premise

CoLLM-DC arises from a specific limitation of Monte Carlo fine-tuning for decentralized multi-LLM collaboration. In MA-REINFORCE-style training, useful learning signal arrives only after rollout termination, so credit assignment is delayed and noisy. The paper emphasizes that this becomes problematic when the horizon $H$ is large or rewards are sparse, because sample efficiency deteriorates and training becomes impractical. CoLLM-DC is proposed as a variance-reduction alternative within a Multi-Agent Actor-Critic framework, with the central design choice that critics remain decentralized rather than centralized (Liu et al., 29 Jan 2026).

The paper also treats decentralization as a practical systems requirement rather than only a modeling preference. In the decentralized setting, inference can be parallelized across agents, deployment is flexible because agents can run independently on separate nodes, memory and storage pressure are reduced because no shared centralized dialog state is needed at execution, and the overall design is described as more scalable and privacy-friendly than centralized communication-heavy systems. CoLLM-DC is intended to preserve these benefits during both training and execution.

A recurrent misconception is that a fully decentralized critic should be preferable whenever decentralized execution is desired. The paper argues otherwise. Although CoLLM-DC better matches decentralized deployment, its local critics are exposed to a more non-stationary learning problem and can be harder to fit, especially when collaborative behavior unfolds over many turns or reward signals are infrequent. This tension between deployment symmetry and training stability is one of the method’s defining characteristics (Liu et al., 29 Jan 2026).

2. Formal setting: decentralized LLM collaboration as a Dec-POMDP

The paper models decentralized LLM collaboration as a subclass of a Dec-POMDP,

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$

Here, $\mathcal{I}$ is the set of $n$ LLM agents, $\mathcal{V}$ is the token vocabulary, $C$ is the input context window size, $M$ is the max output length, $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$ is the global state split into accessible system state and inaccessible user state, $V_{\phi_i}(h_{i,t})$0 is the local observation space of agent $V_{\phi_i}(h_{i,t})$1, $V_{\phi_i}(h_{i,t})$2 is its action space, $V_{\phi_i}(h_{i,t})$3 is the joint reward, $V_{\phi_i}(h_{i,t})$4 is the transition function, and $V_{\phi_i}(h_{i,t})$5 is the episode horizon (Liu et al., 29 Jan 2026).

Each agent maintains a local interaction history

$V_{\phi_i}(h_{i,t})$6

while the joint history is

$V_{\phi_i}(h_{i,t})$7

This distinction is central. CoLLM-DC conditions each critic only on $V_{\phi_i}(h_{i,t})$8, not on $V_{\phi_i}(h_{i,t})$9. All agents nevertheless share the same joint reward. The paper does not introduce a special reward decomposition beyond this shared reward structure, so the critic-type distinction is purely informational: local history for CoLLM-DC, joint history for CoLLM-CC.

The execution protocol is strictly decentralized. At each step, each agent samples a response from its policy, the environment returns the next local observations and a global reward, and each history is updated locally. The framework assumes concurrent execution and no communication among agents during execution. This suggests that CoLLM-DC is most naturally interpreted as a decentralized control method under partial observability rather than as a communication-learning architecture (Liu et al., 29 Jan 2026).

3. Algorithmic structure and update equations

CoLLM-DC uses one actor and one critic per agent: $H$0 During rollout, each agent generates an action

$H$1

the joint action is $H$2, and the environment returns $H$3, next prompts $H$4, and global information $H$5. Histories are then updated as

$H$6

Transitions are stored in a replay buffer $H$7 with tuples of the form

$H$8

This storage design is notable because CoLLM-DC remains decentralized in critic conditioning even though the buffer retains joint information collected during training (Liu et al., 29 Jan 2026).

The policy gradient for agent $H$9 is

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$0

with one-step TD signal

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$1

The importance ratio is inherited from the paper’s off-policy formulation,

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$2

Each critic is trained independently by TD regression: $$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$3 For a minibatch $$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$4, each agent computes critic loss and policy gradient separately, then updates

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$5

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$6

An appendix variant uses parameter sharing, where agents and critics may learn the same latent representation $$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$7. The paper notes that this may allow a single shared model, but also warns that actor and critic objectives differ, so shared representations may suffer from gradient interference. This suggests that decentralization in CoLLM-DC is not only architectural but also optimization-theoretic: local critics are separated partly to avoid confounding distinct learning signals (Liu et al., 29 Jan 2026).

4. Relation to Monte Carlo fine-tuning and CoLLM-CC

The paper compares CoLLM-DC against two reference points: MA-REINFORCE-style Monte Carlo fine-tuning and CoLLM-CC, the centralized-critic variant. For the Monte Carlo baseline, the policy gradient is written as

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$8

with return

$$\langle \mathcal{I}, \mathcal{V}, C, M, \mathcal{S}, \{\mathcal{O}_i\}, \{\mathcal{A}_i\}, R, T, H \rangle.$$9

The paper provides three analytical observations for this Monte Carlo regime: unbiasedness of $\mathcal{I}$0-sampling gradients, variance reduction

$\mathcal{I}$1

and sample cost

$\mathcal{I}$2

These formulas are used to explain why Monte Carlo methods become increasingly unattractive as horizon length grows (Liu et al., 29 Jan 2026).

CoLLM-CC replaces local critics by one centralized critic $\mathcal{I}$3 and uses

$\mathcal{I}$4

The paper’s central contrast is informational. CoLLM-DC trains $\mathcal{I}$5 critics, each conditioned only on local history, whereas CoLLM-CC trains one critic on joint history and may also use extra global information $\mathcal{I}$6 during training. Both methods remain decentralized at execution, but CoLLM-CC receives lower-variance and more stationary learning signals because its critic conditions on more information (Liu et al., 29 Jan 2026).

A common simplification is to frame this comparison as decentralized versus centralized execution. The paper explicitly rejects that framing: both CoLLM-DC and CoLLM-CC are decentralized during execution. The decisive difference lies in training-time value estimation. This suggests that the CoLLM-DC versus CoLLM-CC distinction is best understood as a critic observability trade-off rather than as a deployment trade-off.

5. Empirical performance across writing, coding, and game-playing

The paper evaluates CoLLM-DC in writing, coding, and Minecraft game-playing domains, and the results are sharply domain-sensitive (Liu et al., 29 Jan 2026).

In writing collaboration, the tasks are TLDR summarization and arXiv expansion using two Qwen3-1.7B agents. Performance is measured by a weighted sum of structural quality, style consistency, and logical coherence. On TLDR, CoLLM-DC records time $\mathcal{I}$7 s, cost $\mathcal{I}$8, and score $\mathcal{I}$9, compared with CoLLM-CC at time $n$0 s, cost $n$1, and score $n$2, while MAGRPO scores $n$3. On arXiv expansion, CoLLM-DC records time $n$4 s, cost $n$5, and score $n$6, compared with CoLLM-CC at time $n$7 s, cost $n$8, and score $n$9, while MAGRPO scores $\mathcal{V}$0. The paper interprets these results as evidence that CoLLM-DC can be close to CoLLM-CC in short-horizon, dense-reward writing tasks.

In coding collaboration, the task is CoopHumanEval, a cooperative code generation benchmark requiring meaningful decomposition. Agents are Qwen2.5-Coder-3B and Qwen3-4B-Instruct-2507, with horizon $\mathcal{V}$1. CoLLM-DC reaches pass rate $\mathcal{V}$2, whereas CoLLM-CC reaches $\mathcal{V}$3, MAGRPO $\mathcal{V}$4, AC $\mathcal{V}$5, and GRPO $\mathcal{V}$6. The reported pass@$\mathcal{V}$7 values for CoLLM-DC are $\mathcal{V}$8, versus $\mathcal{V}$9 for CoLLM-CC. The paper attributes the CoLLM-DC deficit to non-convergence and instability.

In Minecraft, the tasks are StrBuild and HouseBuild using Qwen2.5-3B-Instruct and Qwen3-4B-Instruct-2507 with horizon $C$0. On StrBuild, CoLLM-DC records time $C$1, cost $C$2, adjacency $C$3, and IoU $C$4, whereas CoLLM-CC records time $C$5, cost $C$6, adjacency $C$7, and IoU $C$8. On HouseBuild, CoLLM-DC records time $C$9, cost $M$0, HP $M$1, and IoU $M$2, whereas CoLLM-CC records time $M$3, cost $M$4, HP $M$5, and IoU $M$6. The paper concludes that CoLLM-DC substantially underperforms in long-horizon Minecraft because non-stationarity accumulates across turns.

Training overhead on CoopHE reinforces the same conclusion. MAGRPO uses $M$7 samples, $M$8 updates, $M$9 h, and $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$0 GB VRAM; CoLLM-DC uses $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$1 samples, $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$2 updates, $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$3 h, and $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$4 GB VRAM; CoLLM-CC uses $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$5 samples, $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$6 updates, $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$7 h, and $$\mathcal{S} = \mathcal{S}^{\mathrm{sys}} \times \mathcal{S}^{\mathrm{usr}}$$8 GB VRAM. The paper therefore does not present CoLLM-DC as a cheaper default. Critics increase overhead relative to Monte Carlo methods, and CoLLM-CC is reported to be cheaper than CoLLM-DC because it maintains only one critic (Liu et al., 29 Jan 2026).

6. Limitations, recommended usage, and terminological disambiguation

The paper is explicit about CoLLM-DC’s limitations. It is less stable than CoLLM-CC, struggles in long-horizon and sparse-reward tasks, and uses local critics that are hard to train because the environment is non-stationary from each agent’s perspective. The experiments are described as proof-of-concept and do not scale to larger systems. The framework also assumes strict decentralization, with concurrent execution and no communication. In harder settings, the paper states that CoLLM-DC “fails to converge” (Liu et al., 29 Jan 2026).

Its recommended operating regime is correspondingly narrow. CoLLM-DC is most reasonable when fully decentralized training and inference structure is required, the task has short horizons, rewards are dense enough to stabilize TD learning, and a decentralization-preserving alternative to centralized-critic training is desired. When the task is long-horizon, sparse-reward, or difficult to credit assign, the paper recommends CoLLM-CC as the stronger choice. This suggests that CoLLM-DC is best viewed as a deployment-aligned baseline with conditional competitiveness rather than as the paper’s dominant method.

The name also benefits from disambiguation. “CoLLM” appears in unrelated literatures as a framework for composed image retrieval (Huynh et al., 25 Mar 2025), as a recommendation method that integrates collaborative embeddings into LLMs (Zhang et al., 2023), and as an AI engineering toolbox for collider analyses (Esmail et al., 6 Feb 2026). CoLLM-DC specifically denotes the decentralized-critic method in decentralized LLM collaboration and should not be conflated with those other uses of the label.

In summary, CoLLM-DC is a decentralized-critic MAAC formulation in which each LLM agent learns from a local-history value estimator. Its conceptual importance lies in showing that actor-critic methods can be made compatible with decentralized LLM collaboration without sacrificing decentralized execution. Its empirical significance is more qualified: the method is competitive in short-horizon, dense-reward collaboration, but its local critics become a liability in long-horizon, sparse-reward, and highly interactive tasks, where centralized-critic training remains more stable and more sample-efficient (Liu et al., 29 Jan 2026).