---
title: 'HACPO: Collaborative Reinforcement Learning'
url: https://www.emergentmind.com/topics/hacpo
type: topic
---

# HACPO: Collaborative Reinforcement Learning

Searching arXiv for "HACPO" and the cited papers to disambiguate the topic and support the article with current references.
HACPO is an acronym that denotes two distinct research objects in recent arXiv literature. In reinforcement learning, HACPO refers to **Heterogeneous Agent Collaborative Policy Optimization**, the first algorithm proposed within the **Heterogeneous Agent Collaborative Reinforcement Learning (HACRL)** paradigm introduced in "Heterogeneous Agent Collaborative Reinforcement Learning" [2603.02604]. In ranking and preference modeling, HCPO is the **Hierarchical Clustering Partial-Order** model from "Hierarchical Partial-Order Models for Ranking" [2606.25062], and the detailed specification notes that it is "sometimes called 'HACPO' when treated as a clustering model" [2606.25062]. The dominant use of HACPO in the supplied material is the reinforcement-learning sense, but the acronym is therefore polysemous and requires contextual disambiguation.

## 1. Reinforcement-learning meaning of HACPO

Within HACRL, HACPO is defined as **Heterogeneous Agent Collaborative Policy Optimization** [2603.02604]. HACRL addresses a setting in which multiple Large-Language-Model agents, "possibly differing in architecture, size, or training state," are trained toward a common, verifiable reward function, share rollouts during training, and are "deployed independently" at inference time [2603.02604]. The paper characterizes this arrangement as **"collaborative optimization with independent execution"** [2603.02604].

This positioning is important because HACPO is explicitly distinguished from two neighboring traditions. First, unlike LLM-based multi-agent reinforcement learning, it "does not require coordinated deployment" [2603.02604]. Second, unlike knowledge distillation and related on-/off-policy distillation approaches, it enables "bidirectional mutual learning among heterogeneous agents rather than one-directional teacher-to-student transfer" [2603.02604]. The algorithm therefore occupies a specific niche: collaborative on-policy optimization across heterogeneous agents, with training-time sharing but inference-time independence.

The paper states that HACPO is "the first algorithm to instantiate HACRL" [2603.02604]. In the provided description, each agent updates from both its own on-policy data ("homo") and other agents' rollouts ("hete"), with the stated aim of improving "sample efficiency and cross-agent knowledge transfer" [2603.02604]. This suggests that HACPO is best understood not merely as a modified PPO-style objective, but as a formalization of cross-policy sample reuse under heterogeneous tokenizers, parameterizations, and capabilities.

## 2. Algorithmic structure and training workflow

The HACPO training loop is given in explicit stepwise form for \(n\) agents [2603.02604]. Its inputs are agents \(\pi^{(k)}_{\theta_k}\), a shared prompt distribution \(D\), a verifiable reward function \(R(y)\), rollouts per agent per prompt \(G\), clipping hyperparameters \(\epsilon_{\text{low}}, \epsilon_{\text{high}}, \delta, \delta_{\text{step}}\), and an importance-sampling exponent \(\alpha\) [2603.02604]. At each training step, old policies are saved, a minibatch of prompts is sampled, each agent generates \(G\) responses per prompt, rewards are computed, and a shared reward pool \(\mathcal{R}_t\) is formed [2603.02604].

The algorithm then computes rolling capability estimates. For agent \(k\), these are
\[
P_\tau^{(k)} = \frac{1}{G}\sum_i R(y_{\tau,i}^{(k)}),
\qquad
\hat P_t^{(k)} = \frac{1}{K}\sum_{\tau=t-K+1}^{t} P_\tau^{(k)},
\]
which are used to define the **capability ratio**
\[
\omega_t^{(k,j)} = \frac{\hat P_t^{(k)}}{\hat P_t^{(j)}}.
\]
A capability-aware baseline is then formed:
\[
\mu_t^{(k)}=
\frac{1}{nG}\sum_{j=1}^{n}\sum_{i=1}^{G}\omega_t^{(k,j)}R(y_{b,i}^{(j)}),
\]
and the corresponding advantage uses the joint reward standard deviation,
\[
A_{b,i}^{(k)}=\frac{R(y_{b,i}^{(k)})-\mu_t^{(k)}}{\sigma_{t,\text{joint}}},
\qquad
\sigma_{t,\text{joint}}=\mathrm{std}(\{\mathcal{R}_t\}).
\]
These definitions are given directly in the algorithm specification [2603.02604].

For cross-agent updates, the advantage is further rescaled. When the sample comes from another agent \(j\neq k\),
\[
\tilde A_{b,i}^{(k)} = \omega_t^{(j,k)}A_{b,i}^{(j)},
\]
whereas on an agent’s own samples it remains \(A_{b,i}^{(k)}\) [2603.02604]. HACPO also computes a sequence-level importance-sampling ratio
\[
s_{b,i}^{(k,j)}=
\left(
\frac{\pi_{\theta_k}^{(k)}(y_{b,i}^{(j)})}
{\pi_{\theta_{j,\mathrm{old}}}^{(j)}(y_{b,i}^{(j)})}
\right)^{1/|y|},
\]
followed by exponential reweighting and asymmetric clipping for cross-agent samples [2603.02604]. The final surrogate loss for agent \(k\) is
\[
L^{(k)}=
-\frac{1}{G}\sum_{j=1}^{n}\sum_{i=1}^{G}
\mathrm{clipped}(s_{b,i}^{(k,j)})
\cdot
\mathrm{sg}(s_{b,i}^{(k,j)})^\alpha
\cdot
\omega_t^{(j,k)}
\cdot
\tilde A_{b,i}^{(k)},
\]
and parameters are updated by gradient descent [2603.02604].

A notable structural feature is that the algorithm is written to accommodate agents with "its own tokenizer & parameterization" [2603.02604]. This is one of the clearest indicators that HACPO is not limited to lightly perturbed replicas of a single base model.

## 3. The four tailored mechanisms

The HACPO paper organizes its methodological contribution around four named mechanisms [2603.02604]. These mechanisms are introduced to "mitigate capability discrepancies and policy distribution shifts" while preserving "unbiased advantage estimation and optimization correctness" [2603.02604].

| Mechanism | Definition in the paper | Intended role |
|---|---|---|
| Agent-Capability-Aware Advantage Estimation | Capability-weighted baseline \(\mu_t^{(k)}\) and advantage \(A_{t,i}^{(k)}\) using \(\sigma_{t,\text{joint}}\) | Adjusts shared reward normalization across unequal agents |
| Model Capability Discrepancy Coefficient | Cross-agent scaling by \(\omega_t^{(j,k)}\) when \(j\neq k\) | Reweights transferred signal by relative capability |
| Exponential Importance Sampling | Reweight \(s_{t,i}^{(k,j)}\) by \([ \mathrm{sg}(s_{t,i}^{(k,j)}) ]^\alpha\) for \(j\neq k\) and \(s<1\) | Alters cross-agent ratio behavior under mismatch |
| Stepwise Clipping | Asymmetric clipping \(s\leftarrow \mathrm{clamp}(s,1-\delta,1.0)\) with lower bound \(1-\delta+u\delta_{\text{step}}\) | Tightens trust region within the step |

The **Agent-Capability-Aware Advantage Estimation** replaces standard single-model group-relative normalization with a baseline mixed across agents but weighted by capability ratio [2603.02604]. The detailed description explicitly contrasts this with "Standard single-model group-relative advantage" and then defines the mixed baseline and joint-standard-deviation normalization [2603.02604].

The **Model Capability Discrepancy Coefficient** applies only in the heterogeneous case \(j\neq k\), multiplying the transferred advantage by \(\omega_t^{(j,k)}\) [2603.02604]. This indicates that transferred gradients are not treated as equally informative across agents of unequal performance.

The **Exponential Importance Sampling** mechanism defines the sequence-level ratio \(s_{t,i}^{(k,j)}\) and then reweights it for cross-agent samples when \(s<1\) through an exponent \(\alpha\ge 0\) [2603.02604]. The paper presents this as a specific correction for off-policy mismatch between the receiving agent’s current policy and the source agent’s old policy.

The **Stepwise Clipping** mechanism uses asymmetric clipping for \(j\neq k\), with the ratio clamped to \([1-\delta,1.0]\), and then tightens the lower bound after each mini-batch update \(u\) using
\[
\mathrm{lower}_u = 1-\delta + u\delta_{\text{step}},
\qquad
\mathrm{upper}=1.0
\]
[2603.02604]. A plausible implication is that HACPO treats negative policy drift on foreign samples more conservatively than symmetric PPO-style trust-region heuristics would.

## 4. Theoretical guarantees

The paper states two classes of theoretical results: unbiased advantage estimation and gradient consistency [2603.02604].

For **unbiased advantage estimation**, Theorem 1 is stated as follows: under "standard assumptions (reward model shared, \(\omega\) treated independent of current-batch reward noise)," the capability-aware baseline satisfies
\[
E[\mu_t^{(k)}] = E_{y\sim \pi_{\theta_k}}[R(y)].
\]
Corollary 1 then states
\[
E[A_{t,i}^{(k)}]=0.
\]
The proof sketch in the description expands \(E[\mu_t^{(k)}]\) over all agents’ samples and uses an "Assumption (ideal \(\omega\))" so that the capability ratio can be factored out and shown to cancel to the single-agent expected reward [2603.02604].

For **gradient consistency**, Theorem 2 defines \(\nabla J_{\text{homo}}\) and \(\nabla J_{\text{hete}}\) as the gradients of the homogeneous and heterogeneous objectives and states that
\[
\langle \nabla J_{\text{hete}}, \nabla J_{\text{homo}} \rangle > 0.
\]
The accompanying proof sketch rewrites the heterogeneous expectation using an importance-sampling lemma and introduces a positive weight term \(C(y)\) [2603.02604]. The statement then depends on a "mild assumption that \(C(y)\) (which encodes capability ratio and IS¹) is positively correlated with \(g\)’s alignment to the homo-direction" [2603.02604].

These results are narrower than a full global convergence theory. The paper claims **"unbiased advantage, positive gradient alignment"** as strengths [2603.02604], but the assumptions built into Theorem 1 and Theorem 2 delimit the scope of those guarantees. This suggests that HACPO’s theory is aimed at validating the correctness of its collaborative surrogate construction rather than establishing end-to-end optimality.

## 5. Experimental design and empirical results

The experiments in [2603.02604] use **7.5K high-quality MATH problems** for training, with each prompt yielding \(G=8\) rollouts, and employ **verifiable unit tests or formal checks per math problem** as the reward [2603.02604]. Evaluation is performed on **MATH-500, full MATH, GSM8K, AIME2025, AMC23, Minerva, OlympiadBench**, with **accuracy = fraction of correctly verified solutions**; **AIME2025** uses **best@30**, while the others use **avg@1** [2603.02604].

The paper studies three heterogeneity settings: **Heterogeneous State** with **Qwen3-4B vs Qwen3-4B-Instruct**; **Heterogeneous Size** with **Qwen3-1.7B-Base vs Qwen3-4B-Base**; and **Heterogeneous Model** with **Qwen3-4B-Base vs Llama3.2-3B-Instruct** [2603.02604]. Baselines are **GRPO**, **GSPO**, **GSPO×2**, and **Naive**, where Naive is "two-agent rollout share without HACPO’s four mechanisms" [2603.02604].

The summary results reported in the paper are as follows.

| Heterogeneity | Method | AVG(A,B) |
|---|---|---:|
| State (4B vs 4B-Instr) | GSPO | 0.742 |
| State (4B vs 4B-Instr) | GSPO×2 | 0.745 |
| State (4B vs 4B-Instr) | Naive | 0.637 |
| State (4B vs 4B-Instr) | HACPO | 0.784 |
| Size (1.7B vs 4B-Base) | GSPO | 0.523 |
| Size (1.7B vs 4B-Base) | GSPO×2 | 0.525 |
| Size (1.7B vs 4B-Base) | Naive | 0.476 |
| Size (1.7B vs 4B-Base) | HACPO | 0.547 |
| Model (Qwen3-4B vs Llama3.2-3B) | GSPO | 0.465 |
| Model (Qwen3-4B vs Llama3.2-3B) | GSPO×2 | 0.455 |
| Model (Qwen3-4B vs Llama3.2-3B) | Naive | 0.410 |
| Model (Qwen3-4B vs Llama3.2-3B) | HACPO | 0.494 |

The paper states that HACPO **"consistently improves all participating agents"** and **"outperform[s] GSPO by an average of 3.3\% while using only half the rollout cost"** [2603.02604]. In the tabulated summary, the **overall average gain** is given as **+3.3 points over GSPO** [2603.02604]. It further states that HACPO uses \(G\) rollouts per agent, for total \(nG\), but each agent reuses all \(nG\) samples, making it "equivalent to GSPO×2’s data budget but at half physical sampling cost" [2603.02604].

An important negative result is that the **Naive** rollout-sharing baseline performs substantially worse in every reported heterogeneity setting [2603.02604]. This supports the paper’s claim that raw rollout pooling is insufficient and that the four mechanisms are not incidental engineering details.

## 6. Strengths, limitations, and plausible extensions

The strengths section in [2603.02604] lists four properties. **Sample Efficiency** is attributed to the fact that "each rollout is reused across agents (up to \(n\) times)." **Bidirectional Transfer** is motivated by the claim that "even weaker agents contribute unique signals (errors, alternative proofs)." **Theoretically Sound** refers to "unbiased advantage, positive gradient alignment." **Heterogeneity-Robust** refers to operation across "state, size, and model heterogeneity" [2603.02604].

The limitations section is equally explicit. **Hyperparameter Sensitivity** is attached to \(\alpha\), \(\delta\), and \(\delta_{\text{step}}\), which "must be tuned per agent pair" [2603.02604]. **IS Variance** is identified as a constraint under large policy gaps, which "may force very conservative \(\alpha\to 0\) or heavy clipping" [2603.02604]. **Scalability to Many Agents** is described as a concern because controlling pairwise \(\omega\) and importance sampling "might become complex as \(n\) grows" [2603.02604].

The paper also enumerates possible extensions: **Adaptive \(\alpha\) and \(\delta\) per agent pair, based on online drift estimates**; **Hierarchical Collaboration: factor graphs to select most informative cross-agent rollouts**; **Single-Agent Distillation: use HACPO to derive a single "ensemble" policy after collaborative training**; and application to **standard continuous-control MARL**, replacing verifiable rewards with learned critics while retaining heterogeneous trajectory sharing [2603.02604].

These proposals remain prospective in the source. A plausible implication is that HACPO is presently most mature in settings with verifiable rewards and moderate numbers of heterogeneous agents, rather than in large-agent systems with highly nonstationary reward estimation.

## 7. Alternative meaning: HCPO/HACPO in hierarchical partial-order ranking

A second, unrelated use of the acronym appears in "Hierarchical Partial-Order Models for Ranking" [2606.25062]. That paper introduces **hierarchical partial order (HPO) models** and a clustering extension called **HCPO**, with the note that HCPO is "sometimes called 'HACPO' when treated as a clustering model" [2606.25062]. This usage belongs to Bayesian ranking and preference aggregation rather than reinforcement learning.

In that framework, the universe of items is \(\mathcal{M}=\{1,\dots,M\}\), a partial order \(h\) is a binary relation satisfying irreflexivity, antisymmetry, and transitivity, and \(\mathrm{LE}(h)\) denotes the set of linear extensions of \(h\) [2606.25062]. HCPO constructs a hierarchy consisting of a global poset \(h^{(0)}\) and cluster-specific posets \(h^{(g)}\), coupled through latent utilities \(U^{(g)}_{j,:}\in \mathbb{R}^K\) and shrinkage governed by \(\tau\) [2606.25062]. The model includes a noise-free poset-extension likelihood
\[
p_{S_i}(Y_i\mid h^{(g)})=
\begin{cases}
\frac{1}{|\mathrm{LE}(h^{(g)}[S_i])|}, & Y_i\in \mathrm{LE}(h^{(g)}[S_i])\\
0, & \text{otherwise,}
\end{cases}
\]
as well as faster alternatives such as **weighted queue-jump** and **frontier-softmax** [2606.25062].

Inference is performed by MCMC over \((U,c,\rho,\tau,\beta)\)-space, using Gibbs steps for root utilities, truncated-Gaussian or MH updates for leaf utilities, MH updates for \(\rho\) and \(\tau\), and Gibbs/CRP updates for cluster labels under a Pitman–Yor or Dirichlet-process-style prior [2606.25062]. The paper states that experiments on synthetic and real-world data, "including pairwise acoustic preference data and LLM agent traces," show that HPO and HCPO "outperform existing approaches in both predictive performance and structural interpretability" [2606.25062].

Because this ranking-model usage is methodologically independent from Heterogeneous Agent Collaborative Policy Optimization, the acronym **HACPO** should not be interpreted without domain context. In contemporary arXiv usage, HACPO most directly denotes the collaborative reinforcement-learning algorithm of [2603.02604], while [2606.25062] documents a separate naming collision arising from HCPO in hierarchical partial-order modeling.

Source: https://www.emergentmind.com/topics/hacpo