Heterogeneous Agent Collaborative Reinforcement Learning

Abstract: We introduce Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new learning paradigm that addresses the inefficiencies of isolated on-policy optimization. HACRL enables collaborative optimization with independent execution: heterogeneous agents share verified rollouts during training to mutually improve, while operating independently at inference time. Unlike LLM-based multi-agent reinforcement learning (MARL), HACRL does not require coordinated deployment, and unlike on-/off-policy distillation, it enables bidirectional mutual learning among heterogeneous agents rather than one-directional teacher-to-student transfer. Building on this paradigm, we propose HACPO, a collaborative RL algorithm that enables principled rollout sharing to maximize sample utilization and cross-agent knowledge transfer. To mitigate capability discrepancies and policy distribution shifts, HACPO introduces four tailored mechanisms with theoretical guarantees on unbiased advantage estimation and optimization correctness. Extensive experiments across diverse heterogeneous model combinations and reasoning benchmarks show that HACPO consistently improves all participating agents, outperforming GSPO by an average of 3.3\% while using only half the rollout cost.

• The paper introduces HACRL and its HACPO algorithm, enabling collaborative policy optimization among heterogeneous LLMs via shared rollouts.
• Experimental results demonstrate a 3.3% performance improvement over GSPO while halving the rollout budget, underscoring its sample efficiency.
• Four key innovations, including capability-aware advantage estimation and stepwise clipping, are detailed to enhance stability and control in cross-agent learning.

Heterogeneous Agent Collaborative Reinforcement Learning: A Technical Overview

Motivation and Paradigm Distinctions

The HACRL framework introduces a principled method for collaborative policy optimization among heterogeneous LLM agents under the RLVR paradigm. This is particularly motivated by the inefficiencies in isolated on-policy RL optimization for LLMs, where each agent independently samples and verifies high-cost trajectories but fails to leverage the collective value of these rollouts.

A defining characteristic of HACRL, as opposed to classical multi-agent RL (MARL) and knowledge distillation (KD), is its emphasis on independent execution with collaborative optimization: agents learn from shared, verifiable rollouts during training but execute independently at inference, sidestepping the need for coordinated deployment or one-way transfer (Figure 1).

Figure 1: Distinguishing MARL (coordinated execution), KD (uni-directional transfer), and HACRL (independent execution + collaborative optimization).

Formalization of Heterogeneity

HACRL’s collaborative optimization is designed for environments where agents differ in:

1. State: Same architecture and parameter space, differing only by optimization state;
2. Size: Same family but different parameter counts;
3. Model: Distinct architectures (e.g., tokenizers, training objectives).

This taxonomy is crucial since capability and policy distribution disparities among agents are nontrivial sources of bias and instability during collaborative learning. The HACRL objective is explicitly formalized to maximize each agent's performance by leveraging both self- and cross-agent rollouts, under a shared reward but with robust mechanisms to prevent deleterious distribution shifts.

The HACPO Algorithm

HACPO (Heterogeneous Agent Collaborative Policy Optimization) is proposed as a solution to collaborative RL among heterogeneous agents. Its core design is methodologically rigorous, with four algorithmic innovations to maximize safe and effective mutual learning:

1. Agent-Capability-Aware Advantage Estimation: Baseline advantage estimates are calibrated per agent using a capability ratio that reflects recent mean performance, enabling unbiased learning even from agents of unequal proficiency.
2. Model Capabilities Discrepancy Coefficient: Gradients are adaptively scaled to amplify updates from stronger agents and attenuate noise from weaker ones.
3. Exponential Importance Sampling: Sequence-level importance ratios are exponentially reweighted (with a tunable $\alpha$) to control the impact of cross-agent distributional mismatch, especially under severe heterogeneity.
4. Stepwise Clipping: Asymmetric and gradually tightening clipping bounds on cross-agent importance weights suppress high-variance updates and stabilize policy improvement across mini-batches.

   Figure 2: HACPO leverages cross-agent rollouts and introduces mechanisms for capacity-aware credit assignment and robust importance sampling for stable mutual learning.

These mechanisms are underpinned by theoretical analysis: the agent-capability-aware advantage estimator is proven to be unbiased, and the cross-agent optimization direction is guaranteed to be positively aligned with the underlying on-policy objective, provided mild competence alignment.

Experimental Analysis

HACPO is validated across a diverse set of heterogeneity scenarios (state, size, model) and on several mathematical reasoning benchmarks, utilizing the Qwen3 and Llama3.2 model families. Performance is compared to standard single-agent baselines (GRPO, GSPO), resource-matched controls, and naive rollout-sharing, isolating the gains attributable to HACPO’s architectural innovations.

Empirically, HACPO consistently yields significant performance improvements for all collaborating agents, outperforming GSPO by an average of 3.3% while consuming only half the rollout budget. Gains are evident across all heterogeneity settings, even when collaborating agents differ in pretraining corpus, architecture, and tokenizer.

Figure 3: Comparative learning curves of Qwen3-4B and Qwen3-4B-Instruct, demonstrating substantial, asymmetric but mutual improvement with HACPO.

Ablation studies confirm the contribution of each mechanism:

• Removing capability-aware advantage estimation or discrepancy coefficient reduces performance and destabilizes training.
• Larger importance sampling exponents enforce conservativeness but may underutilize cross-agent signal.
• Omitting stepwise clipping leads to instability and poor convergence, validating its crucial role in preventing harmful distributional drift.

Implications and Future Directions

HACRL, instantiated through HACPO, fundamentally reshapes the efficiency frontier in RL-driven LLM post-training. The collaborative approach unlocks substantial sample efficiency by allowing each trajectory to contribute to the optimization of all agents, avoiding wasteful duplication. Additionally, mutual and bidirectional knowledge transfer enables weak agents to contribute valuable exploration paths and strong agents to break their isolated performance ceilings, addressing bottlenecks known in both MARL and KD literature.

The practical implications extend to scalable, vendor-agnostic LLM training ecosystems, where heterogeneous commercial or research LLMs can be co-trained efficiently without requiring synchronized deployment or uniform architectures.

On the theoretical front, the positive alignment guarantees open further inquiry into optimal credit assignment and robust mutual learning under adversarial heterogeneity or non-stationary agent populations.

Conclusion

HACRL and the HACPO algorithm represent a formally sound and empirically validated departure from isolated RL optimization for LLMs. By enabling collaborative rollout sharing among diverse agents, HACPO establishes a new benchmark for sample-efficient, cross-agent knowledge transfer in heterogeneous LLM systems while ensuring inference-time independence. This work provides a foundation for further advancements in collaborative post-training, scalable policy optimization, and efficient knowledge-sharing across the rapidly diversifying landscape of LLMs.