Heterogeneous Reinforcement Learning (HeteroRL)

• HeteroRL is a reformulation of reinforcement learning that explicitly models varied agents, environments, and resource constraints rather than enforcing uniformity.
• It introduces innovative methods like asynchronous updates and sequential advantage decomposition to better handle non-stationarity and inter-agent coordination.
• The framework spans applications from cooperative MARL and federated RL to asynchronous LLM post-training, improving stability and performance in diverse systems.

Searching arXiv for papers on “HeteroRL” and closely related “heterogeneous reinforcement learning” to ground the article in cited work. HeteroRL denotes reinforcement-learning regimes in which heterogeneity is modeled explicitly rather than suppressed through shared policies, homogeneous network topologies, or uniform environment assumptions. In the narrowest current usage, “HeteroRL” names an asynchronous architecture for LLM post-training that decouples rollout sampling from parameter learning under geographically distributed network delay (Zhang et al., 25 Aug 2025). In the broader literature, the same label naturally extends to cooperative MARL without parameter sharing, federated RL with heterogeneous transition kernels, distributed RL over mixed-capacity devices, collaborative optimization among heterogeneous LLM agents, and neuro-symbolic or human-allied RL pipelines that reconcile structured and unstructured inputs (Zhong et al., 2023, Hwang et al., 19 Jul 2025, Rapp et al., 2020, Zhang et al., 3 Mar 2026, Darvishvand et al., 17 Oct 2025).

1. Conceptual scope and defining heterogeneity

HeteroRL is best understood as an umbrella for RL formulations in which one or more core assumptions of homogeneity are relaxed. The literature instantiates this relaxation along several axes: agents may differ in observation spaces, action spaces, sensors, resources, embodiment, or policy parameterization; local environments may differ in transition dynamics; devices may differ in compute, memory, bandwidth, and energy; and training pipelines may mix symbolic and subsymbolic inputs or combine machine rewards with human advice (Dansereau et al., 23 Sep 2025, Hwang et al., 19 Jul 2025, Rapp et al., 2020, Darvishvand et al., 17 Oct 2025).

A central motivation is that many standard MARL and distributed RL methods inherit homogeneous design assumptions. Cooperative MARL has often relied on parameter sharing, but HARL argues that this restricts the policy class to the homogeneous-agent setting and can lead to instability under simultaneous updates (Zhong et al., 2023). HeMAC sharpens the same point empirically: many existing methods are easiest to apply when agents have common observation and action interfaces, and padding heterogeneous spaces into a homogeneous format is treated as an unsatisfactory workaround because it enlarges dimensions, slows learning, and can distort action semantics (Dansereau et al., 23 Sep 2025).

The theoretical objection is not merely architectural. HARL provides an example in which the best shared policy is exponentially suboptimal, with

$\frac{J^*_{\mathrm{share}}}{J^*}=\frac{2}{2^n},$

which formalizes the loss induced by forcing interchangeable behavior onto non-interchangeable agents (Zhong et al., 2023). This suggests that HeteroRL is not a niche variant of MARL or federated RL, but a corrective perspective on settings where symmetry assumptions are structurally false.

2. Major problem formulations

Current work uses HeteroRL in several technically distinct senses. The formulations differ in what is heterogeneous, what is shared, and what the optimization target is.

Formulation	Defining heterogeneity	Representative work
Cooperative MARL	Agents differ in observations, actions, roles, or architectures	HARL/HAML (Zhong et al., 2023), HeMAC (Dansereau et al., 23 Sep 2025), HALyPO (Zhang et al., 4 Mar 2026)
Federated RL	Clients share $\mathcal S,\mathcal A,r,d_0$ but have different $P_k$	FedRQ (Hwang et al., 19 Jul 2025)
Distributed RL over devices	Learners differ in model capacity and resource profile	Partial shared-topology RL (Rapp et al., 2020)
Asynchronous RL for LLMs	Sampler and learner policies differ because of delay-induced staleness	HeteroRL/GEPO (Zhang et al., 25 Aug 2025)
Collaborative RL among heterogeneous LLMs	Agents differ in state, size, or model family, but share task distribution and verifier	HACRL/HACPO (Zhang et al., 3 Mar 2026)
Relational and human-allied RL	Inputs, state factors, and knowledge sources are heterogeneous	RAEL (Darvishvand et al., 17 Oct 2025)

The cooperative multi-agent line is usually formalized as a Dec-POMDP or cooperative Markov game. HeMAC uses

$$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$

with decentralized policies $\pi_i:\tau_i\rightarrow\Delta(A_i)$, and then makes heterogeneity explicit through different fields of view, dynamics, energy constraints, communication affordances, and role specialization (Dansereau et al., 23 Sep 2025). HARL adopts a cooperative Markov game and factorized joint policy $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$, but removes any requirement that agents share observation spaces, action spaces, or network architectures (Zhong et al., 2023).

Federated HeteroRL takes a different form. FedRQ defines $K$ client MDPs

$$\mathcal M_k=\langle \mathcal S,\mathcal A,P_k,r,\gamma\rangle,$$

so the task is common but the transition kernel is client-specific. The global objective is not average return but robust performance over a covering set of plausible dynamics,

$$\max_\pi \inf_{P\in\mathcal P_\omega} V(\pi\mid P),$$

which turns environment heterogeneity into structured uncertainty over transition kernels (Hwang et al., 19 Jul 2025).

Systems-oriented HeteroRL makes the hardware and network stack part of the problem definition. Distributed learning on heterogeneous devices partitions parameters as

$$\theta=(\theta_{\text{local}},\theta_{\text{shared}}),$$

allowing only overlapping layers to synchronize across devices with different-capacity networks (Rapp et al., 2020). HetRL for LLM post-training elevates this systems view into a scheduling problem over an RL workflow graph and a device topology graph, with partitioning strategy $\mathcal S,\mathcal A,r,d_0$0, assignment strategy $\mathcal S,\mathcal A,r,d_0$1, and objective

$\mathcal S,\mathcal A,r,d_0$2

subject to memory and placement constraints (He et al., 13 Dec 2025).

3. Algorithmic mechanisms for cooperative heterogeneous agents

A defining algorithmic pattern in HeteroRL is to replace simultaneous symmetric updates with structured, asymmetry-aware optimization. HARL does this through the multi-agent advantage decomposition lemma,

$\mathcal S,\mathcal A,r,d_0$3

which decomposes a subset advantage into a sequence of conditional single-agent advantages (Zhong et al., 2023). This lemma motivates the sequential update scheme underlying HATRL, HATRPO, and HAPPO: agents are updated one by one, and each later agent optimizes against already-updated earlier agents rather than stale teammates. Within HAML, this becomes a general mirror-learning template, and the resulting algorithms inherit monotonic improvement of joint return and convergence to Nash equilibrium (Zhong et al., 2023).

HALyPO addresses a different failure mode in heterogeneous-agent learning: the rationality gap between decentralized best-response-like updates and centralized cooperative ascent. It defines the disagreement metric

$\mathcal S,\mathcal A,r,d_0$4

treats heterogeneous HRC as a general-sum differentiable game, and rectifies decentralized gradients by solving a quadratic projection that enforces a per-step Lyapunov decrease condition (Zhang et al., 4 Mar 2026). The practical update

$\mathcal S,\mathcal A,r,d_0$5

keeps the update close to the decentralized gradient while removing the component that would increase disagreement (Zhang et al., 4 Mar 2026). This is a notably different use of Lyapunov methods from safe RL in CMDPs: certification is imposed in policy-parameter space, not state space.

CHDRL approaches heterogeneity through algorithmic diversity rather than agent embodiment. It partitions learners into global off-policy agents and local on-policy or evolutionary agents, then couples them with Cooperative Exploration, Local-Global Memory Relay, and Distinctive Update (Zheng et al., 2020). In the CSPC instantiation, SAC acts as the global agent, PPO as an on-policy local agent, and CEM as a local evolutionary agent. Transfer occurs only when score differences exceed a threshold $\mathcal S,\mathcal A,r,d_0$6, and the global replay distribution is explicitly mixed between global and local memories, which preserves sample efficiency without collapsing the heterogeneous update rules into a single optimizer (Zheng et al., 2020).

HACRL extends this asymmetry-aware logic to heterogeneous LLM agents. Its objective for agent $\mathcal S,\mathcal A,r,d_0$7,

$\mathcal S,\mathcal A,r,d_0$8

combines self-generated and cross-agent verified rollouts (Zhang et al., 3 Mar 2026). HACPO then adds four mechanisms: agent-capability-aware advantage estimation, a model capabilities discrepancy coefficient, exponential importance sampling, and stepwise clipping. Cross-agent reuse is based on sequence-level ratios

$\mathcal S,\mathcal A,r,d_0$9

which are clipped to $P_k$0 for $P_k$1, so foreign rollouts can downweight but not upweight relative to self-rollouts (Zhang et al., 3 Mar 2026). The theoretical claims are conditional—unbiasedness requires an ideal capability-ratio assumption, and gradient alignment requires positive competence alignment—but they formalize why naive rollout pooling is unsafe under large capability and distribution gaps (Zhang et al., 3 Mar 2026).

4. Federated, distributed, and resource-aware HeteroRL

One branch of HeteroRL is primarily about non-IID control problems across clients. FedRQ defines Federated Reinforcement Learning with Environment Heterogeneity by keeping the state space, action space, reward function, discount factor, and initial distribution fixed while allowing each client to have its own transition kernel $P_k$2 (Hwang et al., 19 Jul 2025). The method replaces ordinary local Q-learning with a robust update containing a pessimistic next-state term,

$P_k$3

and averages local Q-functions every $P_k$4 steps (Hwang et al., 19 Jul 2025). Under the consistency-of-non-zero-transitions assumption, the synchronized global iterate converges to the optimal robust Q-function $P_k$5 at rate $P_k$6 in sup norm, with bound

$P_k$7

This is one of the clearest theoretical results in the heterogeneous-environment literature (Hwang et al., 19 Jul 2025).

A second branch is resource-centric. Distributed learning on heterogeneous resource-constrained devices allows weak devices to run a lightweight network, powerful devices to run a complex network, and all devices to share only the overlapping initial layers (Rapp et al., 2020). The synchronized object is therefore not the entire model but only $P_k$8. In the RL experiments, both models share the first two convolutional layers and then diverge into device-specific heads; the strong device uses a $P_k$9-parameter network, whereas the weak device uses a $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$0-parameter network (Rapp et al., 2020). The reported qualitative result is “best of both worlds”: high reward is preserved on powerful devices, while weak devices retain much of the cooperative benefit without forcing everyone into the smallest common architecture (Rapp et al., 2020).

This resource-centric view generalizes in HetRL for RL post-training of LLMs. There the heterogeneity is multi-dimensional: GPU memory, compute throughput, HBM bandwidth, intra-machine bandwidth, inter-device latency, and inter-device bandwidth all enter the cost model (He et al., 13 Dec 2025). HetRL formulates scheduling as an NP-hard joint optimization over RL workflow partitioning and device assignment, then decomposes search into five levels: task grouping, coarse-grained GPU assignment, medium-grained GPU assignment, intra-model parallelization, and fine-grained GPU assignment (He et al., 13 Dec 2025). The system reports up to $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$1 throughput improvement in the abstract and $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$2 on average across workloads and settings, with larger gains under stronger network heterogeneity (He et al., 13 Dec 2025). This makes explicit a point often implicit in HeteroRL: the learning algorithm and the execution substrate cannot always be separated cleanly.

5. HeteroRL as asynchronous RL for LLM post-training

The most literal use of the proper noun “HeteroRL” appears in LLM post-training. Here HeteroRL is an asynchronous architecture with one learner node and four sampler nodes, designed for geographically distributed settings in which rollout generation and parameter learning are decoupled and subject to network delay (Zhang et al., 25 Aug 2025). The distinctive problem is neither classical online RL nor classical offline RL. Rollouts are generated by historical sampler policies $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$3, while updates are applied to a newer learner policy $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$4, where $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$5 is a random staleness induced by communication and compute delay (Zhang et al., 25 Aug 2025).

The paper’s main diagnosis is that delay raises $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$6 between learner and sampler policies and thereby causes importance-sampling variance explosion. The empirical correlations between delay, KL divergence, importance-weight variance, and expected-advantage estimation error range from $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$7 to $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$8 (Zhang et al., 25 Aug 2025). To address this, the paper introduces Group Expectation Policy Optimization (GEPO). Instead of token-level weights, it moves to sequence-level probabilities

$$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$9

and replaces the denominator $\pi_i:\tau_i\rightarrow\Delta(A_i)$0 with a group expectation estimator

$\pi_i:\tau_i\rightarrow\Delta(A_i)$1

The resulting GEIW weight is

$\pi_i:\tau_i\rightarrow\Delta(A_i)$2

Theoretical analysis compares this estimator with standard importance sampling and states that, in the high-KL regime,

$\pi_i:\tau_i\rightarrow\Delta(A_i)$3

so the variance advantage grows at least exponentially in KL up to a constant offset (Zhang et al., 25 Aug 2025).

The empirical results on Qwen3-1.7B-Instruct for mathematical reasoning are correspondingly stability-oriented. On MATH-500, GEPO achieves $\pi_i:\tau_i\rightarrow\Delta(A_i)$4 best and $\pi_i:\tau_i\rightarrow\Delta(A_i)$5 last at delay $\pi_i:\tau_i\rightarrow\Delta(A_i)$6, $\pi_i:\tau_i\rightarrow\Delta(A_i)$7 best and $\pi_i:\tau_i\rightarrow\Delta(A_i)$8 last at delay $\pi_i:\tau_i\rightarrow\Delta(A_i)$9, and $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$0 best and $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$1 last at delay $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$2, whereas GRPO and GSPO suffer pronounced late-training collapse (Zhang et al., 25 Aug 2025). This specific HeteroRL formulation therefore refers not to heterogeneous agents or environments, but to asynchronous RL under heterogeneous distributed infrastructure.

6. Representational interfaces, benchmarks, and unresolved issues

Some HeteroRL formulations are heterogeneous not only in agents or systems but also in representations and knowledge sources. RAEL combines relational reinforcement learning with object-centric symbolic extraction and active human advice querying (Darvishvand et al., 17 Oct 2025). In the unstructured case, the frontend applies

$$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$3

mapping images into symbolic relational states; the downstream learner is Relational Fitted Q-learning with gradient-boosted relational regression trees (Darvishvand et al., 17 Oct 2025). Advice is formalized as

$$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$4

where $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$5 is an optional lifted abstraction and $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$6 is the preferred action set (Darvishvand et al., 17 Oct 2025). This use of heterogeneity spans structured state spaces, raw visual input, and human knowledge, but the paper is explicit that the result is still a pipeline with a symbolic bottleneck rather than end-to-end multimodal fusion (Darvishvand et al., 17 Oct 2025).

Benchmarking remains a central problem. HeMAC was introduced precisely because cooperative HeMARL lacked a standardized testbed comparable to ALE, SMAC, or MuJoCo (Dansereau et al., 23 Sep 2025). It provides $$\boldsymbol\pi(\mathbf a\mid s)=\prod_i\pi^i(a^i\mid s)$$7 scenarios organized into Simple Fleet, Fleet, and Complex Fleet, with agent types Quadcopter, Observer, and Provisioner; the design deliberately combines observation heterogeneity, action-space heterogeneity, capability asymmetry, dynamics heterogeneity, resource asymmetry, and communication asymmetry (Dansereau et al., 23 Sep 2025). The main baseline result is sobering: IPPO and MAPPO outperform a heuristic in the simplest scenario, but performance deteriorates substantially in the harder Fleet and Complex Fleet settings, and QMIX performs worst overall after being adapted by padding observations and actions (Dansereau et al., 23 Sep 2025). This suggests that standardized HeteroRL evaluation is still revealing algorithmic brittleness rather than settled progress.

Several open issues recur across the literature. First, many guarantees are narrow: FedRQ proves asymptotic convergence in the tabular case, HARL/HAML proves monotonic improvement and convergence to Nash equilibrium in cooperative MARL, and HALyPO proves contraction of a disagreement metric, but these theories do not jointly cover partial observability, large function approximation, mixed action spaces, or end-to-end multimodal pipelines (Hwang et al., 19 Jul 2025, Zhong et al., 2023, Zhang et al., 4 Mar 2026). Second, heterogeneity often appears together with other hard problems—delayed communication, verifier noise, symbolic abstraction loss, or severe non-stationarity—so improvements can be highly regime-dependent (Zhang et al., 25 Aug 2025, Darvishvand et al., 17 Oct 2025, Zhang et al., 3 Mar 2026). Third, multiple papers show that naive homogenization is a liability, whether through parameter sharing, padding, or uniform architectures, yet heterogeneity-aware alternatives frequently incur extra coordination, scheduling, or second-order optimization overhead (Zhong et al., 2023, Dansereau et al., 23 Sep 2025, He et al., 13 Dec 2025, Zhang et al., 4 Mar 2026).

Taken together, these works suggest that HeteroRL is not a single algorithmic family but a research program organized around one premise: heterogeneity in agents, environments, resources, modalities, and knowledge sources should be treated as first-class structure in the RL objective, update rule, and systems stack, rather than as noise to be averaged away.