HATRPO: Heterogeneous-Agent TRPO

• HATRPO is a heterogeneous-agent extension of TRPO that enables sequential, non-shared policy updates in cooperative multi-agent settings.
• It leverages the multi-agent advantage decomposition lemma and randomized update orders to ensure monotonic improvement of joint returns and convergence to Nash equilibrium.
• Adaptive variants like HATRPO-G and HATRPO-W optimize KL thresholds per agent, enhancing convergence rates and overall performance in complex MARL tasks.

Searching arXiv for recent HATRPO-related papers to ground the article in the cited literature. Heterogeneous-Agent Trust Region Policy Optimization (HATRPO) is a heterogeneous-agent extension of Trust Region Policy Optimization for cooperative multi-agent reinforcement learning (MARL). It is designed for settings in which agents may have different observation spaces, action spaces, abilities, and roles, so full parameter sharing is not appropriate. Across the HATRL/HARL line of work, HATRPO is characterized by two core ingredients: the multi-agent advantage decomposition lemma and a sequential policy update scheme. Together, these support a trust-region method that optimizes a factorized joint policy via ordered per-agent updates, with monotonic improvement of joint return and convergence-to-Nash-equilibrium results established under the stated assumptions (Kuba et al., 2021, Zhong et al., 2023). Subsequent work has treated HATRPO both as a baseline for heterogeneous MARL and as an object of refinement, including adaptive KL-threshold allocation under a joint trust-region budget (Shek et al., 14 Aug 2025), mirror-descent alternatives (Nasiri et al., 2023), fairness-aware variants (Xu et al., 11 Jun 2026), and domain-specific adaptations such as UAV collaborative beamforming (Liu et al., 2024).

1. Conceptual setting and problem formulation

HATRPO is defined in a heterogeneous-agent setting, where agents are not assumed to be interchangeable copies of a single policy. In the homogeneous case, all agents share the same policy parameters, written as

$\pi^i=\pi,\ \forall i\in\mathcal{N},$

which is the full parameter sharing regime. The heterogeneous setting instead allows

$\pi^i \neq \pi^j \quad \text{for } i\neq j,$

and is motivated by environments in which agents have different observation spaces, action spaces, and roles (Zhong et al., 2023).

In the cooperative formulation used for HATRPO, the environment is a Markov game

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$

with agents $\mathcal{N}=\{1,\dots,n\}$, and a factorized joint policy

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$

The objective is

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$

The standard value, action-value, and advantage functions are

$$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$

$$Q^\pi(s,\mathbf{a})=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s,\mathbf{a}_0=\mathbf{a}\right],$$

$A^\pi(s,\mathbf{a})=Q^\pi(s,\mathbf{a})-V^\pi(s)$

(Shek et al., 14 Aug 2025).

The rationale for HATRPO follows directly from the observation that naïve extensions of single-agent trust-region logic do not generally preserve monotonic improvement in MARL. In cooperative games, each agent’s update changes the effective learning environment of the others. The papers emphasize that joint improvement is not, in general, the sum of independent improvements, and give a two-agent example with reward $r(a^1,a^2)=a^1a^2$ to illustrate how simultaneous local improvements can reduce the joint return (Kuba et al., 2021, Zhong et al., 2023). They also argue that parameter sharing can be insufficient because it can force incompatible behavior onto different agents, cause interference when agents have different abilities or goals, and fail when action or observation spaces differ (Zhong et al., 2023).

2. Theoretical foundations: decomposition and sequential improvement

The key theoretical ingredient underlying HATRPO is the multi-agent advantage decomposition lemma. For any cooperative Markov game, any joint policy $\pi^i \neq \pi^j \quad \text{for } i\neq j,$0, any state $\pi^i \neq \pi^j \quad \text{for } i\neq j,$1, and any ordered subset of agents $\pi^i \neq \pi^j \quad \text{for } i\neq j,$2, the lemma states

$\pi^i \neq \pi^j \quad \text{for } i\neq j,$3

(Zhong et al., 2023, Nasiri et al., 2023, Kuba et al., 2021).

This decomposition rewrites joint advantage as a sum of ordered incremental advantage terms. The significance is methodological: agent $\pi^i \neq \pi^j \quad \text{for } i\neq j,$4 can optimize its own contribution conditioned on earlier agents’ already-updated actions, rather than attempting to optimize the entire joint action in one step. The papers describe this as what makes heterogeneous-agent policy optimization tractable without requiring decomposability of the joint value function in the style of VDN or QMIX (Kuba et al., 2021, Nasiri et al., 2023).

The second ingredient is the sequential update scheme. Rather than updating all agents simultaneously, HATRPO samples a random permutation $\pi^i \neq \pi^j \quad \text{for } i\neq j,$5 and updates agent $\pi^i \neq \pi^j \quad \text{for } i\neq j,$6, then $\pi^i \neq \pi^j \quad \text{for } i\neq j,$7 while accounting for $\pi^i \neq \pi^j \quad \text{for } i\neq j,$8’s update, and so on. Randomization of the order is used in the convergence proof; specifically, every permutation must have a fixed non-zero probability of being selected so that, in the limit, no agent is permanently disadvantaged (Zhong et al., 2023, Kuba et al., 2021).

The local surrogate for agent $\pi^i \neq \pi^j \quad \text{for } i\neq j,$9 in the sequential scheme is written as

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$0

(Kuba et al., 2021, Zhong et al., 2023, Shek et al., 14 Aug 2025).

On this basis, the trust-region improvement bound is extended to the multi-agent setting: $$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$1 with

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$2

in the HARL presentation, and equivalently

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$3

in the earlier trust-region MARL formulation (Zhong et al., 2023, Kuba et al., 2021, Nasiri et al., 2023).

The stated guarantees are twofold. First, the HATRL/HATRPO-style algorithm satisfies monotonic improvement,

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$4

Second, under randomized update order where every permutation has non-zero probability, the set of limit points is non-empty and each limit point is a Nash equilibrium (Zhong et al., 2023, Kuba et al., 2021). In later work on $$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$5-fair heterogeneous-agent reinforcement learning, analogous guarantees are retained for a fair-objective extension of HATRPO under finite state and action spaces, bounded positive rewards, $$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$6-soft policies, and a contextualized state construction (Xu et al., 11 Jun 2026).

3. Algorithmic structure and practical update rule

HATRPO is the practical deep-RL approximation of the theoretical HATRL procedure. In the theoretical form, each agent in the selected order solves

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$7

(Zhong et al., 2023, Kuba et al., 2021).

The practical HATRPO replaces the hard maximal-KL penalty by an expected KL constraint. For agent $$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$8, the constrained problem is

$$\mathcal{M}=\langle \mathcal{N},\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma\rangle,$$9

subject to

$\mathcal{N}=\{1,\dots,n\}$0

(Zhong et al., 2023, Kuba et al., 2021). A parallel statement is given in the 2025 joint-constraint paper, which writes the per-agent update with expected KL threshold $\mathcal{N}=\{1,\dots,n\}$1 under the sequential HATRPO rule (Shek et al., 14 Aug 2025).

Using a linear approximation of the objective and a quadratic approximation of the KL constraint, the update becomes TRPO-like: $\mathcal{N}=\{1,\dots,n\}$2 where $\mathcal{N}=\{1,\dots,n\}$3 is the gradient of the surrogate objective, $\mathcal{N}=\{1,\dots,n\}$4 is the Hessian of the expected KL divergence, and $\mathcal{N}=\{1,\dots,n\}$5 is chosen by backtracking line search (Zhong et al., 2023, Kuba et al., 2021).

A practical computational feature of HATRPO is that it uses a single joint advantage estimator rather than a separate critic per agent. The sequential estimator is implemented through an importance-weight term. One form given in the literature is

$\mathcal{N}=\{1,\dots,n\}$6

which enters the policy gradient estimator for agent $\mathcal{N}=\{1,\dots,n\}$7 (Kuba et al., 2021). The HARL exposition states that HATRPO can therefore use a single joint advantage estimator rather than a separate critic per agent (Zhong et al., 2023).

The appendix-level pseudocode summarized in the HARL work proceeds as follows: initialize actor networks, a global value network, and replay buffer; collect trajectories; compute advantage estimates using the global value network and generalized advantage estimation; draw a random permutation of agents; perform sequential natural-gradient updates with conjugate gradient and backtracking line search; update the compound importance weight passed to the next agent; and finally update the value network by regression to returns (Zhong et al., 2023).

4. Relation to adjacent methods and frameworks

HATRPO is repeatedly positioned as the heterogeneous-agent TRPO baseline. Relative to standard TRPO, it differs in four respects stated explicitly in the literature: it optimizes a factorized joint policy into heterogeneous per-agent policies; updates agents sequentially rather than all at once; uses a per-agent surrogate objective conditioned on prior agents’ updates; and proves monotonic improvement for the joint return (Zhong et al., 2023, Kuba et al., 2021).

Relative to homogeneous-agent on-policy methods such as IPPO and MAPPO, the defining contrasts are the absence of required parameter sharing and the sequential update structure (Zhong et al., 2023, Nasiri et al., 2023). The heterogeneous MARL papers state that HATRPO and related methods do not require homogeneous action spaces, parameter sharing, or decomposability of the joint $\mathcal{N}=\{1,\dots,n\}$8-function (Kuba et al., 2021).

HATRPO also occupies a specific place within a broader theoretical lineage. The 2023 HARL paper describes HATRL as the provably correct heterogeneous-agent trust-region learning procedure, with HATRPO and HAPPO derived from tractable approximations. It then introduces Heterogeneous-Agent Mirror Learning (HAML), a general framework that subsumes HATRPO and strengthens the theoretical guarantees for HATRPO and HAPPO while also validating algorithms such as HAA2C, HADDPG, and HATD3 (Zhong et al., 2023). In that formulation, HATRPO is recovered as an instance of HAML using a KL-ball neighborhood

$\mathcal{N}=\{1,\dots,n\}$9

(Zhong et al., 2023).

Mirror-descent work further sharpens this positioning. "Heterogeneous Multi-Agent Reinforcement Learning via Mirror Descent Policy Optimization" identifies HATRPO as the trust-region, natural-gradient, TRPO-style baseline and contrasts it with HAMDPO, which does not explicitly enforce the trust-region constraint, approximately satisfies it via multiple SGD steps, uses simple SGD instead of natural gradient, and adopts a KL direction consistent with mirror descent rather than the one used in HATRPO (Nasiri et al., 2023). The paper treats HATRPO and HAPPO as “extensions of single-agent TRPO/PPO methods designed for MARL with heterogeneous agents” (Nasiri et al., 2023).

The fairness extension in "$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$0-fair heterogeneous agent reinforcement learning" preserves the HATRPO sequential trust-region structure but replaces the utilitarian objective by an $\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$1-fair welfare objective,

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$2

where

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$3

The corresponding fair advantage is

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$4

(Xu et al., 11 Jun 2026). This suggests that HATRPO’s sequential trust-region template is sufficiently general to host alternative welfare criteria while maintaining the original proof architecture.

5. Adaptive trust-region allocation: HATRPO-G and HATRPO-W

A 2025 refinement, "Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach" (Shek et al., 14 Aug 2025), focuses on a specific limitation of original HATRPO: the use of the same KL threshold $\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$5 for every agent. In the original formulation this appears as

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$6

The paper argues that uniform thresholds are suboptimal in heterogeneous settings because agents may differ greatly in learning potential, advantage magnitude, role or significance in the joint task, and curvature or local difficulty of the policy landscape (Shek et al., 14 Aug 2025).

The proposed alternative is to replace identical per-agent thresholds with a joint constrained optimization,

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$7

subject to

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$8

The total trust-region budget is fixed, but agents share it adaptively (Shek et al., 14 Aug 2025).

Two allocation methods are introduced.

The first, HATRPO-G, is a greedy ordering and allocation procedure based on the score

$\pi(\cdot\mid s)=\prod_{i=1}^n \pi_i(\cdot\mid s).$9

where $$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$0 is a numerical stabilizer. Agents are prioritized by improvement-to-divergence ratio, with the algorithm repeatedly evaluating local improvement, computing the ratio, selecting the highest-scoring agent, fixing that policy, and removing the agent from the remaining set (Shek et al., 14 Aug 2025).

The second, HATRPO-W, is a KKT-based allocation rule with water-filling intuition. It relaxes the total KL constraint using a Lagrange multiplier $$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$1,

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$2

approximates the local objective and KL cost as

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$3

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$4

and derives the allocation

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$5

with $$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$6 determined numerically, for example by bisection, under the global budget condition

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$7

(Shek et al., 14 Aug 2025).

The empirical results reported for these adaptive variants are specific. HATRPO-G achieves a 25.2% improvement in final performance over vanilla HATRPO, while HATRPO-W achieves a 22.5% improvement. Both exceed 22.5% gain relative to original HATRPO. The paper also reports that HATRPO-W is more stable, and that HATRPO-G’s standard deviation is 39% greater than HATRPO-W’s (Shek et al., 14 Aug 2025). Across matrix game, differential game, and Multi-Agent MuJoCo benchmarks, both adaptive methods converge faster than original HATRPO; in the matrix game they reach optimal reward in fewer iterations under tight KL budgets, in the differential game they escape a local optimum while HATRPO remains trapped, and in MuJoCo tasks they achieve higher returns, especially on Ant and HalfCheetah variants (Shek et al., 14 Aug 2025).

A common misconception would be to identify the trust region solely with equal per-agent KL radii. The joint-constraint work explicitly argues otherwise: stability does not require equal trust regions, but rather update sizes appropriate to each agent’s contribution and local geometry under a conserved global KL budget (Shek et al., 14 Aug 2025). This is an interpretation internal to that paper’s reformulation, not a replacement of the original HATRPO theory.

6. Empirical behavior and domain-specific adaptations

The original trust-region MARL paper evaluates HATRPO and HAPPO on Multi-Agent MuJoCo and SMAC tasks, including heterogeneous control problems such as Ant, HalfCheetah, Walker, Humanoid, HumanoidStandup, and Swimmer variants. The reported trend is that HATRPO and HAPPO consistently outperform IPPO, MAPPO, and MADDPG in return and variance on Multi-Agent MuJoCo, with HATRPO typically the best or near-best and often slightly better than HAPPO. On SMAC, the methods are competitive, though the paper notes that IPPO and MAPPO already reach 100% win rate on most tested maps, making SMAC a weaker discriminator for the benefits of non-parameter-sharing trust-region MARL (Kuba et al., 2021).

The HARL paper broadens the empirical scope to MPE, MAMuJoCo, SMAC, and SMACv2, reporting that HATRPO generally outperforms or matches strong baselines such as MAPPO and QMIX on heterogeneous tasks, exhibits stable learning curves due to explicit trust-region control, and on the 17-agent Humanoid task in MAMuJoCo still learns successfully where MAPPO fails (Zhong et al., 2023). It also notes that HATRPO outperforms HAPPO on 10 of 15 SMAC/SMACv2 tasks, suggesting that the explicit KL constraint helps stability and robustness (Zhong et al., 2023).

The mirror-descent paper uses HATRPO as a principal baseline on Ant-v2, HalfCheetah-v2, Walker2d-v2, and the SMAC 3s5z map, reporting that HAMDPO outperforms both HATRPO and HAPPO in convergence and overall performance in the reported experiments (Nasiri et al., 2023). This does not negate the earlier HATRPO results; rather, it situates HATRPO as a benchmark embodying the trust-region, heterogeneous, sequential-update paradigm.

A domain-specific adaptation appears in "UAV-enabled Collaborative Beamforming via Multi-Agent Deep Reinforcement Learning" (Liu et al., 2024), where HATRPO serves as the basic framework for a cooperative multi-UAV control problem. The paper formulates a UAV-enabled Collaborative Beamforming Multi-Objective Optimization Problem with objectives

$$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$8

namely maximizing transmission rate and minimizing total motion energy consumption, and converts it into a cooperative Markov game (Liu et al., 2024). It uses HATRPO because of its sequential trust-region update and monotonic-improvement structure, but introduces three modifications in a method called HATRPO-UCB: observation enhancement via spherical coordinates and a reference-point representation for the base station, an agent-specific global state for critic learning, and a Beta distribution policy to handle bounded continuous actions (Liu et al., 2024).

The reported numerical comparison in that application states that HATRPO-UCB converges fastest among the compared methods, around 750 epochs, and attains the best energy consumption results: for the first base station, energy $$J(\pi)=\mathbb{E}\Big[\sum_{t=0}^\infty \gamma^t r_t\Big].$$9 J and rate $$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$0 bps; for the second base station, energy $$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$1 J and rate $$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$2 bps (Liu et al., 2024). This suggests that HATRPO’s trust-region machinery is portable to domain-specific settings, provided observation design, critic structure, and action distributions are adapted to the application.

7. Interpretive themes, assumptions, and ongoing directions

Several recurrent themes organize the literature on HATRPO. First, the method is repeatedly distinguished from both independent learning and homogeneous-agent CTDE by its explicit accommodation of heterogeneity. The papers treat heterogeneous agents as having separate policies, potentially different action spaces, and differentiated roles; HATRPO is therefore not merely “TRPO applied separately to each agent,” but a coordinated sequential trust-region method grounded in an advantage decomposition theorem (Kuba et al., 2021, Zhong et al., 2023, Nasiri et al., 2023).

Second, stability is central to its identity. The trust region is enforced through expected KL constraints in the practical algorithm, and the literature consistently attributes smooth learning curves and robustness to this structure (Kuba et al., 2021, Zhong et al., 2023). At the same time, later work indicates that the original uniform allocation of KL budget may be overly conservative in heterogeneous settings, motivating adaptive scheduling mechanisms such as HATRPO-G and HATRPO-W (Shek et al., 14 Aug 2025). A plausible implication is that the trust-region concept in heterogeneous MARL is shifting from fixed local radii toward structured budget allocation across agents.

Third, HATRPO has become a reference point for broader methodological developments. HAML interprets it as one instance of a more general heterogeneous-agent mirror-learning template (Zhong et al., 2023). HAMDPO reformulates the trust-region idea in mirror-descent terms and replaces natural-gradient machinery with multiple SGD steps (Nasiri et al., 2023). The $$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$3-fair extension replaces utilitarian optimization with welfare-sensitive optimization while preserving monotonic-improvement and Nash-equilibrium conclusions under stated assumptions (Xu et al., 11 Jun 2026). These developments suggest that HATRPO functions both as a practical algorithm and as a canonical theoretical scaffold.

The assumptions needed for the strongest guarantees are also explicit. The original trust-region analyses rely on randomized update order and an $$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$4-soft policy condition in which every policy assigns at least $$V^\pi(s)=\mathbb{E}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right],$$5 probability to every action in every state (Kuba et al., 2021). The fairness extension further assumes finite state and action spaces, bounded positive rewards, and a contextualized state construction (Xu et al., 11 Jun 2026). By contrast, domain-adaptation papers such as the UAV beamforming study explicitly note that theoretical convergence bounds for the full deep multi-agent system with many hyperparameters, nonlinear function approximation, high-dimensional state and action spaces, and stochastic interactions are difficult, so evaluation is empirical rather than proof-based (Liu et al., 2024).

In aggregate, HATRPO occupies a distinct place in heterogeneous MARL: it is the tractable deep-RL implementation of a sequential trust-region method derived from a multi-agent policy-improvement bound; it avoids parameter sharing and value-factorization assumptions; it provides monotonic improvement and Nash-equilibrium results under stated conditions; and it has served as the basis for later work on adaptive KL allocation, mirror descent, fairness-aware objectives, and specialized multi-agent control systems (Kuba et al., 2021, Zhong et al., 2023, Shek et al., 14 Aug 2025, Nasiri et al., 2023, Xu et al., 11 Jun 2026, Liu et al., 2024).