HATRPO-G: Greedy Trust Region in MARL

• The paper demonstrates that HATRPO-G greedily allocates KL budgets based on agents' improvement-to-divergence ratios, enabling faster convergence in heterogeneous-agent settings.
• It modifies baseline HATRPO by replacing uniform per-agent KL thresholds with a shared global budget, addressing bottlenecks and enhancing coordination among agents.
• Empirical benchmarks show that HATRPO-G achieves around a 25% performance gain over HATRPO, though it incurs higher variance compared to the more stable HATRPO-W variant.

Searching arXiv for the cited HATRPO-G and foundational HATRPO/HARL papers. HATRPO-G is the greedy variant of Heterogeneous-Agent Trust Region Policy Optimisation introduced in "Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach" (Shek et al., 14 Aug 2025). It modifies HATRPO by replacing uniform per-agent Kullback-Leibler (KL) thresholds with adaptive allocation under a fixed total KL constraint, prioritizing agents by an improvement-to-divergence ratio. In the reported formulation, the method targets heterogeneous-agent settings in which agents differ in advantage, learning speed, or strategic importance, so that equal trust-region budgets can bottleneck performance, slow convergence, and reinforce locally optimal updates (Shek et al., 14 Aug 2025).

1. Lineage within heterogeneous-agent MARL

HATRPO-G is best understood as a scheduling layer built on top of HATRPO rather than as a standalone departure from the HATRPO paradigm. The foundational HATRPO formulation arises from the extension of trust region methods to cooperative MARL through two ingredients: the multi-agent advantage decomposition lemma and the sequential policy update scheme. In that framework, the joint advantage for a subset of agents is decomposed as

$$A^{i_{1:m}}_{\boldsymbol{\pi}}(s, a^{i_{1:m}}) = \sum_{j=1}^{m} A^{i_j}_{\boldsymbol{\pi}}(s, a^{i_{1:j-1}}, a^{i_j}),$$

which supports agent-by-agent trust-region updates without requiring parameter sharing or restrictive assumptions on decomposibility of the joint value function (Kuba et al., 2021).

The broader theoretical setting is the HARL program, in which HATRPO and HAPPO are derived as tractable approximations to HATRL, and later unified by the HAML framework. In that line of work, HATRPO is characterized as a method for effective cooperation in the general heterogeneous-agent setting, with monotonic improvement of joint return and convergence to Nash Equilibrium under the stated sequential-update conditions (Zhong et al., 2023). HATRPO-G does not replace these antecedents; instead, it addresses a narrower question left open by baseline HATRPO: how to distribute trust-region budget across agents when their marginal learning utility is unequal.

2. Baseline HATRPO and the motivation for a greedy variant

In baseline HATRPO, agents are updated sequentially, and each agent is constrained by a per-agent KL trust region. The 2025 joint-constraint formulation identifies the central limitation as the use of a uniform KL threshold for all agents at each iteration. In heterogeneous settings, this means that high-potential agents may be under-updated while low-potential agents consume trust-region budget with comparatively little benefit (Shek et al., 14 Aug 2025).

The motivation for HATRPO-G follows directly from that observation. The reported problem setting assumes that agents can have different advantage estimates, learning speeds, or strategic importance. Uniform KL constraints therefore become a resource-allocation bottleneck rather than a neutral safety device. The stated consequence is slow convergence, poor escape from local optima, and suboptimal coordination, especially when heterogeneity is pronounced (Shek et al., 14 Aug 2025).

This reframing changes the interpretation of the trust region. In baseline HATRPO, the KL bound is a local stability constraint applied identically across agents. In HATRPO-G, the KL budget becomes a shared resource to be scheduled across agents. A plausible implication is that HATRPO-G preserves the trust-region logic of HATRPO while shifting the design emphasis from per-agent symmetry to cross-agent prioritization.

3. Greedy threshold allocation under a global KL constraint

HATRPO-G allocates the global KL threshold greedily by sequentially selecting agents with the highest improvement-to-divergence ratio and assigning them as much of the remaining KL budget as is optimal (Shek et al., 14 Aug 2025). The method begins by fixing a total KL divergence budget, $\delta_{\text{total}}$, for all agents’ updates in the current iteration and setting each agent’s initial prior policy $\bar{\pi}_i$.

At each iteration, the method computes for each agent $i$ a surrogate improvement objective,

$\mathcal{L}_i(\bar{\pi}_{1:i-1}, \pi_i),$

together with the potential KL divergence for the candidate update. For numerical stability, a small constant $\epsilon > 0$ is added in denominators. The scoring rule is

$$\text{Score}_i = \frac{\mathcal{L}_i(\bar{\pi}_{1:i-1}, \pi_i)} {D^{\max}_{\mathrm{KL}}(\pi_i \parallel \bar{\pi}_i) + \epsilon}.$$

The agent with the highest score is selected, its KL consumption is deducted from the remaining budget, its policy is fixed, and the procedure repeats until all agents are ordered or the KL budget is exhausted (Shek et al., 14 Aug 2025).

The per-agent optimization step is described as

$$\max_{\pi_i}\;\; \mathcal{L}_i(\bar{\pi}_{1:i-1}, \pi_i) \quad \text{s.t.}\quad \mathbb{E}_{s \sim d^{\pi_k}} \!\left[ D_{\mathrm{KL}}\!\left(\pi_i(\cdot|s)\parallel \pi_k^i(\cdot|s)\right) \right] \leq \delta_{\text{remain}},$$

where $\delta_{\text{remain}}$ is the current remaining KL budget (Shek et al., 14 Aug 2025).

The reported interpretation is that HATRPO-G converts constrained sequential policy optimization into a scheduling problem. At each step, it selects the agent whose marginal reward improvement per unit KL is highest and allocates as much KL budget as legally possible to that agent. The paper explicitly relates this intuition to greedy knapsack or matching, and also describes the behavior as a form of soft prioritization (Shek et al., 14 Aug 2025).

4. Relation to HATRPO and HATRPO-W

HATRPO-G is one of two adaptive threshold-allocation methods proposed to improve HATRPO under a global KL constraint. The other is HATRPO-W, a Karush-Kuhn-Tucker-based method that optimizes threshold assignment through water-filling under a shared constraint (Shek et al., 14 Aug 2025).

Method	KL allocation	Reported profile
HATRPO	Identical per-agent KL threshold	Sequential updates; does not reflect heterogeneity
HATRPO-G	Greedy adaptive allocation by improvement-to-divergence ratio	Fastest convergence; higher variance
HATRPO-W	KKT-based water-filling under global KL constraint	Comparable final performance; more stable learning dynamics

The formal contrast is sharp. Baseline HATRPO assigns each agent the same KL threshold. HATRPO-W solves an allocation problem under the global constraint

$$\sum_{i=1}^{m} D^{\max}_{\mathrm{KL}}(\pi_i \parallel \bar{\pi}_i) \leq \delta_{\text{total}},$$

and is described as a global joint optimization approach. HATRPO-G, by contrast, does not perform that joint optimization; it greedily orders updates by the highest reward-improvement-per-KL-cost ratio (Shek et al., 14 Aug 2025).

This difference matters empirically and conceptually. HATRPO-W is reported to have lower variance, whereas HATRPO-G is reported to converge more quickly. The two methods achieve comparable gains in final performance, each exceeding 22.5%, but they occupy different points on the speed–stability trade-off (Shek et al., 14 Aug 2025). A common misconception is to treat the adaptive methods as interchangeable. The reported results do not support that simplification: HATRPO-G is characterized by aggressive prioritization and faster progress, while HATRPO-W is characterized by more stable learning dynamics.

5. Empirical behavior across reported benchmarks

The reported benchmark suite comprises Matrix Game, Differential Game, and Multi-Agent MuJoCo. The Matrix Game is described as having sparse reward and asymmetric advantage structure; the Differential Game as continuous control with local and global optima; and Multi-Agent MuJoCo as physical multi-agent control with high heterogeneity, including Ant, HalfCheetah, Hopper, and Walker (Shek et al., 14 Aug 2025).

Across these benchmarks, HATRPO-G improves final performance by 25.2% over HATRPO on average. HATRPO-G and HATRPO-W are reported to provide similar final performance, with both exceeding 22.5% gains, while HATRPO-G exhibits standard deviation approximately 39% higher than HATRPO-W (Shek et al., 14 Aug 2025). The abstract of the same paper summarizes the same pattern at a higher level: both adaptive methods significantly boost the performance of HATRPO, and HATRPO-W shows more stable learning dynamics as reflected by lower variance (Shek et al., 14 Aug 2025).

The convergence profile is a central reported result. HATRPO-G converges significantly faster than both HATRPO and HATRPO-W, especially under tight KL constraints. In matrix and differential games, it rapidly achieves near-optimal reward and escapes plateaus where HATRPO gets stuck. In environments where agent advantages are skewed, such as cases in which one agent’s choices gate progress for all, HATRPO-G focuses updates on the agents that matter most at that stage of learning (Shek et al., 14 Aug 2025).

The KL-allocation analysis further shows that HATRPO-G’s dynamic assignments track agent advantage: agents with higher advantage receive more KL, and update order changes dynamically over time and learning stages (Shek et al., 14 Aug 2025). This reported behavior is consistent with the core design claim that uniform trust-region budgets are often mismatched to heterogeneous-agent learning dynamics.

6. Scope, implementation, and limitations

The reported practical use case for HATRPO-G is heterogeneous or imbalanced-role multi-agent settings in which agents’ potential learning contributions vary dramatically, or in environments with sparse but highly skewed rewards (Shek et al., 14 Aug 2025). The method is presented as a drop-in replacement for HATRPO’s policy update step: implementation requires computation of per-agent advantage and KL, followed by greedy ranking (Shek et al., 14 Aug 2025).

Its main practical attraction is that it does not require the overhead of global convex optimization as in HATRPO-W. The paper therefore characterizes it as efficient and scalable, particularly when a simple heuristic is preferable or when the number of agents is large (Shek et al., 14 Aug 2025). This suggests that HATRPO-G is especially suitable when wall-clock adaptation speed or escape from poor local minima is more important than minimizing run-to-run dispersion.

The principal reported limitation is higher learning variance relative to HATRPO-W (Shek et al., 14 Aug 2025). The accompanying recommendation is operational rather than theoretical: several random seeds or supplementary regularization may be advisable, and alternating between HATRPO-G for initial exploration and HATRPO-W for late-stage fine-tuning is explicitly suggested (Shek et al., 14 Aug 2025). Another important clarification is that HATRPO-G is not presented as a replacement for the heterogeneous-agent foundations of HATRPO and HARL. Its significance lies in reallocating trust-region budget within that framework, not in altering the underlying sequential-update view of cooperative MARL established by HATRPO and HARL (Kuba et al., 2021).