---
title: 'HATRPO-W: Adaptive Trust Region Optimization'
url: https://www.emergentmind.com/topics/hatrpo-w
type: topic
---

# HATRPO-W: Adaptive Trust Region Optimization

HATRPO-W is a multi-agent trust-region policy optimization method for cooperative multi-agent reinforcement learning (MARL) that extends Heterogeneous-Agent Trust Region Policy Optimization (HATRPO) by allocating the Kullback-Leibler (KL) divergence threshold across agents under a single global KL budget. Introduced in “Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach” [2508.10340], it is a Karush-Kuhn-Tucker-based (KKT-based) method that optimizes threshold assignment under global KL constraints. Its motivating premise is that assigning each agent the same KL threshold can be suboptimal in heterogeneous settings, producing slow or locally optimal updates; HATRPO-W instead redistributes trust-region budgets according to per-agent natural-gradient geometry, while preserving the sequential update structure of HATRPO [2508.10340].

## 1. Research lineage and nomenclature

HATRPO-W is not the original HATRPO. The 2021 work “Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning” introduced HATRPO and HAPPO, established the multi-agent advantage decomposition lemma and the sequential policy update scheme, and justified the monotonic improvement property of HATRPO/HAPPO; that work does not define a variant named “HATRPO-W” [2109.11251]. The 2023 work “Heterogeneous-Agent Reinforcement Learning” likewise develops HATRL, derives HATRPO and HAPPO by tractable approximations, and introduces HAML as a general template for cooperative MARL algorithmic designs, but it also does not define a variant named “HATRPO-W” [2304.09870].

Within that lineage, HATRPO-W should be understood as a later refinement of trust-region budgeting rather than a replacement for the sequential trust-region framework itself. The underlying algorithmic substrate remains the HATRPO regime of agent-by-agent policy improvement in heterogeneous-agent settings; the innovation is the replacement of uniform per-agent KL thresholds by an adaptive allocation under a joint constraint [2508.10340].

## 2. Cooperative-game formulation and the HATRPO baseline

HATRPO-W is formulated in a cooperative Markov (stochastic) game with agents \(i \in \{1,\dots,N\}\), global state \(s \in S\), and joint action \(a=(a_1,\dots,a_N) \in A=\prod_i A_i\). The joint policy factorizes as
\[
\pi(\cdot|s)=\prod_{i=1}^N \pi_i(\cdot|s),
\]
while agents may be heterogeneous, with different observation modalities, action spaces, and policy parameterizations \(\theta_i\). Centralized training with decentralized execution is assumed [2508.10340].

Let \(d^{\pi}(s)\) denote the discounted state visitation distribution under the joint policy \(\pi\), and define the joint advantage
\[
A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s).
\]
For agent \(i\), holding other agents at their old policies, the per-agent advantage used in HATRPO is
\[
A_i^{\pi^{\text{old}}}(s,a_i)
\;\triangleq\;
\mathbb{E}_{a_{-i} \sim \pi_{-i}^{\text{old}}(\cdot|s)}
\big[\,A^{\pi^{\text{old}}}(s,(a_i,a_{-i}))\,\big].
\]
The standard HATRPO surrogate for agent \(i\) updating from \(\pi_i^{\text{old}}\) to \(\pi_i^{\text{new}}\), with other agents fixed at \(\pi_{-i}^{\text{old}}\), is
\[
\max_{\pi_i^{\text{new}}}
\;\;
\mathbb{E}_{s \sim d^{\pi^{\text{old}}},\; a_i \sim \pi_i^{\text{old}}(\cdot|s)}
\left[
\frac{\pi_i^{\text{new}}(a_i\mid s)}{\pi_i^{\text{old}}(a_i\mid s)}
\,A_i^{\pi^{\text{old}}}(s,a_i)
\right],
\]
subject to the per-agent KL trust region
\[
\mathbb{E}_{s \sim d^{\pi^{\text{old}}}}
\big[
\mathrm{KL}\!\big(\pi_i^{\text{old}}(\cdot\mid s)\,\Vert\,\pi_i^{\text{new}}(\cdot\mid s)\big)
\big]
\;\le\;
\varepsilon_i.
\]
Standard HATRPO assigns each agent the same KL threshold. HATRPO-W replaces that uniform assignment by an adaptive allocation under a single global budget [2508.10340].

The natural-gradient quantities are
\[
g_i
\;\triangleq\;
\mathbb{E}_{s \sim d^{\pi^{\text{old}}},\; a_i \sim \pi_i^{\text{old}}}
\big[
\nabla_{\theta_i}\log \pi_i^{\text{old}}(a_i\mid s)\; A_i^{\pi^{\text{old}}}(s,a_i)
\big],
\]
and
\[
F_i
\;\triangleq\;
\mathbb{E}_{s \sim d^{\pi^{\text{old}}},\; a_i \sim \pi_i^{\text{old}}}
\big[
\nabla_{\theta_i}\log \pi_i^{\text{old}}(a_i\mid s)\;
\nabla_{\theta_i}\log \pi_i^{\text{old}}(a_i\mid s)^\top
\big].
\]
Under the TRPO quadratic approximation, the predicted improvement for agent \(i\) with KL budget \(\varepsilon_i\) is
\[
\Delta J_i^{\text{pred}}(\varepsilon_i)
\;=\;
\sqrt{2\,\varepsilon_i}\;\sqrt{g_i^\top F_i^{-1} g_i}.
\]
Writing \(c_i \triangleq g_i^\top F_i^{-1} g_i\) and \(\alpha_i \triangleq \sqrt{2\,c_i}\) makes the threshold-allocation problem explicit [2508.10340].

## 3. Global KL allocation and the KKT solution

The central idea of HATRPO-W is to replace uniform per-agent trust regions with a single weighted global constraint. Given weights \(w_i>0\), the joint budget is
\[
\sum_{i=1}^N
w_i\;\mathbb{E}_{s \sim d^{\pi^{\text{old}}}}
\big[
\mathrm{KL}\!\big(\pi_i^{\text{old}}(\cdot\mid s)\,\Vert\,\pi_i^{\text{new}}(\cdot\mid s)\big)
\big]
\;\le\;
\varepsilon_{\text{tot}}.
\]
Under the standard TRPO per-agent approximation,
\[
\mathbb{E}_s[\mathrm{KL}(\cdot)] \approx \frac{1}{2}\Delta\theta_i^\top F_i \Delta\theta_i \le \varepsilon_i,
\]
so the global constraint reduces to \(\sum_i w_i\,\varepsilon_i \le \varepsilon_{\text{tot}}\) [2508.10340].

HATRPO-W chooses \(\varepsilon_i\) by solving
\[
\max_{\varepsilon_1,\dots,\varepsilon_N \ge 0}
\quad
\sum_{i=1}^N \Delta J_i^{\text{pred}}(\varepsilon_i)
\;=\;
\sum_{i=1}^N \alpha_i \sqrt{\varepsilon_i}
\quad
\text{s.t.}\quad
\sum_{i=1}^N w_i\,\varepsilon_i \le \varepsilon_{\text{tot}}.
\]
The corresponding Lagrangian is
\[
\mathcal{L}(\varepsilon,\lambda,\mu)
\;=\;
\sum_{i=1}^N \alpha_i \sqrt{\varepsilon_i}
\;-\;
\lambda\!\left(\sum_{i=1}^N w_i\,\varepsilon_i - \varepsilon_{\text{tot}}\right)
\;-\;
\sum_{i=1}^N \mu_i\,\varepsilon_i,
\]
with dual variables \(\lambda \ge 0\) and \(\mu_i \ge 0\). For each \(i\) with \(\varepsilon_i>0\),
\[
\frac{\partial \mathcal{L}}{\partial \varepsilon_i}
\;=\;
\frac{\alpha_i}{2\,\sqrt{\varepsilon_i}}
\;-\;
\lambda\,w_i
\;-\;
\mu_i
\;=\; 0
\quad\Longrightarrow\quad
\varepsilon_i
\;=\;
\left(\frac{\alpha_i}{2\,\lambda\,w_i}\right)^2.
\]
Substituting into the active budget constraint yields
\[
\lambda
\;=\;
\tfrac{1}{2}\,
\sqrt{
\frac{1}{\varepsilon_{\text{tot}}}
\sum_{i=1}^N \frac{\alpha_i^2}{w_i}
}.
\]
The resulting closed-form allocation is
\[
\varepsilon_i^*
\;=\;
\varepsilon_{\text{tot}}\,
\frac{\alpha_i^2/w_i^2}{\sum_{j=1}^N \alpha_j^2/w_j}
\;=\;
\varepsilon_{\text{tot}}\,
\frac{c_i/w_i^2}{\sum_{j=1}^N c_j/w_j}.
\]
In the special case \(w_i=1\),
\[
\varepsilon_i^*
\;=\;
\varepsilon_{\text{tot}}\,
\frac{c_i}{\sum_{j=1}^N c_j},
\qquad
c_i = g_i^\top F_i^{-1} g_i.
\]
The paper characterizes this as a separable “water-filling–like” problem and interprets the optimum through the leveling rule
\[
\frac{\alpha_i}{2\,\sqrt{\varepsilon_i^*}\,w_i}
\;=\;
\lambda
\quad\forall i,
\]
so that agents with larger \(c_i\) receive a larger share of the global KL budget, moderated by \(w_i\) [2508.10340].

## 4. Per-agent TRPO step and sequential training procedure

Once the KKT-optimal thresholds are computed, each agent performs the standard TRPO step with its personalized budget:
\[
\Delta \theta_i
\;=\;
\sqrt{
\frac{2\,\varepsilon_i}{g_i^\top F_i^{-1} g_i}
}\;
F_i^{-1} g_i,
\qquad
\theta_i^{\text{new}}
\;=\;
\theta_i^{\text{old}} + \Delta \theta_i.
\]
HATRPO-W therefore changes the step size allocation, not the local trust-region mechanism itself. Agents with higher \(c_i\) make larger steps, but all remain within personalized trust regions [2508.10340].

A training iteration proceeds in four stages. First, trajectories are collected under the current joint policy \(\pi^{\text{old}}\), and per-agent gradients \(g_i\) together with Fisher-vector products for \(F_i^{-1}g_i\) are estimated, typically by conjugate gradient. Second, the thresholds \(\varepsilon_i^*\) are computed in closed form from the KKT solution. Third, agents are updated sequentially, each using its natural-gradient step and an optional line search to ensure the realized average KL satisfies
\[
\mathbb{E}_s[\mathrm{KL}(\pi_i^{\text{old}}\Vert\pi_i^{\text{new}})] \le \varepsilon_i^*.
\]
Fourth, already-updated agents are kept fixed while proceeding to the next agent, and new trajectories are collected for the next outer iteration [2508.10340].

The overhead of the allocation itself is \(O(N)\), since it only requires a single pass to compute \(\sum_j c_j/w_j\). The dominant cost remains per-agent conjugate gradient for \(F_i x = g_i\), reported as \(O(K_{\text{CG}} \cdot |\theta_i|)\) per agent and iteration, using only Fisher-vector products [2508.10340].

## 5. Guarantees and theoretical interpretation

The theoretical status of HATRPO-W is inherited from the trust-region structure of HATRPO. Under standard TRPO assumptions, when the surrogate is optimized under a KL trust region and the line search or damping ensures that the realized KL respects \(\varepsilon_i\), HATRPO’s sequential per-agent updates retain the joint monotonic improvement guarantee; HATRPO-W does not alter the per-agent TRPO guarantee, but only reallocates the KL budgets \(\varepsilon_i\) under a global cap [2508.10340]. The allocation subproblem is strictly concave in \(\varepsilon_i\) over \(\varepsilon_i \ge 0\), its constraints are linear, and the KKT system has a unique solution when \(\varepsilon_{\text{tot}}>0\) and \(c_i \ge 0\); feasibility holds for any \(\varepsilon_{\text{tot}}\ge 0\) [2508.10340].

This guarantee structure rests on earlier HATRPO theory. The 2021 formulation establishes the multi-agent advantage decomposition lemma, the sequential policy update scheme, and a monotonic improvement theorem stating that a sequence of joint policies updated by the theoretical algorithm satisfies \(J(\boldsymbol{\pi}_{k+1})\geq J(\boldsymbol{\pi}_{k})\) for all \(k\), with convergence to Nash equilibria under randomized update order in cooperative Markov games [2109.11251]. The 2023 HARL framework generalizes this perspective through HATRL and HAML, and proves that algorithms derived from HAML inherently enjoy monotonic improvement of joint return and convergence to Nash Equilibrium [2304.09870].

A common misconception is to treat HATRPO-W as a weighted objective over agents. The defining construction is narrower: it is a weighted or globally budgeted allocation of KL thresholds, not a redefinition of the cooperative reward. The objective being optimized in the allocation stage is the sum of TRPO-predicted improvements \(\sum_i \alpha_i\sqrt{\varepsilon_i}\) under \(\sum_i w_i \varepsilon_i \le \varepsilon_{\text{tot}}\) [2508.10340].

## 6. Empirical behavior, comparison, and limitations

HATRPO-W is evaluated on a matrix game, a differential Gaussian game, and Multi-Agent MuJoCo benchmarks including Ant, HalfCheetah, Hopper, and Walker with heterogeneous agent decompositions such as \(2\times4\) Ant, \(4\times2\) Ant, and \(8\times1\) Ant. Baselines are MADDPG, MAPPO, HAPPO, and HATRPO, with centralized training, sequential agent updates, consistent hardware, 5 seeds on MuJoCo, and at least 10 seeds on the smaller games [2508.10340]. The reported findings are that both HATRPO-W and HATRPO-G improve final performance over HATRPO, with gains exceeding \(22.5\%\), while HATRPO-G can reach approximately \(25.2\%\). HATRPO-W is reported to demonstrate lower variance and more stable learning dynamics across tasks, to reduce steps to \(99\%\) of maximum reward in the 4-agent matrix game under tighter KL budgets, to escape local optima in the differential game by adaptive allocation to the high-impact agent, and to achieve notably higher returns on Ant and HalfCheetah variants with KL usage heatmaps showing meaningful, non-uniform allocation over time [2508.10340].

The comparison with HATRPO-G clarifies the role of the weighted KKT solution. HATRPO-W solves a principled KKT system under a global budget and produces a closed-form \(\varepsilon_i^*\) proportional to \(c_i\), whereas HATRPO-G greedily prioritizes agents with the highest local gain per KL and allocates budget iteratively. The paper characterizes HATRPO-W as typically yielding smoother, lower-variance training, while HATRPO-G can be more aggressive and may produce faster initial improvement at the cost of higher variance [2508.10340].

The principal limitations are also explicit. Performance is sensitive to \(\varepsilon_{\text{tot}}\): budgets that are too small can overly constrain learning, while budgets that are too large can break the surrogate’s guarantees. Misallocation can occur if \(c_i\) is noisy because of poor advantage estimates or inaccurate Fisher approximations. Computational overhead remains that of TRPO/HATRPO, since conjugate gradient and Fisher-vector products are still required per agent. The method remains on-policy, so off-policy corrections must be handled carefully to maintain KL control. The formulation is compatible with discrete and continuous policies, but accurate KL estimation is required; Gaussian policies should use exact KL, and categorical policies use a sum over logits [2508.10340].

Source: https://www.emergentmind.com/topics/hatrpo-w