---
title: 'H-MAPPO: Hierarchical Multi-Agent PPO'
url: https://www.emergentmind.com/topics/hierarchical-multi-agent-proximal-policy-optimization-h-mappo
type: topic
---

# H-MAPPO: Hierarchical Multi-Agent PPO

Searching arXiv for the cited H-MAPPO and related MAPPO papers to ground the article.
Hierarchical Multi-Agent Proximal Policy Optimization (H-MAPPO) denotes a class of methods that combine hierarchy or temporal abstraction, multi-agent coordination, and PPO-style policy optimization, usually within a centralized-training, decentralized-execution regime. In the surveyed arXiv literature, the label is applied to materially different constructions: a communication-enhanced hierarchical extension of MAPPO for continuous wake control in large wind farms, an explicit two-layer target-selection and target-driving architecture for multi-agent shepherding, a graph-based global-planner/local-controller system for emergency-corridor formation, a three-layer constrained formulation for privacy-aware collaborative DNN inference, and an option-based decomposition for electric-bus charging scheduling [2305.10161] [2504.02479] [2601.04177] [2603.00129] [2505.10296]. This suggests that H-MAPPO is best understood as a family of structured MAPPO instantiations rather than a single canonical algorithm.

## 1. Conceptual definition and scope

At its core, H-MAPPO preserves the MAPPO pattern of decentralized actors with a centralized critic, but imposes additional structure on either the action space, the agent topology, the temporal scale of decisions, or all three. In the wind-farm formulation, the method is explicitly presented as a communication-based, hierarchical extension of MAPPO for continuous, multivariate control of axial induction factor \(\alpha\), yaw angle \(\gamma\), and tilt angle \(\beta\) over large turbine arrays [2305.10161]. In the electric-bus setting, the hierarchy is formalized through two augmented MDPs derived from the original scheduling MDP by means of the options framework, separating charger-and-trip allocation from charging-power control [2505.10296]. In the shepherding formulation, the hierarchy is explicit and modular: a high-level policy selects targets, while a low-level policy drives the selected target through continuous control [2504.02479].

The same term also appears in settings where hierarchy is architectural rather than option-theoretic. A traffic-control formulation uses a two-level graph-based architecture in which a high-level planner operates on a global interaction graph and low-level controllers operate on local subgraphs, all trained with MAPPO [2601.04177]. A privacy-aware edge-inference formulation decomposes a constrained multi-agent problem into three hierarchically structured policy layers: an auto-regressive deployment policy, a Lagrangian-enhanced association-and-partitioning policy, and an attention-based resource-allocation policy [2603.00129].

Several adjacent papers do not explicitly define an H-MAPPO algorithm but nevertheless clarify the conceptual envelope of the term. A MAPPO-based coverage controller with LSTM actors and self-attention critics is presented as directly relevant for designing a hierarchical variant, especially because it handles variable agent counts and value decomposition [2211.05952]. A multi-robot task-allocation framework similarly does not define H-MAPPO explicitly, but it embodies a high-level task-and-coalition policy together with a low-level motion planner, which is a recognizable hierarchical MAPPO pattern [2412.20397]. A plausible implication is that current usage of H-MAPPO is organized more by design principles than by a fixed algorithmic template.

## 2. Forms of hierarchy used in practice

The literature instantiates hierarchy in several non-equivalent ways. One common form is **spatial or communication hierarchy**. In the wind-farm controller, the farm is partitioned into overlapping Wind Turbine Aggregators (WTAs), each acting as one learning agent. Neighboring WTAs overlap through shared high-level turbines, and communication proceeds by intra-group aggregation, inter-group sharing, and intra-group broadcast back to low-level turbines [2305.10161]. This chain-like overlapping structure is designed to reflect spatial locality in wake interactions.

A second form is **task hierarchy**. The shepherding architecture decomposes the overall problem into target selection and target driving. The high-level policy outputs a discrete target index \(T_i^\*\), and the low-level policy outputs continuous control \(\mathbf{u}_i\in\mathbb{R}^2\) conditioned on the selected target [2504.02479]. The multi-robot task-allocation framework follows a similar logic: the MAPPO policy selects task locations while a motion planner computes paths, so high-level task commitment and low-level execution are explicitly separated [2412.20397].

A third form is **temporal hierarchy**. In the queue-jump and emergency-corridor system, the high-level planner runs every \(\Delta t_{\text{high}}=5\) simulation steps (2.5 s), whereas low-level controllers run every simulation step (0.5 s) [2601.04177]. The shepherding system also enforces temporal abstraction during training by holding target assignments fixed for \(n_w=100\) steps, so that a single high-level selection decision is evaluated over a block of low-level control steps [2504.02479]. In the charging formulation, options persist until termination; charger and trip allocation operate at a slower and variable time scale than power decisions [2505.10296].

A fourth form is **functional hierarchy across heterogeneous roles**. The edge-inference CMDP assigns distinct policy layers to model deployment, user association and partitioning, and server resource allocation, with different action spaces, timescales, and critics [2603.00129]. This is hierarchy by system function rather than by homogeneous agent abstraction.

These constructions also show that H-MAPPO is not restricted to two levels. The shepherding and traffic papers use exactly two levels [2504.02479] [2601.04177]; the edge-inference paper uses three policy layers [2603.00129]; and the wind-farm paper uses overlapping high-level and low-level turbine roles inside a distributed communication structure [2305.10161]. A common misconception is therefore that H-MAPPO necessarily means a fixed two-level options architecture. The surveyed formulations do not support that restriction.

## 3. Optimization structure and MAPPO mechanics

Despite architectural variation, the optimization backbone remains recognizably PPO-like. In the wind-farm formulation, each WTA actor uses the clipped PPO surrogate
\[
L_{\text{policy}^m}(\theta)=
\mathbb{E}_t\!\left[
\min\!\left(
r_t^m(\theta)\hat{A}_t^m,\;
\operatorname{clip}(r_t^m(\theta),1-\epsilon,1+\epsilon)\hat{A}_t^m
\right)\right],
\]
with \(\gamma=0.99\), \(\lambda=0.95\), \(\epsilon=0.2\), entropy coefficient \(0.01\), and a shared critic \(V_\phi(s)\) that observes the global farm state during training [2305.10161]. The shepherding formulation uses the standard PPO clipped objective together with a value term and entropy regularization, with \(c_v=0.5\), \(\gamma=0.98\), \(\lambda=0.95\), and \(\epsilon=0.2\); the low-level driving PPO and the high-level MAPPO selection policy are trained separately [2504.02479].

The edge-inference formulation extends this pattern into a constrained setting. User agents optimize a Lagrangian objective
\[
\mathcal{L}_{\mathrm{usr}}(\pi_{\theta_{\mathrm{usr}}},\lambda)
=
J_{\mathrm{usr}}^R(\pi_{\theta_{\mathrm{usr}}})
-
\lambda\!\left(
J_{\mathrm{usr}}^C(\pi_{\theta_{\mathrm{usr}}})-\bar{J}^C
\right),
\]
with separate reward and cost critics and projected dual ascent
\[
\lambda^{t+1}
=
\max\!\Bigl(
0,\;
\lambda^t+\alpha_\lambda(\hat{J}_{\mathrm{usr}}^C-\bar{J}^C)
\Bigr),
\]
thereby embedding MAPPO inside a CMDP with long-term delay constraints [2603.00129]. In that formulation, hierarchy is coupled to safe RL rather than only to temporal abstraction.

The charging formulation is distinctive in that a single centralized critic \(V_\phi(S_t,o_t)\) serves both levels. The high-level value is expressed as
\[
V^{\rm H}(S_t^{\rm H})
=
\sum_{o_t}\pi^{\rm H}(o_t\mid S_t)\,V^{\rm L}(S_t^{\rm L}),
\]
while low-level actors use MAPPO with decentralized execution conditioned on the current option \(o_t\) [2505.10296]. The high-level composite policy is
\[
\pi^{\rm H}(o_t\mid S_t,o_{t-1})
=
(1-\beta_{o_{t-1}}(S_t))\mathbb{I}_{o_t=o_{t-1}}
+
\beta_{o_{t-1}}(S_t)\mu(o_t\mid S_t),
\]
which makes explicit how option persistence and option switching are learned [2505.10296].

A related but non-hierarchical theoretical development is Coordinated PPO. CoPPO replaces the isolated PPO ratio by coordinated products of per-agent ratios and introduces double clipping, yielding dynamic credit assignment among concurrently updated policies [2111.04051]. Although CoPPO is not presented as H-MAPPO, it is directly relevant to hierarchical designs because it formalizes how coordinated step-size adaptation can mitigate high variance in multi-agent PPO-style training.

## 4. Representation, communication, and structured coordination

A defining feature of most H-MAPPO variants is that hierarchy is paired with a structured representation of interactions. In the wind-farm controller, communication is embedded directly in the observation \(o^m(t)\) through neighboring messages \(\zeta^m(t)\), and actions include both continuous turbine controls and outgoing local messages \(b^m(t)\) [2305.10161]. The communication graph is local and hierarchical rather than fully connected, and high-level turbines mediate information exchange between neighboring WTAs.

Graph-based variants push this idea further. The emergency-corridor system constructs a dynamic interaction graph \(G_t=(V_t,E_t,\mathbf{X}_t,\mathbf{E}_t)\) with node features, typed edges, and graph-attention message passing. The high-level planner uses a 3-layer GAT on the global graph; the low-level controller uses a 2-layer GAT on local subgraphs and conditions continuous actions on an embedding of the selected high-level strategy [2601.04177]. This architecture is explicitly designed for variable agent counts.

Spatial communication also appears in multi-robot task allocation. There, the actor consumes cropped multi-channel grid observations, while intention sharing is implemented through intention-map channels that encode neighbors’ planned paths with exponential decay along the future trajectory [2412.20397]. The learned task-selection policy is therefore informed not only by observed tasks and obstacles but also by neighboring plans. The paper does not call this H-MAPPO, but it exhibits the same high-level/low-level split that hierarchical MAPPO designs often seek.

Attention-based compression is another recurring coordination device. In the charging formulation, each bus constructs a query from its own local state and keys and values from other buses’ local states, producing a context vector
\[
S_{i,t}^{\rm att}
=
\sum_{m\ne i} w_{i,m}^{\rm att}V_m
\]
that replaces the full global state as the input to the high-level agent network [2505.10296]. In the edge-inference formulation, attention is used at the resource-allocation layer to allocate compute and bandwidth among associated users, while the domain-coverage MAPPO framework identifies self-attention critics and LSTM-based actors as particularly useful for scalable, variable-agent reasoning [2603.00129] [2211.05952].

Taken together, these designs suggest a broad architectural regularity: hierarchical MAPPO is rarely only about adding “levels.” It also tends to require inductive bias for locality, partial observability, variable team size, or combinatorial action structure.

## 5. Representative instantiations and empirical behavior

The empirical literature is strongly domain-specific, but it establishes a consistent pattern: hierarchical structuring is usually introduced to improve scalability, coordination, or safety under large action spaces and localized interactions.

| Paper and domain | Hierarchical structure | Reported results |
|---|---|---|
| "Collective Large-scale Wind Farm Multivariate Power Output Control Based on Hierarchical Communication Multi-Agent Proximal Policy Optimization" [2305.10161] | Overlapping WTAs with high-level and low-level turbines; hierarchical communication; multivariate continuous control | For 22 turbines, M-HCMAPPO achieves **22.79 MW**, with **+35.41%** vs PID, **+9.99%** vs CMPC, **+7.96%** vs M-MADDPG, and **+1.70%** vs M-CMPC |
| "Hierarchical Policy-Gradient Reinforcement Learning for Multi-Agent Shepherding Control of Non-Cohesive Targets" [2504.02479] | High-level target selection plus low-level continuous driving PPO | RL remains robust under \(30\%\) variations in target parameters, with **>97.5% success rate** versus heuristic policies dropping to **~84.2% success** |
| "Hierarchical GNN-Based Multi-Agent Learning for Dynamic Queue-Jump Lane and Emergency Vehicle Corridor Formation" [2601.04177] | Global high-level planner on a global graph, local low-level controllers on local subgraphs, trained with MAPPO | Emergency-vehicle travel time is reduced by **28.3%** compared to baselines and **44.6%** compared to uncoordinated traffic; collision rate is **0.3%** and background traffic efficiency is **81%** |
| "Safe Multi-Agent Deep Reinforcement Learning for Privacy-Aware Edge-Device Collaborative DNN Inference" [2603.00129] | Three-layer HC-MAPPO-L with deployment, association/partitioning, and allocation policies; Lagrangian constraint handling | Constrained variants maintain average delay below **3s**; service success rate reaches **>96–99%** at high diversity |
| "Optimizing Electric Bus Charging Scheduling with Uncertainties Using Hierarchical Deep Reinforcement Learning" [2505.10296] | Option-based high-level allocation and low-level MAPPO charging-power control with attention-enhanced scalability | DAC-MAPPO-E matches MILP-D within about **0.18–0.32%** average cost gap across scenarios; in a \(M=20, N=10\) scenario, DAC-MAPPO-E converges around **18k episodes** while basic DAC-MAPPO does not converge after **30k episodes** |
| "Efficient Domain Coverage for Vehicles with Second-Order Dynamics via Multi-Agent Reinforcement Learning" [2211.05952] | Not an explicit H-MAPPO method, but a MAPPO foundation for hierarchical variants with LSTM actors and attention critics | Convergence times are **13.35 s** vs **25.87 s** for 6 agents, **16.76 s** vs **28.62 s** for 8 agents, and **23.99 s** vs **29.72 s** for 9 agents |

These results should not be collapsed into a single performance narrative, because the tasks, rewards, constraints, and baselines are heterogeneous. What they do show, in a directly documented sense, is that hierarchical structuring has been used to address at least four recurring difficulties: continuous multivariate control over large coupled systems, temporal separation between strategic and tactical decisions, combinatorial assignment problems, and safe optimization under long-term constraints.

## 6. Limitations, misconceptions, and open directions

A first limitation is definitional. Several papers explicitly state that they do not implement a hierarchical MAPPO algorithm, even though their architectures are useful for building one [2211.05952] [2412.20397]. This indicates that the term still lacks strict standardization. A second misconception is that hierarchy alone guarantees safe or stable behavior. The edge-inference study shows the opposite: unconstrained H-MAPPO and H-IPPO violate the delay target by a large margin, whereas HC-MAPPO-L uses Lagrangian relaxation precisely because hierarchy by itself is insufficient for constraint satisfaction [2603.00129].

A third limitation concerns the interface between levels. In the shepherding architecture, the low-level driving policy is trained first and then frozen while the high-level selection MAPPO is trained on top of it [2504.02479]. The paper identifies the possibility that a fixed low-level policy can create a suboptimal interface between selection and control. The task-allocation framework exhibits a related issue in another form: low-level motion is delegated to A* rather than learned, which is effective for collision-free path execution but does not learn robot dynamics, energy, or complex kinematics [2412.20397]. The charging paper, by contrast, couples levels through a shared critic and learned termination, but it also introduces substantial modeling complexity [2505.10296].

Scalability remains an open problem even where hierarchy is already present. The coverage paper notes that scaling attention- and LSTM-based MAPPO from the tested regime to dozens or hundreds of agents would likely require hierarchical grouping, multi-scale attention, or more structured critics [2211.05952]. The shepherding paper achieves reuse from a \(N=2,M=5\) training setup to \(N=10,M=100\) through topological sensing, but it also reports imperfect containment under limited sensing and herdability issues [2504.02479]. The edge-inference study addresses scalability through auto-regressive decisions and attention, yet its success depends on a carefully engineered decomposition into deployment, association, partitioning, and allocation layers [2603.00129].

Communication learning is another unresolved direction. Several hierarchical systems either use no explicit communication, as in the shepherding architecture where agents only observe one another’s positions [2504.02479], or rely on predesigned message structures, as in the wind-farm WTA hierarchy [2305.10161]. The literature surveyed here repeatedly points toward learned communication, hierarchical critics across levels, and joint end-to-end training as natural extensions. This suggests that future H-MAPPO work will likely be shaped as much by representation learning and inter-level credit assignment as by PPO mechanics themselves.

Source: https://www.emergentmind.com/topics/hierarchical-multi-agent-proximal-policy-optimization-h-mappo