H-MAPPO: Hierarchical Multi-Agent PPO

• H-MAPPO is a family of methods that integrates hierarchical structures with MAPPO’s centralized training and decentralized execution to enhance coordination and scalability.
• It employs varied hierarchies—spatial, task, temporal, and functional—to tackle continuous control, task allocation, and safety under complex system constraints.
• Empirical studies across wind farms, shepherding, traffic management, and electric bus scheduling demonstrate significant improvements in robustness, efficiency, and safety.

Searching arXiv for the cited H-MAPPO and related MAPPO papers to ground the article. Hierarchical Multi-Agent Proximal Policy Optimization (H-MAPPO) denotes a class of methods that combine hierarchy or temporal abstraction, multi-agent coordination, and PPO-style policy optimization, usually within a centralized-training, decentralized-execution regime. In the surveyed arXiv literature, the label is applied to materially different constructions: a communication-enhanced hierarchical extension of MAPPO for continuous wake control in large wind farms, an explicit two-layer target-selection and target-driving architecture for multi-agent shepherding, a graph-based global-planner/local-controller system for emergency-corridor formation, a three-layer constrained formulation for privacy-aware collaborative DNN inference, and an option-based decomposition for electric-bus charging scheduling (Zhang et al., 2023, Covone et al., 3 Apr 2025, Su, 7 Jan 2026, Wang et al., 23 Feb 2026, Qi et al., 15 May 2025). This suggests that H-MAPPO is best understood as a family of structured MAPPO instantiations rather than a single canonical algorithm.

1. Conceptual definition and scope

At its core, H-MAPPO preserves the MAPPO pattern of decentralized actors with a centralized critic, but imposes additional structure on either the action space, the agent topology, the temporal scale of decisions, or all three. In the wind-farm formulation, the method is explicitly presented as a communication-based, hierarchical extension of MAPPO for continuous, multivariate control of axial induction factor $\alpha$, yaw angle $\gamma$, and tilt angle $\beta$ over large turbine arrays (Zhang et al., 2023). In the electric-bus setting, the hierarchy is formalized through two augmented MDPs derived from the original scheduling MDP by means of the options framework, separating charger-and-trip allocation from charging-power control (Qi et al., 15 May 2025). In the shepherding formulation, the hierarchy is explicit and modular: a high-level policy selects targets, while a low-level policy drives the selected target through continuous control (Covone et al., 3 Apr 2025).

The same term also appears in settings where hierarchy is architectural rather than option-theoretic. A traffic-control formulation uses a two-level graph-based architecture in which a high-level planner operates on a global interaction graph and low-level controllers operate on local subgraphs, all trained with MAPPO (Su, 7 Jan 2026). A privacy-aware edge-inference formulation decomposes a constrained multi-agent problem into three hierarchically structured policy layers: an auto-regressive deployment policy, a Lagrangian-enhanced association-and-partitioning policy, and an attention-based resource-allocation policy (Wang et al., 23 Feb 2026).

Several adjacent papers do not explicitly define an H-MAPPO algorithm but nevertheless clarify the conceptual envelope of the term. A MAPPO-based coverage controller with LSTM actors and self-attention critics is presented as directly relevant for designing a hierarchical variant, especially because it handles variable agent counts and value decomposition (Zhao et al., 2022). A multi-robot task-allocation framework similarly does not define H-MAPPO explicitly, but it embodies a high-level task-and-coalition policy together with a low-level motion planner, which is a recognizable hierarchical MAPPO pattern (Bezerra et al., 2024). A plausible implication is that current usage of H-MAPPO is organized more by design principles than by a fixed algorithmic template.

2. Forms of hierarchy used in practice

The literature instantiates hierarchy in several non-equivalent ways. One common form is spatial or communication hierarchy. In the wind-farm controller, the farm is partitioned into overlapping Wind Turbine Aggregators (WTAs), each acting as one learning agent. Neighboring WTAs overlap through shared high-level turbines, and communication proceeds by intra-group aggregation, inter-group sharing, and intra-group broadcast back to low-level turbines (Zhang et al., 2023). This chain-like overlapping structure is designed to reflect spatial locality in wake interactions.

A second form is task hierarchy. The shepherding architecture decomposes the overall problem into target selection and target driving. The high-level policy outputs a discrete target index $T_i^\*$, and the low-level policy outputs continuous control $\mathbf{u}_i\in\mathbb{R}^2$ conditioned on the selected target (Covone et al., 3 Apr 2025). The multi-robot task-allocation framework follows a similar logic: the MAPPO policy selects task locations while a motion planner computes paths, so high-level task commitment and low-level execution are explicitly separated (Bezerra et al., 2024).

A third form is temporal hierarchy. In the queue-jump and emergency-corridor system, the high-level planner runs every $\Delta t_{\text{high}}=5$ simulation steps (2.5 s), whereas low-level controllers run every simulation step (0.5 s) (Su, 7 Jan 2026). The shepherding system also enforces temporal abstraction during training by holding target assignments fixed for $n_w=100$ steps, so that a single high-level selection decision is evaluated over a block of low-level control steps (Covone et al., 3 Apr 2025). In the charging formulation, options persist until termination; charger and trip allocation operate at a slower and variable time scale than power decisions (Qi et al., 15 May 2025).

A fourth form is functional hierarchy across heterogeneous roles. The edge-inference CMDP assigns distinct policy layers to model deployment, user association and partitioning, and server resource allocation, with different action spaces, timescales, and critics (Wang et al., 23 Feb 2026). This is hierarchy by system function rather than by homogeneous agent abstraction.

These constructions also show that H-MAPPO is not restricted to two levels. The shepherding and traffic papers use exactly two levels (Covone et al., 3 Apr 2025, Su, 7 Jan 2026); the edge-inference paper uses three policy layers (Wang et al., 23 Feb 2026); and the wind-farm paper uses overlapping high-level and low-level turbine roles inside a distributed communication structure (Zhang et al., 2023). A common misconception is therefore that H-MAPPO necessarily means a fixed two-level options architecture. The surveyed formulations do not support that restriction.

3. Optimization structure and MAPPO mechanics

Despite architectural variation, the optimization backbone remains recognizably PPO-like. In the wind-farm formulation, each WTA actor uses the clipped PPO surrogate

$$L_{\text{policy}^m}(\theta)= \mathbb{E}_t\!\left[ \min\!\left( r_t^m(\theta)\hat{A}_t^m,\; \operatorname{clip}(r_t^m(\theta),1-\epsilon,1+\epsilon)\hat{A}_t^m \right)\right],$$

with $\gamma=0.99$, $\lambda=0.95$, $\gamma$0, entropy coefficient $\gamma$1, and a shared critic $\gamma$2 that observes the global farm state during training (Zhang et al., 2023). The shepherding formulation uses the standard PPO clipped objective together with a value term and entropy regularization, with $\gamma$3, $\gamma$4, $\gamma$5, and $\gamma$6; the low-level driving PPO and the high-level MAPPO selection policy are trained separately (Covone et al., 3 Apr 2025).

The edge-inference formulation extends this pattern into a constrained setting. User agents optimize a Lagrangian objective

$\gamma$7

with separate reward and cost critics and projected dual ascent

$\gamma$8

thereby embedding MAPPO inside a CMDP with long-term delay constraints (Wang et al., 23 Feb 2026). In that formulation, hierarchy is coupled to safe RL rather than only to temporal abstraction.

The charging formulation is distinctive in that a single centralized critic $\gamma$9 serves both levels. The high-level value is expressed as

$\beta$0

while low-level actors use MAPPO with decentralized execution conditioned on the current option $\beta$1 (Qi et al., 15 May 2025). The high-level composite policy is

$\beta$2

which makes explicit how option persistence and option switching are learned (Qi et al., 15 May 2025).

A related but non-hierarchical theoretical development is Coordinated PPO. CoPPO replaces the isolated PPO ratio by coordinated products of per-agent ratios and introduces double clipping, yielding dynamic credit assignment among concurrently updated policies (Wu et al., 2021). Although CoPPO is not presented as H-MAPPO, it is directly relevant to hierarchical designs because it formalizes how coordinated step-size adaptation can mitigate high variance in multi-agent PPO-style training.

4. Representation, communication, and structured coordination

A defining feature of most H-MAPPO variants is that hierarchy is paired with a structured representation of interactions. In the wind-farm controller, communication is embedded directly in the observation $\beta$3 through neighboring messages $\beta$4, and actions include both continuous turbine controls and outgoing local messages $\beta$5 (Zhang et al., 2023). The communication graph is local and hierarchical rather than fully connected, and high-level turbines mediate information exchange between neighboring WTAs.

Graph-based variants push this idea further. The emergency-corridor system constructs a dynamic interaction graph $\beta$6 with node features, typed edges, and graph-attention message passing. The high-level planner uses a 3-layer GAT on the global graph; the low-level controller uses a 2-layer GAT on local subgraphs and conditions continuous actions on an embedding of the selected high-level strategy (Su, 7 Jan 2026). This architecture is explicitly designed for variable agent counts.

Spatial communication also appears in multi-robot task allocation. There, the actor consumes cropped multi-channel grid observations, while intention sharing is implemented through intention-map channels that encode neighbors’ planned paths with exponential decay along the future trajectory (Bezerra et al., 2024). The learned task-selection policy is therefore informed not only by observed tasks and obstacles but also by neighboring plans. The paper does not call this H-MAPPO, but it exhibits the same high-level/low-level split that hierarchical MAPPO designs often seek.

Attention-based compression is another recurring coordination device. In the charging formulation, each bus constructs a query from its own local state and keys and values from other buses’ local states, producing a context vector

$\beta$7

that replaces the full global state as the input to the high-level agent network (Qi et al., 15 May 2025). In the edge-inference formulation, attention is used at the resource-allocation layer to allocate compute and bandwidth among associated users, while the domain-coverage MAPPO framework identifies self-attention critics and LSTM-based actors as particularly useful for scalable, variable-agent reasoning (Wang et al., 23 Feb 2026, Zhao et al., 2022).

Taken together, these designs suggest a broad architectural regularity: hierarchical MAPPO is rarely only about adding “levels.” It also tends to require inductive bias for locality, partial observability, variable team size, or combinatorial action structure.

5. Representative instantiations and empirical behavior

The empirical literature is strongly domain-specific, but it establishes a consistent pattern: hierarchical structuring is usually introduced to improve scalability, coordination, or safety under large action spaces and localized interactions.

Paper and domain	Hierarchical structure	Reported results
"Collective Large-scale Wind Farm Multivariate Power Output Control Based on Hierarchical Communication Multi-Agent Proximal Policy Optimization" (Zhang et al., 2023)	Overlapping WTAs with high-level and low-level turbines; hierarchical communication; multivariate continuous control	For 22 turbines, M-HCMAPPO achieves 22.79 MW, with +35.41% vs PID, +9.99% vs CMPC, +7.96% vs M-MADDPG, and +1.70% vs M-CMPC
"Hierarchical Policy-Gradient Reinforcement Learning for Multi-Agent Shepherding Control of Non-Cohesive Targets" (Covone et al., 3 Apr 2025)	High-level target selection plus low-level continuous driving PPO	RL remains robust under $\beta$8 variations in target parameters, with >97.5% success rate versus heuristic policies dropping to ~84.2% success
"Hierarchical GNN-Based Multi-Agent Learning for Dynamic Queue-Jump Lane and Emergency Vehicle Corridor Formation" (Su, 7 Jan 2026)	Global high-level planner on a global graph, local low-level controllers on local subgraphs, trained with MAPPO	Emergency-vehicle travel time is reduced by 28.3% compared to baselines and 44.6% compared to uncoordinated traffic; collision rate is 0.3% and background traffic efficiency is 81%
"Safe Multi-Agent Deep Reinforcement Learning for Privacy-Aware Edge-Device Collaborative DNN Inference" (Wang et al., 23 Feb 2026)	Three-layer HC-MAPPO-L with deployment, association/partitioning, and allocation policies; Lagrangian constraint handling	Constrained variants maintain average delay below 3s; service success rate reaches >96–99% at high diversity
"Optimizing Electric Bus Charging Scheduling with Uncertainties Using Hierarchical Deep Reinforcement Learning" (Qi et al., 15 May 2025)	Option-based high-level allocation and low-level MAPPO charging-power control with attention-enhanced scalability	DAC-MAPPO-E matches MILP-D within about 0.18–0.32% average cost gap across scenarios; in a $\beta$9 scenario, DAC-MAPPO-E converges around 18k episodes while basic DAC-MAPPO does not converge after 30k episodes
"Efficient Domain Coverage for Vehicles with Second-Order Dynamics via Multi-Agent Reinforcement Learning" (Zhao et al., 2022)	Not an explicit H-MAPPO method, but a MAPPO foundation for hierarchical variants with LSTM actors and attention critics	Convergence times are 13.35 s vs 25.87 s for 6 agents, 16.76 s vs 28.62 s for 8 agents, and 23.99 s vs 29.72 s for 9 agents

These results should not be collapsed into a single performance narrative, because the tasks, rewards, constraints, and baselines are heterogeneous. What they do show, in a directly documented sense, is that hierarchical structuring has been used to address at least four recurring difficulties: continuous multivariate control over large coupled systems, temporal separation between strategic and tactical decisions, combinatorial assignment problems, and safe optimization under long-term constraints.

6. Limitations, misconceptions, and open directions

A first limitation is definitional. Several papers explicitly state that they do not implement a hierarchical MAPPO algorithm, even though their architectures are useful for building one (Zhao et al., 2022, Bezerra et al., 2024). This indicates that the term still lacks strict standardization. A second misconception is that hierarchy alone guarantees safe or stable behavior. The edge-inference study shows the opposite: unconstrained H-MAPPO and H-IPPO violate the delay target by a large margin, whereas HC-MAPPO-L uses Lagrangian relaxation precisely because hierarchy by itself is insufficient for constraint satisfaction (Wang et al., 23 Feb 2026).

A third limitation concerns the interface between levels. In the shepherding architecture, the low-level driving policy is trained first and then frozen while the high-level selection MAPPO is trained on top of it (Covone et al., 3 Apr 2025). The paper identifies the possibility that a fixed low-level policy can create a suboptimal interface between selection and control. The task-allocation framework exhibits a related issue in another form: low-level motion is delegated to A* rather than learned, which is effective for collision-free path execution but does not learn robot dynamics, energy, or complex kinematics (Bezerra et al., 2024). The charging paper, by contrast, couples levels through a shared critic and learned termination, but it also introduces substantial modeling complexity (Qi et al., 15 May 2025).

Scalability remains an open problem even where hierarchy is already present. The coverage paper notes that scaling attention- and LSTM-based MAPPO from the tested regime to dozens or hundreds of agents would likely require hierarchical grouping, multi-scale attention, or more structured critics (Zhao et al., 2022). The shepherding paper achieves reuse from a $T_i^\*$0 training setup to $T_i^\*$1 through topological sensing, but it also reports imperfect containment under limited sensing and herdability issues (Covone et al., 3 Apr 2025). The edge-inference study addresses scalability through auto-regressive decisions and attention, yet its success depends on a carefully engineered decomposition into deployment, association, partitioning, and allocation layers (Wang et al., 23 Feb 2026).

Communication learning is another unresolved direction. Several hierarchical systems either use no explicit communication, as in the shepherding architecture where agents only observe one another’s positions (Covone et al., 3 Apr 2025), or rely on predesigned message structures, as in the wind-farm WTA hierarchy (Zhang et al., 2023). The literature surveyed here repeatedly points toward learned communication, hierarchical critics across levels, and joint end-to-end training as natural extensions. This suggests that future H-MAPPO work will likely be shaped as much by representation learning and inter-level credit assignment as by PPO mechanics themselves.