HAG-PS: Hierarchical Adaptive Parameter Sharing

• HAG-PS is a hierarchical framework that organizes agents into adaptive groups, coupling structured parameter sharing with group-specific policy learning.
• It leverages a shared global trunk with local actor–critic heads and identity embeddings to achieve efficient coordination and memory savings in multi-agent reinforcement learning.
• Empirical studies on urban mobility and cooperative pairing highlight HAG-PS’s improvements in service fulfillment and policy stability compared to flat sharing methods.

Hierarchical Adaptive Grouping-based Parameter Sharing (HAG-PS) denotes a family of parameter-sharing designs in which agents are grouped at multiple levels, policies or modules are shared within those groups, and group structure is adjusted or selected in conjunction with control decisions. In its explicit multi-agent reinforcement learning formulation for urban mobility resource allocation, HAG-PS combines a shared global feature trunk, local actor–critic heads, adaptive split–merge grouping based on encoded trajectories, and learnable identity embeddings under centralized training with decentralized execution (CTDE) (Nooshi et al., 27 Jul 2025). A related hierarchical reinforcement learning formulation treats grouping itself as a high-level decision and low-level actions as intra-group behavior, making HAG-PS a broader architectural pattern for coupling group formation and policy learning rather than a single immutable algorithm (Hu, 11 Jan 2025).

1. Conceptual definition and scope

HAG-PS is organized around three coupled ideas. First, grouping is explicit: each agent is assigned to a group, or a high-level controller selects a grouping structure. Second, parameter sharing is structured: some parameters are shared broadly, while others are shared only within subgroups. Third, grouping is adaptive or decision-dependent rather than purely fixed. In the hierarchical RL formulation, each agent $i$ is assigned through a discrete variable $z_i \in \{1,\dots,K\}$, the high-level policy is written as $\pi_{\text{high}}(z \mid s)$, and low-level intra-group policies are written as $\pi_k(a^i \mid o^i; \theta_k)$ for agents assigned to group $k$ (Hu, 11 Jan 2025).

The mobility-resource-allocation instantiation makes the hierarchy concrete. Agents represent regional coordinators, the service area is discretized into rectangular regions, and HAG-PS uses global groups and local groups. A shared global trunk $T_{\theta_c}$ transforms the global state into a context embedding, while compact local actor–critic heads specialize within local groups. Agent individuality is retained through trainable identity embeddings $e_i$, so specialization is not reduced to literal parameter copying (Nooshi et al., 27 Jul 2025).

A central boundary condition is that not every grouping-based sharing method is HAG-PS in the strong sense. Flat selective sharing schemes such as Selective Parameter Sharing (SePS) cluster agents once before RL and then keep assignments fixed; they are important precursors, but they do not implement hierarchical adaptive grouping during training (Christianos et al., 2021). This distinction matters because the term “hierarchical” refers not merely to sharing different modules, but to a layered organization of grouping and control.

2. Antecedents in selective parameter sharing

The immediate methodological background is the literature on parameter sharing in multi-agent reinforcement learning. That literature established that sharing parameters reduces the number of trainable parameters, shortens training times, and can improve sample efficiency, but it also showed that indiscriminate sharing can be harmful when agents differ in reward functions, observation transitions, or action spaces. The reported failure modes include representation bottlenecks, interference between heterogeneous agents, and sharp environment dependence; in Blind-Particle Spread, full sharing can collapse to a local minimum, and even identity-conditioned full sharing can deteriorate as role diversity increases (Christianos et al., 2021).

SePS was introduced precisely to avoid those pathologies. Its core idea is to learn an embedding for each agent that captures agent-centric observation transition and reward functions, then cluster agents in that embedding space and share parameters only within clusters. Formally, the method defines approximations $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$ and $\hat{R}^i: O^i \times A^i \to \mathbb{R}$, learns a latent representation with an encoder–decoder objective, and then applies $k$-means to encoder means $z_i \in \{1,\dots,K\}$0 to obtain a deterministic mapping $z_i \in \{1,\dots,K\}$1 and group-shared policies $z_i \in \{1,\dots,K\}$2 and values $z_i \in \{1,\dots,K\}$3 (Christianos et al., 2021).

For HAG-PS, SePS functions as a flat lower-level precursor rather than a direct implementation. The SePS paper explicitly states that grouping is learned prior to RL training and is static during policy learning, that the implemented structure is flat rather than hierarchical, and that dynamic re-partitioning is only suggested as future work. This makes SePS an important source of bottom-level grouping logic, but not a full HAG-PS system (Christianos et al., 2021).

3. Architectural patterns and mathematical formulation

Two explicit instantiations define the core HAG-PS design space. In the mobility setting, the environment state aggregates temporal encodings, regional availability, demand statistics, and environmental features, while each agent takes a vector relocation action $z_i \in \{1,\dots,K\}$4. The policy is hierarchical: for agent $z_i \in \{1,\dots,K\}$5 in local group $z_i \in \{1,\dots,K\}$6,

$z_i \in \{1,\dots,K\}$7

where $z_i \in \{1,\dots,K\}$8 are global trunk parameters, $z_i \in \{1,\dots,K\}$9 are local head parameters, and $\pi_{\text{high}}(z \mid s)$0 is a low-dimensional trainable identity embedding. The MARL objective is

$\pi_{\text{high}}(z \mid s)$1

The per-agent reward combines service quality and relocation cost: $\pi_{\text{high}}(z \mid s)$2 This design yields hierarchical sharing because large trunk parameters are amortized across agents, while local heads remain compact and group-specific (Nooshi et al., 27 Jul 2025).

In the cooperative pairing formulation, HAG-PS appears as a two-level hierarchy with high-level grouping options and low-level actions. The grouping variable is an option $\pi_{\text{high}}(z \mid s)$3 that specifies a partition of teams into disjoint pairs,

$\pi_{\text{high}}(z \mid s)$4

The high-level policy is $\pi_{\text{high}}(z \mid s)$5, and the low-level intra-option policy is

$\pi_{\text{high}}(z \mid s)$6

Here the paper uses complete sharing of $\pi_{\text{high}}(z \mid s)$7 across all teams and options, which is a strong form of parameter sharing appropriate to the homogeneous-agent assumption adopted in that work (Hu, 11 Jan 2025).

These two formulations emphasize different aspects of HAG-PS. The mobility formulation prioritizes hierarchical trunk–head decomposition, adaptive local regrouping, and identity-based specialization in a heterogeneous urban system. The pairing formulation prioritizes timescale separation, option-critic structure, and permutation-invariant representations for homogeneous cooperative teams. A plausible implication is that HAG-PS is best understood as an architectural template whose concrete instantiation depends on whether heterogeneity, combinatorial grouping, or memory efficiency is the dominant systems concern.

4. Adaptive grouping mechanisms and optimization

The defining adaptive mechanism in the mobility formulation is split–merge regrouping based on recent trajectories. After each episode, agent $\pi_{\text{high}}(z \mid s)$8 encodes its last $\pi_{\text{high}}(z \mid s)$9-step trajectory with a variational LSTM: $\pi_k(a^i \mid o^i; \theta_k)$0 For each local group $\pi_k(a^i \mid o^i; \theta_k)$1, the centroid embedding and within-group dispersion are

$\pi_k(a^i \mid o^i; \theta_k)$2

$\pi_k(a^i \mid o^i; \theta_k)$3

If $\pi_k(a^i \mid o^i; \theta_k)$4 exceeds a threshold $\pi_k(a^i \mid o^i; \theta_k)$5 and the group is large enough, the group is bisected with $\pi_k(a^i \mid o^i; \theta_k)$6-means. If two groups are sufficiently close under symmetric KL divergence, they are merged. Regrouping frequency is itself adaptive through the exponentially smoothed average dispersion $\pi_k(a^i \mid o^i; \theta_k)$7 and the interval

$\pi_k(a^i \mid o^i; \theta_k)$8

This makes regrouping frequent when behaviors drift and infrequent when local groups stabilize (Nooshi et al., 27 Jul 2025).

Optimization in that setting uses Proximal Policy Optimization. The combined objective is

$\pi_k(a^i \mid o^i; \theta_k)$9

Gradients are propagated through the global trunk, local heads, and identity embeddings, with reported hyperparameters $k$0, GAE $k$1, policy learning rate $k$2, value learning rate $k$3, and local heads implemented as MLPs with 128 hidden units (Nooshi et al., 27 Jul 2025).

The hierarchical RL pairing formulation uses CTDE with an option-critic-style centralized value structure: $k$4

$k$5

$k$6

To avoid enumerating the full option space, the method uses a permutation-invariant critic and a greedy maximum-weight perfect matching solver over pair scores $k$7. Termination is fixed to weekly intervals rather than learned: $k$8 This illustrates a second route to HAG-PS: adaptive grouping can be expressed either as explicit regrouping of parameter-sharing units or as high-level option selection over group structures (Hu, 11 Jan 2025).

5. Empirical evidence, scalability, and domain-specific behavior

The most explicit empirical evidence for HAG-PS comes from the mobility study on 1,232,838 Citi Bike trips from January 2024, aggregated into $k$9 regions covering Manhattan. In that setting, HAG-PS achieved a fulfilled service ratio of $T_{\theta_c}$0 and rebalanced 472,212 bikes. The reported baselines were No-Share $T_{\theta_c}$1, Share-All $T_{\theta_c}$2, CDS $T_{\theta_c}$3, SePS $T_{\theta_c}$4, and DyPS $T_{\theta_c}$5. Ablation results further attribute the gain to specific components: removing identity embeddings reduces availability by approximately $T_{\theta_c}$6 percentage points, disabling split–merge or hierarchical grouping yields drops of approximately $T_{\theta_c}$7 and $T_{\theta_c}$8 points, and fixing the regrouping period reduces approximately $T_{\theta_c}$9 points (Nooshi et al., 27 Jul 2025).

The same study also provides an explicit memory-efficiency analysis. A naive per-agent parameterization has memory $e_i$0, while HAG-PS reduces this to

$e_i$1

where $e_i$2 is the number of global groups, $e_i$3 the trunk parameter count, $e_i$4 the number of local groups, $e_i$5 the head parameter count, and $e_i$6 the identity embedding size. Because the trunk is large and shared while local heads are small, the design is intended for urban-scale MARL under realistic memory budgets (Nooshi et al., 27 Jul 2025).

The hierarchical RL grouping paper provides evidence in a different domain: weekly pairing of cooperative teams. Its evaluation uses 10 teams, each with 10 subjects, a binary action space, reward defined as the square root of the next day’s step count, $e_i$7, training for 10,000 iterations, evaluation for 1,000 steps, and averaging over 20 repetitions. The reported values for the proposed method are $e_i$8 average reward and $e_i$9 $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$0-discounted cumulative reward, alongside fixed-action and random baselines. The significance of this result is less in the absolute metric than in the demonstration that grouping and policy can be learned simultaneously through a hierarchical controller with CTDE and option-critic structure (Hu, 11 Jan 2025).

The precursor SePS results clarify when hierarchical or selective sharing is needed. In heterogeneous settings, selective sharing strongly outperformed naive full sharing: for example, in BPS-h(3) with 200 agents, NoPS was infeasible, while SePS achieved $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$1 against SAP $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$2 and SAP+id $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$3. In C-RWARE(3) with 16 agents, SePS achieved $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$4, versus NoPS $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$5, SAP $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$6, and SAP+id $\hat{P}^i: O^i \times A^i \to \Delta(O^i)$7. These results do not constitute HAG-PS directly, but they provide the empirical rationale for moving from indiscriminate sharing toward group-aware and eventually hierarchical sharing (Christianos et al., 2021).

6. Variants, misconceptions, and open problems

A recurrent misconception is that any adaptive sharing scheme is already hierarchical. The supplied literature consistently distinguishes flat methods from true HAG-PS. AdaPS, for example, is described as adaptive, grouping-based parameter sharing in which agents are clustered by learned identity features and then routed to distinct subnetworks carved from a single shared backbone via fixed binary masks. However, it is explicitly characterized as a flat, one-shot grouping method rather than a hierarchical, coarse-to-fine grouping scheme, with fixed hard masks and no online reassignment during RL (Li et al., 2023).

A second misconception is that hierarchy always implies the same kind of adaptation. In the pairing formulation, adaptation is at the level of selecting weekly pairings through options and maximum-weight matching, while termination is fixed rather than learned (Hu, 11 Jan 2025). In the mobility formulation, adaptation is continuous through VLSTM trajectory encoding, split–merge logic, and a regrouping interval controlled by dispersion (Nooshi et al., 27 Jul 2025). These are not interchangeable mechanisms; they address different sources of non-stationarity.

The broader literature in the supplied corpus also uses HAG-PS as an organizing template outside its original MARL mobility setting. Hierarchical Adapters Merging (HAM) is presented as a continual learning instantiation in which tasks are dynamically grouped by adapter similarity, consolidated within groups through pruning and concatenation, and then globally merged into a single inference-time adapter (Coleman et al., 16 Sep 2025). The Hierarchical Recurrent Adapter (HRA) is described as a parameter-sharing hierarchy with a shared recurrent controller and task-level adapter heads reused across layers of a speech backbone (Munkhdalai et al., 2024). HAPS, a hierarchical LLM routing framework, maps HAG-PS to a high-level architecture router, a low-level parameter generator, and a shared backbone that is used by both routers (Tian et al., 9 Jan 2026). These mappings suggest that HAG-PS has become a general design vocabulary for hierarchical grouping plus structured sharing, even when the domain is not cooperative MARL.

The open problems are consistent across instantiations. The mobility paper notes that split–merge decisions depend on variational encodings and KL thresholds, that station capacities and travel-time constraints are not explicitly modeled, and that formal convergence guarantees are not provided (Nooshi et al., 27 Jul 2025). The pairing paper highlights the combinatorial growth of grouping options, reliance on homogeneity assumptions, and potential instability if termination were to be learned rather than fixed (Hu, 11 Jan 2025). The SePS precursor already identified a core unresolved issue: static partitions learned from early trajectories may fail when agents later diverge in roles, which directly motivates the adaptive regrouping agenda that HAG-PS seeks to address (Christianos et al., 2021).

Taken together, the literature portrays HAG-PS not as a single canonical algorithm, but as a principled response to the central tension in large-scale cooperative learning: full sharing is memory-efficient but vulnerable to interference, while fully independent policies are expressive but computationally expensive. HAG-PS addresses that tension by layering sharing, specialization, and regrouping within one system.