HC-MAPPO-L: Hierarchical Constrained MAPPO

• The paper introduces a hierarchical multi-agent RL framework that uses adaptive Lagrangian relaxation to strictly enforce long-term delay constraints.
• HC-MAPPO-L decomposes the CMDP into three policy layers to jointly solve DNN deployment, user-server association, partitioning, and resource allocation in edge computing.
• Empirical results and theoretical analyses demonstrate its scalability and superior balance among delay, energy, and privacy costs under dynamic conditions.

Hierarchical Constrained Multi-Agent Proximal Policy Optimization with Lagrangian relaxation (HC-MAPPO-L) is a multi-agent deep reinforcement learning framework designed to address safe resource-constrained decision-making in privacy-aware collaborative DNN inference over edge-device and server systems. It operates by decomposing a high-dimensional, delay-constrained Constrained Markov Decision Process (CMDP) into three hierarchical policy layers and augments Multi-Agent Proximal Policy Optimization (MAPPO) with adaptive Lagrangian relaxation to provide strict enforcement of long-term average delay constraints. This paradigm enables the joint, coordinated solution of DNN model deployment, user-server association, model partitioning, and resource allocation, consistently yielding superior balance among delay, energy, and privacy cost under dynamic service and resource conditions (Wang et al., 23 Feb 2026).

1. CMDP Formulation and Objectives

The system dynamics, reward structures, and constraints in HC-MAPPO-L are formalized as a CMDP with global state and joint action spaces. At each decision slot $t$, the global state $S(t)$ comprises: deployment configuration $X(t)$, user-server associations $Y(t)$, partitioning decisions $Z(t)$, and fine-grained resource allocations $F(t)$ (computation) and $B(t)$ (bandwidth).

The instantaneous cost minimized by the system is

$$\mathrm{Cost}(t) = \mu_1\,\bar C^{\rm priv}(t) + \mu_2\,\bar e(t)$$

where $\bar C^{\rm priv}(t)$ denotes average per-user privacy cost and $\bar e(t)$ denotes average energy, with corresponding weights.

The only global constraint enforced is a long-term average delay bound, expressed as

$S(t)$0

where $S(t)$1 is the maximum tolerable average system-wide inference delay. Each user agent also locally enforces a stricter version involving only its own incurred delay.

Per-agent shaped rewards further decompose by layer:

• Deployment reward: $S(t)$2 combines cache hit and migration penalties.
• User reward: $S(t)$3 is negative weighted sum of privacy cost, energy, and excess delay, or a fixed failure cost.
• Allocation reward: $S(t)$4 is negative mean delay among server $S(t)$5's associated users.

Lagrangian relaxation introduces a nonnegative dual variable $S(t)$6 and defines the saddle-point optimization: $S(t)$7 allowing penalized policy optimization to balance reward and delay constraint via dual updates.

2. Hierarchical Policy Decomposition

HC-MAPPO-L decomposes the CMDP into three layered policies under the Centralized Training, Decentralized Execution (CTDE) paradigm. The layers are strictly top-down in coordination:

Layer	Role	Policy Structure
Deployment	Model placement on servers every $S(t)$8 slots	GRU-based auto-regressive macro-action
User Association/Partitioning	Dynamic user-server selection, DNN partition points	MLP with Lagrangian-penalized gradient
Resource Allocation	Per-user CPU/bandwidth assignment on servers	Attention-based allocation

Top Layer (Deployment): Each server $S(t)$9 observes recent traffic, host history, and deployment state. Actions are variable-length macro-actions encoded as auto-regressive GRU outputs, drastically reducing combinatorial complexity from $X(t)$0 to $X(t)$1 for $X(t)$2 models. Value baselines are per-step and aggregated across the sequence.

Middle Layer (User Association/Partitioning): Each user observes model and input embeddings and the full deployment configuration $X(t)$3. Actions select both the optimal server and DNN partition position. Constraint handling is carried out by inserting the Lagrange multiplier into the policy's gradient objective.

Bottom Layer (Resource Allocation): Each server computes per-user computation and bandwidth allocations using contextualized key-query attention across active users. This enables $X(t)$4 scalability (for $X(t)$5 users), and masking skips inactive/new users efficiently.

3. Adaptive Lagrangian Dual Optimization

HC-MAPPO-L tightly couples reward maximization with long-term constraint satisfaction through adaptive Lagrangian dual updates. After each batch of data collection, the empirical average discounted cost $X(t)$6 is updated and the dual variable $X(t)$7 is projected-ascended as: $X(t)$8 with $X(t)$9 as the dual learning rate. This process increases the penalty when constraints are violated and relaxes it otherwise, maintaining strict delay satisfaction.

4. Training Process and Algorithmic Modifications

The training algorithm proceeds in nested phases: data collection, advantage estimation, dual update, and policy/critic optimization. Key technical augmentations over standard MAPPO include:

• Auto-regressive deployment policy: GRU sequence modeling reduces macro-action search space.
• Dual-penalized user actor update: The user policy’s gradient includes both reward and constraint terms ($Y(t)$0).
• Cost-specific critic: A separate critic estimates $Y(t)$1 for accurate constraint handling.
• Generalized Advantage Estimation (GAE) for both cost and reward streams.
• Entropy regularization and PPO clipping are applied to all policy layers.

Critical hyperparameters include: $Y(t)$2, PPO clip $Y(t)$3, LR for actor/critic $Y(t)$4, deployment entropy coeff $Y(t)$5, user entropy coeff $Y(t)$6, dual LR $Y(t)$7, and deployment interval $Y(t)$8 slots. These were found to deliver robust convergence and strict delay compliance (≤3 s) across various task scales.

5. Complexity, Scalability, and Empirical Insights

HC-MAPPO-L’s hierarchical decomposition and architectural choices yield efficient scaling:

• Deployment layer: $Y(t)$9 (macro-action GRU) versus $Z(t)$0 (naive)
• User layer: $Z(t)$1 (per-step MLP)
• Allocation layer: Reduces to $Z(t)$2 per episode via masked attention

Overall training scales linearly in the number of servers, users, and models. CTDE ensures scalability to large user sets, while top-down coordination eliminates circular dependencies between layers. This structure enables efficient handling of diverse problem sizes and model library cardinalities.

Empirically, HC-MAPPO-L consistently satisfies stringent delay constraints while outperforming representative baselines on joint energy/privacy cost across varying resource and demand regimes. A plausible implication is that the hierarchical and constraint-aware design may remain robust as edge inference ecosystems grow in scope and heterogeneity.

6. Theoretical Properties and Limitations

Theoretical guarantees are provided for local convergence of the Lagrangian-PPO saddle-point optimization under conventional PPO assumptions of expressivity and sufficiently small learning rates. Global optimality is not formally established; empirical evaluations demonstrate practical stability and effective constraint enforcement.

The adoption of separate policy and cost critics, together with entropy regularization and dual-penalized gradients, distinguishes HC-MAPPO-L within constrained multi-agent RL for edge inference. However, limitations include reliance on accurate credit assignment for nested MARL hierarchies and the absence of formal worst-case guarantees on constraint infeasibility or global solution optimality.

7. Related Work and Future Directions

HC-MAPPO-L advances constrained multi-agent RL by integrating hierarchical decomposition with Lagrangian relaxation in privacy-aware edge computing. Key design elements—auto-regressive deployment, Lagrangian user updates, and scalable attention for resource assignment—address the tractability and coordination challenges of the edge inference domain.

Potential future dimensions include incorporating additional constraints (e.g., fairness, robustness), adaptive reward shaping for adversarial environments, or integrating model uncertainty quantification into the hierarchical control loop. A plausible implication is that generalizing hierarchical constrained RL frameworks to broader federated and resource-sharing contexts could significantly enhance safe decentralized AI decision-making under multi-faceted constraints (Wang et al., 23 Feb 2026).