Stackelberg Multi-Agent DDPG (ST-MADDPG)

• ST-MADDPG is a hierarchical multi-agent reinforcement learning framework that models interactions as a bi-level Stackelberg game with explicit leader-follower roles.
• It employs deterministic policy parameterization and centralized critics to couple agents’ policy updates, enhancing coordination and handling task asymmetry.
• Empirical studies show ST-MADDPG yields significant improvements over baseline methods in returns, safety metrics, and multi-objective performance in complex domains.

Stackelberg Multi-Agent Deep Deterministic Policy Gradient (ST-MADDPG) is a hierarchical extension of the canonical multi-agent actor-critic reinforcement learning paradigm, specifically the Multi-Agent Deep Deterministic Policy Gradient (MADDPG), adapted to multi-agent domains that exhibit agent asymmetry or inherent hierarchical interactions. In ST-MADDPG, agents are partitioned into leaders and followers, and their joint policy learning and optimization is explicitly modeled as a bi-level Stackelberg game. This formulation allows the leader to select its policy anticipating the best-response of the follower, enabling more sophisticated coordination, efficient handling of task asymmetry, and, in various settings, improvement over Nash equilibria achieved by simultaneous-agent updates.

1. Mathematical Formulation of Stackelberg Markov Games in ST-MADDPG

ST-MADDPG addresses two-player or multi-player Markov games $(S, \{A_i\}_{i=1}^N, P, \{r_i\}_{i=1}^N, \gamma)$, imposing a strict leader-follower structure. The standard bi-level objective is as follows for the two-agent (leader/follower) case (Yang et al., 2023, Zhang et al., 2019):

• The leader policy $\pi_1(\cdot\,|\,s; \theta_1)$ and the follower policy $\pi_2(\cdot\,|\,s; \theta_2)$ are deterministic mappings, with the joint transition $s' \sim P(s, a_1, a_2)$ and rewards $r_1(s, a_1, a_2), r_2(s, a_1, a_2)$. The optimization proceeds via bi-level nesting:

$$\begin{aligned} &\text{Follower: } \theta_2^*(\theta_1) = \arg\max_{\theta_2} J_2(\theta_1, \theta_2) \ &\text{Leader: } \theta_1^* = \arg\max_{\theta_1} J_1(\theta_1, \theta_2^*(\theta_1)) \end{aligned}$$

where $J_i(\theta_1, \theta_2)$ is the expected cumulative reward for agent $i$. This formalism induces a Stackelberg equilibrium characterized by the leader's anticipation of the follower's best-response.

• The objective can be extended to incorporate constraints (e.g., safety, resource budgets) via Lagrangian relaxation, yielding augmented Lagrangian objectives at both the leader and follower levels (Zheng et al., 2024).

2. Deterministic Policy Parameterization and Critic Architecture

ST-MADDPG parameterizes leader and follower with deterministic actors, denoted as:

• Leader: $\mu_1(s;\theta_1): S \to A_1$ (or $\mu_1(o_1;\theta_1)$ for local obs),
• Follower: $\pi_1(\cdot\,|\,s; \theta_1)$0 (enabling explicit conditioning on leader action).

Centralized critics are trained to estimate joint action-values:

• $\pi_1(\cdot\,|\,s; \theta_1)$1 for the leader,
• $\pi_1(\cdot\,|\,s; \theta_1)$2 for the follower.

Augmented critics for costs or additional objectives are used in constrained or multi-objective domains (Zheng et al., 2024, Hayla et al., 26 Feb 2025).

Network architectures are typically multi-layer perceptrons, with hidden layers of 128–512 units and ReLU nonlinearities, final actor outputs constrained (e.g., by $\pi_1(\cdot\,|\,s; \theta_1)$3) to satisfy action bounds (Yang et al., 2023, Zhang et al., 2019).

3. Bi-Level Gradient Computation and Policy Update Mechanics

The core algorithmic distinction in ST-MADDPG arises in the leader’s gradient computation:

• The follower updates its policy by standard DPG:

$\pi_1(\cdot\,|\,s; \theta_1)$4

where $\pi_1(\cdot\,|\,s; \theta_1)$5, $\pi_1(\cdot\,|\,s; \theta_1)$6.

• The leader's update embeds anticipation of the follower's best-response:

$\pi_1(\cdot\,|\,s; \theta_1)$7

This total-derivative Stackelberg correction uses the follower’s Jacobian to reflect how changes in the leader policy indirectly drive changes in the follower’s best-response. In practice, Hessian-vector products required for the correction are computed via conjugate gradients and regularized by $\pi_1(\cdot\,|\,s; \theta_1)$8 for numerical stability (Yang et al., 2023, Zhang et al., 2019).

• In constrained and multi-objective domains, policy and multiplier updates (for Lagrange multipliers) are performed on different timescales, with the inner follower loop run until near-convergence to ensure the leader's update follows the correct bi-level structure (Zheng et al., 2024, Zhang et al., 2019).

4. Extensions: Multi-Objective and Hierarchical Stackelberg Structures

In complex domains with layered infrastructure or conflicting goals (e.g., vehicular metaverse scenarios), ST-MADDPG is embedded into hierarchical Stackelberg games and augmented to handle multiple, possibly non-aligned objectives (Hayla et al., 26 Feb 2025). In such settings:

• The environment is modeled as an MDP-overlaid Stackelberg game with multiple leader-follower stages (e.g., cloud–edge–vehicle), each level solving best-responses with respect to resource quotas, pricing, or migration strategies.
• ST-MADDPG policies are trained using GCN-encoded representations of spatial-temporal features, supporting scalable and topology-invariant operation.

Multi-objective rewards are formed as

$\pi_1(\cdot\,|\,s; \theta_1)$9

where each term corresponds to critical performance or cost metrics (Hayla et al., 26 Feb 2025).

5. Empirical Results and Performance Benchmarks

Experimental evaluations across domains yield:

• In competitive robotics and autocurricula, ST-MADDPG consistently produces policies that achieve higher mean returns and greater robustness in the face of task asymmetry as compared to symmetric MADDPG, with leader agents exploiting their hierarchy-induced advantage (Yang et al., 2023). In adversarial Hopper, ST-MADDPG reaches approximately $\pi_2(\cdot\,|\,s; \theta_2)$0 improvement in returns over MADDPG under adversarial attacks.
• For safe multi-agent RL in autonomous driving, the constrained Stackelberg extension (CS-MADDPG) achieves higher safety rates (up to $\pi_2(\cdot\,|\,s; \theta_2)$1 in intersection, $\pi_2(\cdot\,|\,s; \theta_2)$2–$\pi_2(\cdot\,|\,s; \theta_2)$3 in racetrack, compared to $\pi_2(\cdot\,|\,s; \theta_2)$4–$\pi_2(\cdot\,|\,s; \theta_2)$5 for baselines), and higher total rewards (Zheng et al., 2024).
• In multi-objective vehicular metaverse optimization, ST-MADDPG achieves $\pi_2(\cdot\,|\,s; \theta_2)$6 latency reduction, $\pi_2(\cdot\,|\,s; \theta_2)$7 improved resource utilization, $\pi_2(\cdot\,|\,s; \theta_2)$8 lower migration cost, and $\pi_2(\cdot\,|\,s; \theta_2)$9 better user experience versus baseline agents (Hayla et al., 26 Feb 2025).

Ablation studies consistently show the importance of each algorithmic innovation: omitting Stackelberg incentives or multi-objective weighting substantially degrades performance (Hayla et al., 26 Feb 2025).

6. Theoretical Analysis and Convergence Guarantees

Convergence analysis indicates:

• For tabular/CS-Q-learning variants, the induced Bellman operator under Stackelberg equilibrium assumptions is a contraction mapping, implying almost sure convergence to the unique fixed point via stochastic approximation theory (Zheng et al., 2024, Zhang et al., 2019).
• For deep function approximation settings, local convergence to a differential Stackelberg equilibrium is guaranteed under standard smoothness, sufficient exploration, and time-scale separation assumptions (Yang et al., 2023).
• The Stackelberg update structure ensures the leader’s equilibrium payoff is never worse than, and usually strictly better than, that attainable under simultaneous-play Nash equilibria in zero-sum settings with unique best-responses (Yang et al., 2023).

7. Comparisons, Limitations, and Extensions

ST-MADDPG differs fundamentally from MADDPG by:

• Imposing a hierarchical, sequential policy update (leader then follower) versus symmetric, simultaneous updates.
• Incorporating Stackelberg gradients (i.e., leader updates that account for anticipated follower response) rather than simple local policy gradients.
• In constrained and multi-objective domains, using bi-level optimization with explicit Lagrangian multipliers and cost critics, enabling constraint satisfaction and safety guarantees (Zheng et al., 2024).

Empirical findings suggest Stackelberg hierarchy may alleviate equilibrium selection pathologies and improve robustness and exploration in the presence of agent asymmetry (Yang et al., 2023, Zhang et al., 2019). Extensions under study include stochastic multi-leader/follower games, POMDP adaptation via deep generative modeling, and real-world field validation in vehicular networks (Hayla et al., 26 Feb 2025).

Variant	Stackelberg Hierarchy	Constraint Handling	Multi-Objective
MADDPG	No	No	Limited
ST-MADDPG	Yes	No	Possible
CS-MADDPG (Safe-MARL)	Yes	Yes	Yes
ST-MADDPG (Veh. Meta.)	Yes (hierarchical)	Yes	Yes

ST-MADDPG provides a unifying framework for leader–follower coordination in continuous multi-agent domains, exploiting explicit Stackelberg reasoning to model, anticipate, and shape complex agent interactions across a range of safety-critical and resource-constrained applications.