Stackelberg Dynamics in Multi-Agent Games

• Stackelberg dynamics are sequential game-theoretic models where leaders act first and followers respond optimally, establishing a hierarchical leader–follower structure.
• They recast multi-stage decision-making into a unified constrained optimization problem solved via methods like primal–dual interior point and Newton iterations.
• This framework has critical applications in economics, control, and learning environments, offering computational strategies for approximating local equilibria in complex settings.

A Stackelberg dynamic is a sequential game-theoretic structure in which one set of players (leaders) acts first, with subsequent sets of players (followers) optimally responding given the leader’s actions. In dynamic games, the interaction unfolds over time and is typically governed by complex system dynamics (deterministic or stochastic), possibly with state and control constraints and asymmetric information. Stackelberg dynamics have deep applications across economics, control, operations, and learning-theoretic environments. Rigorous computation of Stackelberg equilibria in nonlinear, constrained, multi-stage settings has been a longstanding challenge due to the hierarchical and coupled nature of the optimization tasks.

1. Mathematical Structure of Feedback Stackelberg Dynamic Games

Consider a finite-horizon, discrete-time dynamic game with $N$ players, continuous state space $\mathbb{R}^n$, and continuous control space $\mathbb{R}^m$. At each stage $t=0,\dots,T$, the system evolves as

$x_{t+1} = f_t(x_t, u_t),\quad x_0 \text{ given},$

with $u_t = (u_t^1,\ldots,u_t^N)$ each player’s action at $t$, and $$f_t: \mathbb{R}^n \times \mathbb{R}^m \to \mathbb{R}^n$$ being $\mathcal{C}^2$. Each player $i$ has stage cost $\ell_t^i(x_t,u_t)$, terminal cost $\ell^i_{T+1}(x_{T+1})$, equality constraints $h_t^i(x_t,u_t)=0$, and inequality constraints $g_t^i(x_t,u_t)\geq 0$. The global objective is to compute a local feedback Stackelberg equilibrium (FSE): a trajectory and set of feedback policies $(x^*, u^*)$, $\pi_t^i(\cdot)$ such that at each $t$ and for each $i$, $(x^*,u^*)$ is locally optimal under the Stackelberg leader–follower hierarchy.

The leader’s problem at each stage appears as a nested bilevel optimization, with follower optimal responses explicitly encoded as constraints. Instead of literal nesting (which leads to infeasible recursion in the general nonlinear case), all followers' best-response policies $\pi_t^j$ are included in the leader’s problem as algebraic constraints:

$$u_t^j - \pi_t^j(x_t, u_t^{1:j-1}) = 0,\quad \forall j > i,$$

and similarly into future stages for the "unrolled" Stackelberg hierarchy.

This "folded" representation recasts the entire hierarchical dynamic game as a single, large, constrained optimization problem in all states, controls, and feedback laws, suitable for KKT-based characterizations.

2. KKT System, Second-Order Conditions, and Structure

For the constrained linear-quadratic (LQ) approximation, where

$x_{t+1} = A_t x_t + \sum_{i} B_t^i u_t^i + c_t,$

and quadratic costs/linear constraints, the Lagrangian for each player $i$ includes Lagrange multipliers for

• dynamics ($\lambda_t$)
• equality constraints ($\mu_t^i$)
• inequality constraints ($\nu_t^i$)
• leader–follower reaction constraints ($\psi_t^{i \rightarrow j}$)

The KKT system consists of:

• Stationarity: $\nabla_{x_t, u_t} L^i = 0$ for all $i, t$,
• Primal feasibility: state transitions, equality/inequality constraints, reaction constraints,
• Dual feasibility: $\nu_t^i \geq 0$,
• Complementarity: $\nu_t^{i\top} g_t^i(x_t, u_t) = 0$.

The global system is a large, sparse block-structured linear-complementarity problem. Strict local optimality (local FSE) requires the Hessian of the global Lagrangian to be positive definite on the tangent space to the set of active constraints (critical cone).

3. Primal–Dual Interior-Point and Newton Methods for LQ Subproblems

For the constrained LQ subproblem, complementarity is enforced through a log-barrier relaxation

$\nu_t^i g_t^i = \mu > 0$

with homotopy parameter $\mu \rightarrow 0$. Defining the full residual $K_\mu(z)=0$ ($z$ being all primal/dual variables), a damped Newton iteration is employed:

$$\Delta z = -[\nabla_z K_\mu(z)]^{-1} K_\mu(z), \quad z \leftarrow z + \alpha \Delta z,$$

with $\alpha$ determined by line search on a merit function to maintain strict feasibility.

Convergence of this PDIP–Newton method is established under:

• LICQ (linear independence constraint qualification)
• Nongeneracy (uniform bound on $[\nabla K_\mu(z)]^{-1}$)
• Lipschitz gradient properties

Locally, convergence is quadratic in $z$, superlinear for each fixed $\mu$, and geometric in $\mu$, i.e.,

$$\| K_\mu(z_{k+1}) \| \leq \rho\, \| K_\mu(z_k)\|,\quad \rho < 1.$$

4. Successive LQ Linearization for Nonlinear Games

For general nonlinear dynamics and costs, an outer iterative scheme is adopted:

1. Linearize dynamics and constraints at the current guess $z^k$.
2. Quadraticize all players’ Lagrangians at $z^k$, yielding a sequence of LQ Stackelberg games.
3. Solve each LQ game by the above PDIP–Newton scheme.
4. Update to $z^{k+1}$, ensuring the first-order (KKT) system for the LQ subproblem matches the first-order Taylor expansion at $z^k$—termed KKT-jet alignment.

Under twice-differentiability, LICQ, strong second-order, and boundedness assumptions, this procedure converges exponentially:

$$\exists\, \rho \in (0,1),\; k_0: \forall k \geq k_0, \quad \| K_\mu(z^{k+1}) \| \leq \rho\, \| K_\mu(z^k) \|$$

as $\mu \to 0$, recovering a (local) solution to the full nonlinear KKT system, i.e., an approximate FSE.

5. Implementation Complexity, Initialization, and Limitations

• Complexity: Each Newton step requires solving a sparse linear system of size $O((n + m + \text{\#multipliers}) \cdot T)$. The worst-case scaling is $O(T(Nn + Nm)^3)$, but actual performance is much better with structured solvers (block Gaussian elimination, etc.).
• Initialization: The PDIP method is robust to infeasible starts. The log-barrier ensures that $g_t^i>0$, and feasibility is enforced asymptotically.
• Limitations: The method is local (not global): convergence is to a local FSE defined by the initial guess and regularity conditions. Nonconvexity in the original game means the method provides an approximate (not global) Stackelberg equilibrium, and success depends on strict complementarity and suitable initialization.

Component	Mathematical Object	Complexity / Considerations
Outer iteration	LQ-linearizations & PDIP	Each iteration $O(T(Nn+Nm)^3)$ (worst-case)
Newton step (inner loop)	Linear-complementarity KKT	Sparsity structure exploited
Initialization	Arbitrary ($g_t^i>0$)	Feasibility achieved asymptotically

6. Synthesis and Practical Application Recipe

The feedback Stackelberg solution approach consists of:

• Formulate the feedback Stackelberg dynamic game as a high-dimensional, constrained optimization problem by embedding the hierarchical (leader–follower) structure as explicit algebraic constraints.
• Characterize local equilibria by writing the coupled KKT system, including the reaction constraints that encode best-response mappings.
• Solve the resulting large-scale linear-complementarity system by a primal–dual interior-point Newton-type method, with log-barrier homotopy to enforce inequality constraints and complementarity.
• Integrate this solver in an outer loop that successively LQ-approximates the nonlinear game (i.e., intelligent sequential quadratic programming), guaranteeing that KKT systems align iteratively with the global nonlinear structure.
• Under standard regularity conditions, obtain exponential-rate local convergence to an approximate feedback Stackelberg equilibrium.

This method achieves strong performance in numerically challenging settings (multiple players, nonconvex constraints, infeasible initial conditions), offering a computationally viable and theoretically grounded route for computing local FSEs in multi-player, constrained, continuous-state-action dynamic games. The main restriction is locality: only local convergence is guaranteed, and structural nonconvexity precludes global optimality guarantees. Nonetheless, this approach represents a significant advance in the practical computation of Stackelberg dynamic equilibria in complex nonlinear domains (Li et al., 28 Jan 2024).