Stackelberg Markov Game: Dynamics & Equilibria

• SMG is a dynamic leader–follower game defined by sequential commitment and Markov policies, replacing Nash equilibria with a Stackelberg equilibrium notion.
• SMG methodologies employ contraction mappings, convex–concave structures, and potential game reductions to ensure tractability and convergence of learning algorithms.
• SMGs find applications in energy markets, security, and adversarial learning, while also presenting challenges in partial observability and multi-leader extensions.

A Stackelberg Markov Game (SMG) is a dynamic leader–follower game in which state transitions are Markovian and the leader commits to a policy while anticipating followers’ best responses. In the single-leader single-follower discounted formulation, an SMG is specified by

$$\mathcal{G}_S := (\mathcal{S}, \mathcal{A}_L, \mathcal{A}_F, P, r_L, r_F, \gamma),$$

with stationary Markov policies $\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$ and $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$, and value functions defined by discounted returns under the induced trajectory law. Standard Markov games consider simultaneous play with Nash equilibria, whereas SMGs impose sequential commitment and replace Nash with a Stackelberg equilibrium notion under dynamic state coupling; the same idea also appears in continuous-state continuous-action, Bayesian, mean-field, private-information, and continuous-time formulations (He et al., 19 Sep 2025, Goktas et al., 2024).

1. Canonical model and mathematical structure

In the canonical discounted formulation, the state space $\mathcal{S}$ is measurable, the leader and follower action spaces $\mathcal{A}_L,\mathcal{A}_F$ are finite, $P(\cdot\mid s,a_L,a_F)$ is the stochastic transition kernel, and $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$ are one-step rewards. The leader and follower evaluate stationary Markov policies through

$$V_L^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_L^t\,r_L(s_t,a_t^L,a_t^F)\Big],\qquad V_F^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_F^t\,r_F(s_t,a_t^L,a_t^F)\Big].$$

A standing regularity condition in this setting is that rewards and transition kernel are continuous in $(s,a_L,a_F)$ and uniformly bounded (He et al., 19 Sep 2025).

A second major formulation treats zero-sum Markov Stackelberg games with continuous state and action spaces. There the game is

$\game \doteq (\states,\outeractions,\inneractions,\initstates,\reward,\constr,\trans,\discount),$

the leader acts first at each state, and the follower responds from a state- and leader-dependent feasible set

$\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$0

This coupled-constraint view makes the one-shot Stackelberg problem

$\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$1

the dynamic analogue of a constrained min–max program, with the stochastic dynamics absorbed into the cumulative payoff and state-visitation distribution (Goktas et al., 2024).

Across formulations, the defining asymmetry is commitment. In standard Markov games, players optimize simultaneously; in SMGs, the leader chooses a policy first and the follower optimizes conditionally on that commitment. This asymmetry survives under several notational variants. The spatial-temporal sequential Markov game (STMG) factorizes the joint policy as

$\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$2

thereby encoding a leader–follower hierarchy within each time step rather than only at the intertemporal level (Zhang et al., 2023).

2. Equilibrium concepts

The basic equilibrium notion in the discounted single-leader single-follower setting is the stationary Stackelberg equilibrium (SSE). For a fixed leader policy $\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$3, the follower computes a stationary best response

$\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$4

and the leader then solves

$\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$5

The paper enforcing this definition uses uniqueness of best responses to avoid ambiguity; this corresponds to an optimistic, leader-favorable selection, whereas the pessimistic case introduces set-valued discontinuities and is left for future work (He et al., 19 Sep 2025).

In convex–concave zero-sum models with coupled constraints, the equilibrium notion is formulated recursively. A policy profile is an $\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$6-Stackelberg equilibrium if the follower is $\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$7-optimal under the leader’s policy, the coupled constraints are satisfied up to $\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$8, and the leader is within $\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)$9 of the min–max value. This differs from generalized Nash equilibrium: in coupled-constraint settings, a generalized Nash equilibrium need not coincide with Stackelberg equilibrium, even though saddle points and Nash equilibria coincide in zero-sum games without coupling (Goktas et al., 2024).

Incomplete-information settings replace a single follower by a type-indexed family of followers. In Bayesian Stackelberg Markov games, the attacker or follower has type $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$0, each type plays a deterministic best response to the leader’s Markov stationary mixed strategy, and the leader maximizes expected value under a belief $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$1 over types: $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$2 The equilibrium is a Bayesian Strong Stackelberg Equilibrium, again with tie-breaking in favor of the leader (Sengupta et al., 2020).

Private-information versions modify the policy class itself. In smart-grid demand response, users’ storage levels are private information, so the lower-level game is defined over private Markovian strategies (PMS), where user $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$3’s policy depends only on the public renewable state and its own private storage state. The corresponding equilibrium notion is private Markovian equilibrium (PME), under which no user can improve by unilateral deviation from its PMS (Huang et al., 6 Sep 2025).

Mean-field variants extend the equilibrium object from a policy pair to a policy–distribution triple. In the stationary Stackelberg mean-field equilibrium (SS-MFE), the follower must be optimal given the leader policy and the mean field, the mean field must satisfy a consistency condition $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$4, and the leader must be optimal given the induced follower policy and mean field (He et al., 19 Sep 2025).

3. Existence, uniqueness, and structurally tractable subclasses

For the single-leader single-follower discounted model, existence and uniqueness of SSE follow from a contraction argument. If rewards and transitions satisfy continuity and boundedness, best responses are unique, the best-response maps are Lipschitz,

$\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$5

and $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$6, then the composition $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$7 is a contraction on the leader policy space, so Banach’s fixed-point theorem yields a unique stationary Stackelberg equilibrium (He et al., 19 Sep 2025).

The mean-field extension requires a nested contraction argument. With a continuum of followers, the paper introduces Lipschitz constants for the follower best response, the mean-field update $\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$8, and the leader best response. A unique stationary SS-MFE exists if

$\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)$9

The first inequality makes the inner mean-field loop contractive, and the second makes the leader-facing composite map contractive (He et al., 19 Sep 2025).

A different tractable class arises from convex–concave structure. In continuous-state zero-sum SMGs, if the objective is convex in the leader parameters and concave in the follower parameters, the coupled feasible correspondence is concave and convex-valued, Slater’s condition holds, and unbiased stochastic first-order oracles with bounded variance are available, then nested SGDA and saddle-point-oracle SGD converge to Stackelberg equilibrium in polynomial time. The paper further supplies sufficient conditions for convexity and concavity through stochastic convexity or stochastic concavity of transitions and convexity or concavity of rewards (Goktas et al., 2024).

Another tractable subclass is the private-state demand-response game. There, the lower-level one-stage game becomes an exact potential game after eliminating dominated actions in consumption, and the finite-horizon lower-level Markov game admits a pure PME. The PME can be computed in polynomial time by converting the lower-level game into an auxiliary Markov potential game and solving for a pure Markovian equilibrium through potential maximization. The paper emphasizes that computing equilibrium in general Markov games is hard, and polynomial-time algorithms are rarely available (Huang et al., 6 Sep 2025).

These results identify three distinct sources of tractability: contraction in best-response space, convex–concave min–max structure, and potential-game reductions. They also delineate where tractability ends. Outside these structured classes, several papers explicitly leave pessimistic Stackelberg selection, partial observability, non-stationary commitments, continuous-space extensions, or multi-leader competition as open problems (He et al., 19 Sep 2025, Goktas et al., 2024).

4. Computation and learning algorithms

A central computational difficulty in SMGs is that exact best-response dynamics are often intractable. One remedy is softmax smoothing. In the discounted single-leader single-follower setting, the follower best response is approximated by a Boltzmann policy

$\mathcal{S}$0

and more generally

$\mathcal{S}$1

The softmax map is Lipschitz, and when the action gap $\mathcal{S}$2 is bounded below,

$\mathcal{S}$3

Using an $\mathcal{S}$4-net and projection,

$\mathcal{S}$5

the iterates converge to within an explicit $\mathcal{S}$6-dependent bound of the true SSE when $\mathcal{S}$7 (He et al., 19 Sep 2025).

The same work gives a three-step reinforcement-learning framework. First, fix $\mathcal{S}$8 and learn the follower’s best response $\mathcal{S}$9. Second, fix $\mathcal{A}_L,\mathcal{A}_F$0 and learn the leader’s best response $\mathcal{A}_L,\mathcal{A}_F$1. Third, iterate until $\mathcal{A}_L,\mathcal{A}_F$2. Softmax smoothing, $\mathcal{A}_L,\mathcal{A}_F$3-net projection, and optional entropy regularization are used to stabilize learning, and each best-response subproblem can be solved with off-the-shelf RL such as PPO (He et al., 19 Sep 2025).

In convex–concave zero-sum SMGs, learning can be phrased as nested stochastic gradient descent–ascent. The inner loop solves a Lagrangian saddle-point problem over follower parameters and multipliers, and the outer loop performs projected SGD on the leader parameters. With stepsizes in $\mathcal{A}_L,\mathcal{A}_F$4, the averaged iterate is an $\mathcal{A}_L,\mathcal{A}_F$5-Stackelberg equilibrium after $\mathcal{A}_L,\mathcal{A}_F$6 stochastic gradient evaluations for nested SGDA, or $\mathcal{A}_L,\mathcal{A}_F$7 with a saddle-point oracle; if the leader value is strongly convex, these rates improve to $\mathcal{A}_L,\mathcal{A}_F$8 and $\mathcal{A}_L,\mathcal{A}_F$9. The same paper extends the argument to trajectory-based policy gradients in continuous-state continuous-action SMGs (Goktas et al., 2024).

Under incomplete information, Bayesian Strong Stackelberg Q-learning embeds a Bayesian stage-game solve inside Q-learning updates. Separate Q-tables are maintained for the leader and each follower type, and the Bellman target uses the Bayesian SSE value at the next state. Under bounded rewards, $P(\cdot\mid s,a_L,a_F)$0, sufficient exploration, and Robbins–Monro stepsizes, the iterates converge almost surely to a fixed point of the SSE Bellman operator, and the induced policy profile converges to a Bayesian SSE (Sengupta et al., 2020).

When the follower utility is unknown but linearly parameterized, the dynamic Stackelberg formulation yields a no-regret learning problem. The leader maintains a confidence set over the unknown parameter $P(\cdot\mid s,a_L,a_F)$1, updates it from observed follower best responses, and computes optimistic $P(\cdot\mid s,a_L,a_F)$2-conservative policies by backward induction. With probability at least $P(\cdot\mid s,a_L,a_F)$3,

$P(\cdot\mid s,a_L,a_F)$4

for an explicit polynomial constant $P(\cdot\mid s,a_L,a_F)$5, and the regret is independent of the size of the state space (Lauffer et al., 2022).

In partially observed security settings with unknown dynamics, model-free Expected Sarsa can be applied on a belief-state MDP produced by the common-information approach. Particle filters approximate the Bayes belief update, and the follower best response is recomputed at each belief state, yielding an $P(\cdot\mid s,a_L,a_F)$6-Markov perfect Stackelberg equilibrium under filter accuracy and learning assumptions (Mishra et al., 2020).

5. Structural extensions

Mean-field structure is the most prominent extension. In the single-leader case, the leader interacts with an infinite population of followers whose state distribution $P(\cdot\mid s,a_L,a_F)$7 evolves endogenously. The discrete-time master-equation approach constructs equilibrium-generating functions $P(\cdot\mid s,a_L,a_F)$8, where $P(\cdot\mid s,a_L,a_F)$9 is the common belief on the leader’s type and $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$0 is the mean field. Backward recursion yields follower and leader value functions $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$1, and forward recursion updates both the belief and the mean field through maps $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$2 and $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$3, thereby computing all Stackelberg mean-field equilibria (Vasal et al., 2022). The same architecture has been extended to multiple leaders, finite major followers, and an infinite number of minor followers; the resulting Stackelberg mean-field equilibrium with multiple leaders (SMFE-ML) is characterized by a discrete-time master equation over common beliefs and mean fields (Vasal, 2022).

Private information can be placed in either the leader or the followers. In stochastic Stackelberg games with asymmetric information, a common-agent representation maps common information into prescriptions for both players, reducing the dynamic game to a belief-state recursion. The backward algorithm computes per-time fixed points rather than a global fixed point over complete histories (Vasal, 2020). In Bayesian Stackelberg Markov games, follower types are encoded by a prior and possibly updated through observations, which situates SMGs within the broader landscape of incomplete-information stochastic games while preserving the leader’s commitment structure (Sengupta et al., 2020). In smart-grid demand response, the private-state formulation goes further: followers observe the public renewable state and their own storage levels, whereas the leader observes only aggregate demand (Huang et al., 6 Sep 2025).

Several papers move from discrete-time stochastic games to continuous-time or semi-Markov dynamics while retaining the leader–follower semantics. The online educational forum model represents the interaction as a continuous-time Markov chain whose generator depends on instructor and student arrival-rate actions, and solves the resulting single-leader multiple-followers Stackelberg problem through a mixed-integer linear program (Vallam et al., 2021). Spatial-temporal moving target defense uses a semi-Markov Stackelberg game in which the defender chooses both migration probabilities and defending-period lengths, and computes $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$4-optimal stationary strategies via value iteration or relative value iteration combined with MIQP subproblems (Li et al., 2020). Zero-sum stochastic linear-quadratic Stackelberg differential games with Markovian regime switching derive regime-dependent feedback laws from coupled differential Riccati equations, while linear-quadratic mean-field stochastic Stackelberg differential games with random exit time use a two-stage decomposition driven by the stopping time (Wu et al., 2024, Gou et al., 2021).

Another extension concerns the action order itself. STMG treats the joint policy as a sequential conditional factorization within each time step, allowing asymmetric training with symmetric execution in multi-agent reinforcement learning (Zhang et al., 2023). A later analysis shows that, in an $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$5-level Stackelberg game, changing the decision order typically yields an overdetermined system, so the equilibrium point shifts unless special structural conditions hold; this motivates hierarchical priority adjustment, where an upper policy chooses the order and a lower STMG executes under that order (Liu et al., 8 May 2026).

6. Applications, empirical evidence, and recurrent limitations

Energy-market policy design is a leading application. In the single-leader mean-field SMG for electricity tariffs, the leader is a utility or state commission setting time-varying retail tariffs, and the followers are heterogeneous prosumers and consumers grouped by aggregators at three nodes in a 3-bus transmission system with 4 generators and 3 lines. The leader pursues economic efficiency, equity across income groups, and grid stability; equity is measured by Energy Expenditure Incidence (EEI), and stability by Incremental Mean Volatility,

$$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$6

Using PPO for both leader and followers over a 100-day simulation, learned policies reduce IMV by roughly 3 units relative to a baseline without storage or RL, reduce the EEI gap between prosumers and consumers from about $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$7 to about $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$8 of income, converge to higher fixed charges for higher-income groups, and reshape charging to midday with net injections during evening peaks (He et al., 19 Sep 2025).

Security is another major domain. Reach–avoid problems are modeled as convex–concave zero-sum SMGs in which the leader acts first, the follower reacts under safety constraints, and nested SGDA or saddle-point-oracle gradient methods converge in polynomial time to recursive Stackelberg equilibrium policies. In experiments with Dubins-car dynamics, Stackelberg-trained protagonists achieve higher win rates and better safety–liveness trade-offs than simultaneous generalized-Nash baselines (Goktas et al., 2024). In moving target defense, Bayesian Strong Stackelberg Q-learning improves the state of the art for web-application security and converges to optimal movement policies in domains with incomplete information about adversaries, while the spatial-temporal Markov Stackelberg formulation yields lower long-run average costs than Bayesian Stackelberg and uniform-random baselines under source-destination-dependent switching costs and random exploitation times (Sengupta et al., 2020, Li et al., 2020).

Adversarial federated learning introduces a further Bayesian SMG variant. The defense problem is formulated under a mixture of uncertain, unknown, and adaptive poisoning attacks, and the proposed meta-Stackelberg defense combines pre-training and online adaptation. Under smoothness, strict competitiveness, and a Stackelberg Polyak–Łojasiewicz condition, meta-Stackelberg learning converges to a first-order $$r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}$$9-meta-equilibrium in $$V_L^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_L^t\,r_L(s_t,a_t^L,a_t^F)\Big],\qquad V_F^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_F^t\,r_F(s_t,a_t^L,a_t^F)\Big].$$0 outer iterations with $$V_L^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_L^t\,r_L(s_t,a_t^L,a_t^F)\Big],\qquad V_F^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_F^t\,r_F(s_t,a_t^L,a_t^F)\Big].$$1 samples per iteration and $$V_L^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_L^t\,r_L(s_t,a_t^L,a_t^F)\Big],\qquad V_F^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_F^t\,r_F(s_t,a_t^L,a_t^F)\Big].$$2 inner steps; empirically it is robust against mixed poisoning and backdoor attacks (Li et al., 2024).

Applications also appear outside security and energy. In online educational forums, a CTMC Stackelberg Markov model reproduces effective incentive design, student heterogeneity, non-monotonic participation with increasing instructor involvement, and the super-poster phenomenon, while delivering an MILP-based optimal instructor plan (Vallam et al., 2021). In smart-grid demand response with up to 50 users, the PME framework scales beyond the small-user regimes emphasized in prior studies (Huang et al., 6 Sep 2025).

Across these formulations, several limitations recur. Many analyses focus on stationary policies and stationary equilibria rather than history-dependent subgame-perfect policies; optimistic tie-breaking is often used to enforce unique best responses; full observability is typically assumed; single-leader settings dominate; continuous state or action spaces frequently require additional measure-theoretic or convex–concave structure; and partial observability, multi-leader competition, robust or pessimistic Stackelberg selection, off-policy learning, and adaptive or dynamically chosen decision orders remain active extensions rather than resolved theory (He et al., 19 Sep 2025, Goktas et al., 2024, Liu et al., 8 May 2026). These repeated caveats suggest that SMGs are best viewed not as a single equilibrium model, but as a family of leader–follower Markovian decision frameworks whose tractability depends sharply on informational structure, population scale, and the analytical regularity of the best-response mapping.