---
title: 'Stackelberg Markov Game: Dynamics & Equilibria'
url: https://www.emergentmind.com/topics/stackelberg-markov-game-smg
type: topic
---

# Stackelberg Markov Game: Dynamics & Equilibria

A Stackelberg Markov Game (SMG) is a dynamic leader–follower game in which state transitions are Markovian and the leader commits to a policy while anticipating followers’ best responses. In the single-leader single-follower discounted formulation, an SMG is specified by
\[
\mathcal{G}_S := (\mathcal{S}, \mathcal{A}_L, \mathcal{A}_F, P, r_L, r_F, \gamma),
\]
with stationary Markov policies \(\pi_L:\mathcal{S}\to\Delta(\mathcal{A}_L)\) and \(\pi_F:\mathcal{S}\to\Delta(\mathcal{A}_F)\), and value functions defined by discounted returns under the induced trajectory law. Standard Markov games consider simultaneous play with Nash equilibria, whereas SMGs impose sequential commitment and replace Nash with a Stackelberg equilibrium notion under dynamic state coupling; the same idea also appears in continuous-state continuous-action, Bayesian, mean-field, private-information, and continuous-time formulations [2509.16296] [2401.12437].

## 1. Canonical model and mathematical structure

In the canonical discounted formulation, the state space \(\mathcal{S}\) is measurable, the leader and follower action spaces \(\mathcal{A}_L,\mathcal{A}_F\) are finite, \(P(\cdot\mid s,a_L,a_F)\) is the stochastic transition kernel, and \(r_L,r_F:\mathcal{S}\times\mathcal{A}_L\times\mathcal{A}_F\to\mathbb{R}\) are one-step rewards. The leader and follower evaluate stationary Markov policies through
\[
V_L^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_L^t\,r_L(s_t,a_t^L,a_t^F)\Big],\qquad
V_F^{\pi_L,\pi_F}(s)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma_F^t\,r_F(s_t,a_t^L,a_t^F)\Big].
\]
A standing regularity condition in this setting is that rewards and transition kernel are continuous in \((s,a_L,a_F)\) and uniformly bounded [2509.16296].

A second major formulation treats zero-sum Markov Stackelberg games with continuous state and action spaces. There the game is
\[
\game \doteq (\states,\outeractions,\inneractions,\initstates,\reward,\constr,\trans,\discount),
\]
the leader acts first at each state, and the follower responds from a state- and leader-dependent feasible set
\[
\coupledactions(\state,\outeraction)\doteq \{\,\inneraction\in \inneractions \mid \constr(\state,\outeraction,\inneraction)\ge 0\,\}.
\]
This coupled-constraint view makes the one-shot Stackelberg problem
\[
\min_{\outer\in\outerset}\ \max_{\inner\in \coupledactions(\outer)}\ \cumulutil(\outer,\inner)
\]
the dynamic analogue of a constrained min–max program, with the stochastic dynamics absorbed into the cumulative payoff and state-visitation distribution [2401.12437].

Across formulations, the defining asymmetry is commitment. In standard Markov games, players optimize simultaneously; in SMGs, the leader chooses a policy first and the follower optimizes conditionally on that commitment. This asymmetry survives under several notational variants. The spatial-temporal sequential Markov game (STMG) factorizes the joint policy as
\[
\boldsymbol{\pi}(a\mid s)=\prod_{k=1}^N \pi_k(a_k\mid s,a_{<k}),
\]
thereby encoding a leader–follower hierarchy within each time step rather than only at the intertemporal level [2304.10351].

## 2. Equilibrium concepts

The basic equilibrium notion in the discounted single-leader single-follower setting is the stationary Stackelberg equilibrium (SSE). For a fixed leader policy \(\pi_L\), the follower computes a stationary best response
\[
\pi_F \in \mathrm{BR}(\pi_L):=\arg\max_{\pi_F}V_F^{\pi_L,\pi_F}(s),
\]
and the leader then solves
\[
\pi_L^* \in \arg\max_{\pi_L}V_L^{\pi_L,\mathrm{BR}(\pi_L)}(s).
\]
The paper enforcing this definition uses uniqueness of best responses to avoid ambiguity; this corresponds to an optimistic, leader-favorable selection, whereas the pessimistic case introduces set-valued discontinuities and is left for future work [2509.16296].

In convex–concave zero-sum models with coupled constraints, the equilibrium notion is formulated recursively. A policy profile is an \((\varepsilon,\delta)\)-Stackelberg equilibrium if the follower is \(\delta\)-optimal under the leader’s policy, the coupled constraints are satisfied up to \(\delta\), and the leader is within \(\varepsilon\) of the min–max value. This differs from generalized Nash equilibrium: in coupled-constraint settings, a generalized Nash equilibrium need not coincide with Stackelberg equilibrium, even though saddle points and Nash equilibria coincide in zero-sum games without coupling [2401.12437].

Incomplete-information settings replace a single follower by a type-indexed family of followers. In Bayesian Stackelberg Markov games, the attacker or follower has type \(\theta\), each type plays a deterministic best response to the leader’s Markov stationary mixed strategy, and the leader maximizes expected value under a belief \(b\) over types:
\[
U_D^{\pi_D}(b)=\mathbb{E}_{\theta\sim b}\big[V_D^{\pi_D,\mathrm{BR}(\pi_D,\theta)}(s_0,\theta)\big].
\]
The equilibrium is a Bayesian Strong Stackelberg Equilibrium, again with tie-breaking in favor of the leader [2007.10457].

Private-information versions modify the policy class itself. In smart-grid demand response, users’ storage levels are private information, so the lower-level game is defined over private Markovian strategies (PMS), where user \(i\)’s policy depends only on the public renewable state and its own private storage state. The corresponding equilibrium notion is private Markovian equilibrium (PME), under which no user can improve by unilateral deviation from its PMS [2509.12225].

Mean-field variants extend the equilibrium object from a policy pair to a policy–distribution triple. In the stationary Stackelberg mean-field equilibrium (SS-MFE), the follower must be optimal given the leader policy and the mean field, the mean field must satisfy a consistency condition \(\mu=\Gamma(\mu,\pi_L,\pi_F)\), and the leader must be optimal given the induced follower policy and mean field [2509.16296].

## 3. Existence, uniqueness, and structurally tractable subclasses

For the single-leader single-follower discounted model, existence and uniqueness of SSE follow from a contraction argument. If rewards and transitions satisfy continuity and boundedness, best responses are unique, the best-response maps are Lipschitz,
\[
\|\mathrm{BR}_i(\pi_{-i})-\mathrm{BR}_i(\pi_{-i}')\|_1\le d_i\|\pi_{-i}-\pi_{-i}'\|_1,
\]
and \(d_Ld_F<1\), then the composition \(T(\pi_L):=\mathrm{BR}_L(\mathrm{BR}_F(\pi_L))\) is a contraction on the leader policy space, so Banach’s fixed-point theorem yields a unique stationary Stackelberg equilibrium [2509.16296].

The mean-field extension requires a nested contraction argument. With a continuum of followers, the paper introduces Lipschitz constants for the follower best response, the mean-field update \(\Gamma\), and the leader best response. A unique stationary SS-MFE exists if
\[
d_\mu^\mu+d_\mu^F d_F^\mu<1
\quad\text{and}\quad
\frac{d_F^L+d_\mu^L}{1-(d_F^\mu+d_\mu^\mu+d_\mu^F)}<1.
\]
The first inequality makes the inner mean-field loop contractive, and the second makes the leader-facing composite map contractive [2509.16296].

A different tractable class arises from convex–concave structure. In continuous-state zero-sum SMGs, if the objective is convex in the leader parameters and concave in the follower parameters, the coupled feasible correspondence is concave and convex-valued, Slater’s condition holds, and unbiased stochastic first-order oracles with bounded variance are available, then nested SGDA and saddle-point-oracle SGD converge to Stackelberg equilibrium in polynomial time. The paper further supplies sufficient conditions for convexity and concavity through stochastic convexity or stochastic concavity of transitions and convexity or concavity of rewards [2401.12437].

Another tractable subclass is the private-state demand-response game. There, the lower-level one-stage game becomes an exact potential game after eliminating dominated actions in consumption, and the finite-horizon lower-level Markov game admits a pure PME. The PME can be computed in polynomial time by converting the lower-level game into an auxiliary Markov potential game and solving for a pure Markovian equilibrium through potential maximization. The paper emphasizes that computing equilibrium in general Markov games is hard, and polynomial-time algorithms are rarely available [2509.12225].

These results identify three distinct sources of tractability: contraction in best-response space, convex–concave min–max structure, and potential-game reductions. They also delineate where tractability ends. Outside these structured classes, several papers explicitly leave pessimistic Stackelberg selection, partial observability, non-stationary commitments, continuous-space extensions, or multi-leader competition as open problems [2509.16296] [2401.12437].

## 4. Computation and learning algorithms

A central computational difficulty in SMGs is that exact best-response dynamics are often intractable. One remedy is softmax smoothing. In the discounted single-leader single-follower setting, the follower best response is approximated by a Boltzmann policy
\[
\pi_F^{\mathrm{soft}}(a\mid s)\propto \exp\big(\beta\,Q_F^{\pi_L}(s,a)\big),
\]
and more generally
\[
\mathrm{softmax}_{\alpha_i}(Q_i^{*,\pi_{-i}}(s,\cdot))(a)
=
\frac{\exp(\alpha_i Q_i^{*,\pi_{-i}}(s,a))}
{\sum_{a'}\exp(\alpha_i Q_i^{*,\pi_{-i}}(s,a'))}.
\]
The softmax map is Lipschitz, and when the action gap \(\delta\) is bounded below,
\[
\|\mathrm{softmax}_\alpha(Q)-\mathrm{argmax}_U(Q)\|_1\le 2n e^{-\alpha\delta}.
\]
Using an \(\varepsilon\)-net and projection,
\[
\hat{\pi}_F^{k}=\mathrm{proj}_\varepsilon\big(\mathrm{softmax}_{\alpha_F}(Q_F^{*,\hat{\pi}_L^{k}})\big),\qquad
\hat{\pi}_L^{k+1}=\mathrm{proj}_\varepsilon\big(\mathrm{softmax}_{\alpha_L}(Q_L^{*,\hat{\pi}_F^{k}})\big),
\]
the iterates converge to within an explicit \(\varepsilon\)-dependent bound of the true SSE when \(d_Ld_F<1\) [2509.16296].

The same work gives a three-step reinforcement-learning framework. First, fix \(\pi_L^k\) and learn the follower’s best response \(\pi_F^{k*}=\mathrm{BR}_F(\pi_L^k)\). Second, fix \(\pi_F^{k*}\) and learn the leader’s best response \(\pi_L^{k+1}=\mathrm{BR}_L(\pi_F^{k*})\). Third, iterate until \(\|\pi_L^{k+1}-\pi_L^k\|_1\le \mathrm{tol}\). Softmax smoothing, \(\varepsilon\)-net projection, and optional entropy regularization are used to stabilize learning, and each best-response subproblem can be solved with off-the-shelf RL such as PPO [2509.16296].

In convex–concave zero-sum SMGs, learning can be phrased as nested stochastic gradient descent–ascent. The inner loop solves a Lagrangian saddle-point problem over follower parameters and multipliers, and the outer loop performs projected SGD on the leader parameters. With stepsizes in \(\Theta(1/\sqrt{t+1})\), the averaged iterate is an \((\varepsilon+\delta,\delta)\)-Stackelberg equilibrium after \(\tilde{O}(1/(\varepsilon^2\delta^2))\) stochastic gradient evaluations for nested SGDA, or \(\tilde{O}(1/\varepsilon^2)\) with a saddle-point oracle; if the leader value is strongly convex, these rates improve to \(\tilde{O}(1/(\varepsilon\delta^2))\) and \(\tilde{O}(1/\varepsilon)\). The same paper extends the argument to trajectory-based policy gradients in continuous-state continuous-action SMGs [2401.12437].

Under incomplete information, Bayesian Strong Stackelberg Q-learning embeds a Bayesian stage-game solve inside Q-learning updates. Separate Q-tables are maintained for the leader and each follower type, and the Bellman target uses the Bayesian SSE value at the next state. Under bounded rewards, \(\gamma\in(0,1)\), sufficient exploration, and Robbins–Monro stepsizes, the iterates converge almost surely to a fixed point of the SSE Bellman operator, and the induced policy profile converges to a Bayesian SSE [2007.10457].

When the follower utility is unknown but linearly parameterized, the dynamic Stackelberg formulation yields a no-regret learning problem. The leader maintains a confidence set over the unknown parameter \(\theta^*\), updates it from observed follower best responses, and computes optimistic \(\varepsilon\)-conservative policies by backward induction. With probability at least \(1-\delta\),
\[
\mathrm{Regret}(T)\le C\cdot T^{1-\frac{1}{p}} + H\sqrt{2T\ln\!\left(\frac{1}{\delta}\right)},
\]
for an explicit polynomial constant \(C\), and the regret is independent of the size of the state space [2202.04786].

In partially observed security settings with unknown dynamics, model-free Expected Sarsa can be applied on a belief-state MDP produced by the common-information approach. Particle filters approximate the Bayes belief update, and the follower best response is recomputed at each belief state, yielding an \(\varepsilon\)-Markov perfect Stackelberg equilibrium under filter accuracy and learning assumptions [2005.11853].

## 5. Structural extensions

Mean-field structure is the most prominent extension. In the single-leader case, the leader interacts with an infinite population of followers whose state distribution \(z_t\) evolves endogenously. The discrete-time master-equation approach constructs equilibrium-generating functions \(\theta_t[\pi_t,z_t]\), where \(\pi_t\) is the common belief on the leader’s type and \(z_t\) is the mean field. Backward recursion yields follower and leader value functions \(V_t^f,V_t^l\), and forward recursion updates both the belief and the mean field through maps \(F\) and \(\phi\), thereby computing all Stackelberg mean-field equilibria [2201.05959]. The same architecture has been extended to multiple leaders, finite major followers, and an infinite number of minor followers; the resulting Stackelberg mean-field equilibrium with multiple leaders (SMFE-ML) is characterized by a discrete-time master equation over common beliefs and mean fields [2209.03186].

Private information can be placed in either the leader or the followers. In stochastic Stackelberg games with asymmetric information, a common-agent representation maps common information into prescriptions for both players, reducing the dynamic game to a belief-state recursion. The backward algorithm computes per-time fixed points rather than a global fixed point over complete histories [2005.01997]. In Bayesian Stackelberg Markov games, follower types are encoded by a prior and possibly updated through observations, which situates SMGs within the broader landscape of incomplete-information stochastic games while preserving the leader’s commitment structure [2007.10457]. In smart-grid demand response, the private-state formulation goes further: followers observe the public renewable state and their own storage levels, whereas the leader observes only aggregate demand [2509.12225].

Several papers move from discrete-time stochastic games to continuous-time or semi-Markov dynamics while retaining the leader–follower semantics. The online educational forum model represents the interaction as a continuous-time Markov chain whose generator depends on instructor and student arrival-rate actions, and solves the resulting single-leader multiple-followers Stackelberg problem through a mixed-integer linear program [2112.01239]. Spatial-temporal moving target defense uses a semi-Markov Stackelberg game in which the defender chooses both migration probabilities and defending-period lengths, and computes \(\varepsilon\)-optimal stationary strategies via value iteration or relative value iteration combined with MIQP subproblems [2002.10390]. Zero-sum stochastic linear-quadratic Stackelberg differential games with Markovian regime switching derive regime-dependent feedback laws from coupled differential Riccati equations, while linear-quadratic mean-field stochastic Stackelberg differential games with random exit time use a two-stage decomposition driven by the stopping time [2408.17030] [2104.13528].

Another extension concerns the action order itself. STMG treats the joint policy as a sequential conditional factorization within each time step, allowing asymmetric training with symmetric execution in multi-agent reinforcement learning [2304.10351]. A later analysis shows that, in an \(N\)-level Stackelberg game, changing the decision order typically yields an overdetermined system, so the equilibrium point shifts unless special structural conditions hold; this motivates hierarchical priority adjustment, where an upper policy chooses the order and a lower STMG executes under that order [2605.07240].

## 6. Applications, empirical evidence, and recurrent limitations

Energy-market policy design is a leading application. In the single-leader mean-field SMG for electricity tariffs, the leader is a utility or state commission setting time-varying retail tariffs, and the followers are heterogeneous prosumers and consumers grouped by aggregators at three nodes in a 3-bus transmission system with 4 generators and 3 lines. The leader pursues economic efficiency, equity across income groups, and grid stability; equity is measured by Energy Expenditure Incidence (EEI), and stability by Incremental Mean Volatility,
\[
\mathrm{IMV}=\lim_{T\to\infty}\frac{1}{T}\sum_{t=1}^T |\mathrm{LMP}_{t+1}-\mathrm{LMP}_t|.
\]
Using PPO for both leader and followers over a 100-day simulation, learned policies reduce IMV by roughly 3 units relative to a baseline without storage or RL, reduce the EEI gap between prosumers and consumers from about \(1\%\) to about \(0.7\%\) of income, converge to higher fixed charges for higher-income groups, and reshape charging to midday with net injections during evening peaks [2509.16296].

Security is another major domain. Reach–avoid problems are modeled as convex–concave zero-sum SMGs in which the leader acts first, the follower reacts under safety constraints, and nested SGDA or saddle-point-oracle gradient methods converge in polynomial time to recursive Stackelberg equilibrium policies. In experiments with Dubins-car dynamics, Stackelberg-trained protagonists achieve higher win rates and better safety–liveness trade-offs than simultaneous generalized-Nash baselines [2401.12437]. In moving target defense, Bayesian Strong Stackelberg Q-learning improves the state of the art for web-application security and converges to optimal movement policies in domains with incomplete information about adversaries, while the spatial-temporal Markov Stackelberg formulation yields lower long-run average costs than Bayesian Stackelberg and uniform-random baselines under source-destination-dependent switching costs and random exploitation times [2007.10457] [2002.10390].

Adversarial federated learning introduces a further Bayesian SMG variant. The defense problem is formulated under a mixture of uncertain, unknown, and adaptive poisoning attacks, and the proposed meta-Stackelberg defense combines pre-training and online adaptation. Under smoothness, strict competitiveness, and a Stackelberg Polyak–Łojasiewicz condition, meta-Stackelberg learning converges to a first-order \(\varepsilon\)-meta-equilibrium in \(O(\varepsilon^{-2})\) outer iterations with \(O(\varepsilon^{-4})\) samples per iteration and \(O(\log(\varepsilon^{-1}))\) inner steps; empirically it is robust against mixed poisoning and backdoor attacks [2410.17431].

Applications also appear outside security and energy. In online educational forums, a CTMC Stackelberg Markov model reproduces effective incentive design, student heterogeneity, non-monotonic participation with increasing instructor involvement, and the super-poster phenomenon, while delivering an MILP-based optimal instructor plan [2112.01239]. In smart-grid demand response with up to 50 users, the PME framework scales beyond the small-user regimes emphasized in prior studies [2509.12225].

Across these formulations, several limitations recur. Many analyses focus on stationary policies and stationary equilibria rather than history-dependent subgame-perfect policies; optimistic tie-breaking is often used to enforce unique best responses; full observability is typically assumed; single-leader settings dominate; continuous state or action spaces frequently require additional measure-theoretic or convex–concave structure; and partial observability, multi-leader competition, robust or pessimistic Stackelberg selection, off-policy learning, and adaptive or dynamically chosen decision orders remain active extensions rather than resolved theory [2509.16296] [2401.12437] [2605.07240]. These repeated caveats suggest that SMGs are best viewed not as a single equilibrium model, but as a family of leader–follower Markovian decision frameworks whose tractability depends sharply on informational structure, population scale, and the analytical regularity of the best-response mapping.

Source: https://www.emergentmind.com/topics/stackelberg-markov-game-smg