Finite-Time Analysis of Q-Value Iteration for General-Sum Stackelberg Games

Abstract: Reinforcement learning has been successful both empirically and theoretically in single-agent settings, but extending these results to multi-agent reinforcement learning in general-sum Markov games remains challenging. This paper studies the convergence of Stackelberg Q-value iteration in two-player general-sum Markov games from a control-theoretic perspective. We introduce a relaxed policy condition tailored to the Stackelberg setting and model the learning dynamics as a switching system. By constructing upper and lower comparison systems, we establish finite-time error bounds for the Q-functions and characterize their convergence properties. Our results provide a novel control-theoretic perspective on Stackelberg learning. Moreover, to the best of the authors' knowledge, this paper offers the first finite-time convergence guarantees for Q-value iteration in general-sum Markov games under Stackelberg interactions.

• The paper establishes the first finite-time convergence guarantee for Stackelberg Q-value iteration in two-player general-sum Markov games, with errors bounded by 6γ^k/(1−γ) + 3ε/(1−γ) for both players.
• The method models changing leader–follower policies as affine switching systems and uses upper and lower comparison iterations to prove geometric decay under an ε-relaxed best-response condition.
• Experiments confirm the theoretical bounds but show a conservative, nonzero error floor, highlighting the need for tighter slack estimates and extensions to sampled, asynchronous, and function-approximation settings.

Overview

This paper studies the convergence of Stackelberg Q-value iteration in two-player general-sum Markov games. The authors, Jeong and Lee of KAIST, depart from equilibrium-based analyses that dominate Nash-style MARL theory and instead track the evolution of the Q-function iterates directly. Their central contribution is a finite-time error bound for both players' Q-functions under an $\epsilon$-relaxed best-response condition, obtained by modeling the iteration as an affine switching system and analyzing upper and lower comparison systems. The authors state that this is the first finite-time convergence guarantee for Q-value iteration in general-sum Markov games under Stackelberg interactions.

Motivation and positioning

Convergence analysis for general-sum Markov games is difficult because the Nash equilibrium operator is not a contraction and multiple equilibria may exist at a given state; value iteration can even exhibit cyclic behavior. Existing Stackelberg learning results are largely local or asymptotic, or rely on restrictive assumptions such as myopic followers or centralized coordination. Prior finite-time work in Stackelberg general-sum settings (e.g., Bai et al.) did not extend to Markov games or to Q-value iteration. This paper fills that gap by adapting the switching-system framework previously applied to single-agent Q-learning and minimax (zero-sum) Q-learning by the same group.

The setting is a two-player discounted Markov game with leader action space $\mathcal{A}$ and follower action space $\mathcal{B}$. The follower's policy depends on the leader's action, and the induced bilevel structure makes the Bellman operator asymmetric and policy-dependent: each player's update uses the other's current greedy actions, so the effective transition operator changes with the iterates.

Relaxed policy condition

The key analytical device replaces Bowling's worst-case-response assumption from Nash Q-learning — which does not transfer to the asymmetric Stackelberg setting — with an $\epsilon$-relaxation on the upper bound. Concretely, Assumption 2 requires that for all states, iterations, and arbitrary policies $\mu^1,\mu^2$, the on-policy values satisfy inequalities of the form

$$Q^1_k(s,a_k(s),b_k(s,a_k(s))) \le Q^1_k(s,a_k(s),\mu^2(s)) + \epsilon,$$

and analogously for the follower and for the equilibrium pair $(a_*,b_*)$. A Stackelberg Q-function inequality lemma first establishes the corresponding lower bounds directly from the greedy definitions. The paper then proves boundedness of the iterates ($\|Q^i_k\|_\infty \le 1/(1-\gamma)$) and shows that any $\epsilon \ge 2/(1-\gamma)$ satisfies the relaxed condition, so the assumption is always satisfiable — though this worst-case certificate is loose, a point borne out empirically.

Switching-system analysis and main result

The core construction reduces convergence to stability of affine switching systems $x_{k+1} = A_{\sigma_k}x_k + b_{\sigma_k}$. Two comparison iterations are built: an upper system using the pair $\mathcal{A}$0 with an additive $\mathcal{A}$1 term, and a lower system using $\mathcal{A}$2 with $\mathcal{A}$3. Induction arguments show these systems sandwich the true iterate whenever they sandwich it at initialization. In vectorized form, using the Stackelberg action-selection matrix $\mathcal{A}$4 built from Kronecker products of basis vectors, the upper system reads

$\mathcal{A}$5

with the lower system symmetric. Since $\mathcal{A}$6, unrolling yields geometric decay despite the time-varying switching signal. Combining the two bounds via the triangle inequality gives the main theorem:

$\mathcal{A}$7

and the same bound holds for the follower. The first term vanishes geometrically; the second is a non-vanishing residual proportional to $\mathcal{A}$8. An important consequence is that exact convergence to the Stackelberg fixed point is not guaranteed unless $\mathcal{A}$9: the guarantee is convergence to an $\mathcal{B}$0-neighborhood. The proof also relies on uniqueness of the maximizers in the $\mathcal{B}$1 operations, a standard but nontrivial regularity assumption.

Numerical experiments

Validation uses a single-state, deterministic two-action-per-player game with $\mathcal{B}$2, admitting a closed-form unique Stackelberg equilibrium $\mathcal{B}$3 and closed-form $\mathcal{B}$4. Across five random initializations, the empirical sup-norm errors of both players remain below the theoretical bounds at every iteration and decay geometrically, consistent with the theorem. The experiments also expose the conservatism of the global slack: the constant $\mathcal{B}$5 is dominated by early iterations when randomly initialized policies are unstable, whereas the iteration-dependent $\mathcal{B}$6 shrinks as policies stabilize and yields substantially tighter bounds. Notably, none of the bounds converges to zero because the required $\mathcal{B}$7 values remain strictly positive, confirming the residual-term behavior predicted by the theory. The experimental scope is limited to a single-state game, so the empirical evidence does not test multi-state dynamics where the switching structure is richer.

Limitations and open questions

Several limitations are explicit. First, the $\mathcal{B}$8-relaxation introduces a permanent error floor; whether conditions exist under which $\mathcal{B}$9 can be driven to zero (e.g., via gap assumptions on Q-values) is not addressed. Second, the existence lemma only certifies $\epsilon$0, which makes the residual bound vacuous in practice; the tighter data-dependent $\epsilon$1 observed experimentally is not characterized theoretically. Third, the analysis assumes deterministic policies with unique maximizers and synchronous full-information updates; stochastic approximation, asynchronous sampling, and function approximation are excluded. Finally, the empirical validation covers only a degenerate one-state environment. The authors identify extension to stochastic-approximation-based Stackelberg Q-learning and tightening of the assumptions as open directions.

Conclusion

The paper provides a control-theoretic, switching-system-based finite-time analysis of Stackelberg Q-value iteration in two-player general-sum Markov games, yielding explicit $\epsilon$2-decaying error bounds with an $\epsilon$3-dependent residual under a relaxed best-response condition. It extends prior switching-system analyses from single-agent and zero-sum settings to the asymmetric Stackelberg case, and its experiments confirm the bounds while highlighting the conservativeness of global slack certificates. The main open issues are eliminating the residual error floor and extending the guarantees to sampled, asynchronous implementations.