Zero-Sum Symmetric Markov Games

• Zero-sum symmetric Markov games are defined by two players with symmetric state and action spaces, anti-symmetric rewards, and a max–min equilibrium framework.
• The models use dynamic programming via the Shapley operator and ergodic analysis to study finite-horizon, discounted, and average-payoff formulations.
• Advanced learning methods, including optimistic value iteration, policy gradients, and decentralized Q-learning, enable efficient equilibrium computation.

Zero-sum symmetric Markov games are two-player stochastic games in which a Markovian state process evolves under the players’ actions, the players’ payoffs are opposite, and the roles of the two players are structurally symmetric. In the classical finite-state formulation, they generalize Markov decision processes by replacing a single controller with a max–min interaction over states, actions, rewards, and transitions; in more recent work, symmetry is made explicit either at the level of per-state payoff matrices or at the level of policy-induced returns over trajectories (Renault, 2019, Ablin et al., 29 Aug 2025).

1. Formal model and notions of symmetry

A standard discounted two-player zero-sum Markov game is specified by a tuple

$(S, A, O, P, R, \gamma),$

where $S$ is a finite state space, $A$ and $O$ are the action sets of the two players, $P(s,a,o,s')$ is the Markov transition model, $R(s,a,o)$ is the expected one-step reward for the first player, and $\gamma \in (0,1]$ is the discount factor. The second player’s reward is $-R(s,a,o)$, so the interaction is zero-sum (Lagoudakis et al., 2012). In the finite-horizon simultaneous-move formulation, the game is written as

$(S, A, H, P, u),$

with a finite horizon $H$, a common action set $S$0, a state partition $S$1 with $S$2, a transition kernel $S$3, and a one-step payoff $S$4 for player 1, with player 2 receiving $S$5 (Ablin et al., 29 Aug 2025).

The state-value function under policies $S$6 is the expected cumulative payoff from a state. In the finite-horizon model this is written

$S$7

and the game value at the initial state is

$S$8

The same max–min structure underlies the discounted and average-payoff formulations (Ablin et al., 29 Aug 2025).

Symmetry is not unique. One recent formalization distinguishes three notions. A game is SSG if every state corresponds to a symmetric zero-sum matrix game,

$S$9

It is MSG if for every pair of Markov policies,

$A$0

and HSG if the same anti-symmetry holds for every pair of history-dependent policies. This separates per-state skew-symmetry from trajectory-level symmetry and from symmetry over the full policy class (Ablin et al., 29 Aug 2025).

A common misconception is that symmetry is exhausted by identical action sets and skew-symmetric one-step payoffs. The later classification shows that symmetry can instead be imposed on Markov-policy returns or on history-dependent returns, so symmetric zero-sum Markov games form a hierarchy rather than a single model class (Ablin et al., 29 Aug 2025).

2. Value functions, Shapley operators, and equilibrium structure

For zero-sum Markov games, the local dynamic programming object is the state-action value

$A$1

and the value function is the minimax value of the one-state matrix game induced by $A$2: $A$3 At each state, the minimax mixed action can be computed by a linear program with variables $A$4 and $A$5, subject to probability constraints and adversarial lower bounds against every opponent action (Lagoudakis et al., 2012).

The classical finite-state stochastic-game model uses the Shapley operator. For a repeated game with perfect information, finite state space $A$6, state-dependent rewards $A$7, and transition kernels $A$8, the operator is

$A$9

with $O$0. This operator is monotone, additively homogeneous, and sup-norm nonexpansive. For the $O$1-stage game, $O$2 and $O$3 (Akian et al., 2014).

In the tutorial treatment of finite stochastic games, the $O$4-stage values satisfy the recursive Shapley equation

$O$5

and the $O$6-discounted values satisfy

$O$7

The finite-horizon game admits Markov optimal strategies, while the discounted game admits stationary optimal strategies (Renault, 2019).

Within structurally symmetric zero-sum Markov games, symmetry is often imposed by identical state and action spaces, symmetric dynamics, and anti-symmetric rewards. Under those conditions, one can restrict attention to symmetric Nash equilibria in which the two policies are identical or related by the symmetry transformation of the game (Wang et al., 2023).

3. Asymptotic values, mean payoff, and ergodicity

The long-run theory of zero-sum Markov games splits into two closely related questions: convergence of discounted and finite-horizon values, and state-independence of average payoff. In the finite-state stochastic-game setting, $O$8 admits a bounded Puiseux expansion around $O$9, so $P(s,a,o,s')$0 exists; moreover, the $P(s,a,o,s')$1-stage values converge and satisfy

$P(s,a,o,s')$2

The same finite model has a uniform value: both players can uniformly guarantee the common limit for sufficiently long horizons (Renault, 2019).

The Big Match shows why asymptotic theory is subtler than discounted dynamic programming. In that absorbing game, $P(s,a,o,s')$3 for all $P(s,a,o,s')$4 and $P(s,a,o,s')$5, but stationary or Markov strategies are not sufficient for player 1 to uniformly guarantee any strictly positive value; a history-dependent strategy is required to uniformly guarantee the uniform value $P(s,a,o,s')$6 (Renault, 2019). This is the standard counterexample to the idea that stationary optimality in discounted games automatically transfers to long-run uniform optimality.

For average-payoff repeated games with perfect information, the central object is the mean payoff vector

$P(s,a,o,s')$7

when the limit exists. If the ergodic equation

$P(s,a,o,s')$8

has a solution, then $P(s,a,o,s')$9, so the mean payoff is independent of the initial state (Akian et al., 2014). The paper on ergodicity sharpens this into a structural criterion: for bounded rewards, the following are equivalent.

First, the recession function $R(s,a,o)$0 has only trivial fixed points, i.e. only constants. Second, for every state-dependent perturbation $R(s,a,o)$1, the perturbed game $R(s,a,o)$2 has a constant mean payoff vector. Third, for every $R(s,a,o)$3, the equation

$R(s,a,o)$4

is solvable. Fixed points of $R(s,a,o)$5 are the nonlinear harmonic functions of the payment-free Shapley operator, so ergodicity is exactly uniqueness modulo constants of these nonlinear harmonic functions (Akian et al., 2014).

In the finite-action perfect-information case, ergodicity is also characterized combinatorially. The payment-free Shapley operator has a nontrivial fixed point if and only if there exists a pair of conjugate subsets of states in two directed hypergraphs built from the support of the transition probabilities. Consequently, ergodicity depends only on the support of the transition kernel, not on the numerical values of the positive probabilities, and can be checked in polynomial time when the number of states is fixed (Akian et al., 2014). Symmetry does not remove these structural issues: it is a special case of the general framework, and symmetric games may still be non-ergodic.

4. Learning, policy optimization, and equilibrium computation

In finite-horizon simultaneous-move Markov games with linear structure, optimistic least-squares minimax value iteration yields provable learning guarantees in both offline and online settings. The model assumes a feature map

$R(s,a,o)$6

with linear rewards and transitions. Under this assumption, the paper proves a

$R(s,a,o)$7

upper bound on the offline duality gap and on online regret. A central technical feature is that optimism for simultaneous moves requires both upper and lower confidence bounds, which define an auxiliary general-sum matrix game at each state; to avoid the hardness of Nash equilibrium computation in general-sum matrix games, the algorithm computes a Coarse Correlated Equilibrium instead (Xie et al., 2020).

For infinite-horizon discounted games with function approximation, a policy-based algorithm with log-linear policies and natural-policy-gradient structure is proved to find a near-optimal policy within a polynomial number of samples and iterations under regularity conditions on the game and the approximation class (Zhao et al., 2021). This line of work treats the zero-sum Markov game as the natural competitive extension of policy optimization in MDPs.

A more explicitly symmetric optimization framework uses entropy-regularized optimistic multiplicative weights update together with slower value updates. In the full-information tabular setting, the proposed single-loop method updates both agents symmetrically and achieves finite-time last-iterate linear convergence to the quantal response equilibrium of the regularized problem; by controlling the amount of regularization, this yields sublinear last-iterate convergence to a Nash equilibrium in both discounted infinite-horizon and finite-horizon episodic zero-sum Markov games (Cen et al., 2022).

Decentralized learning has also been studied in a radically uncoupled regime. In infinite-horizon discounted zero-sum Markov games, a two-timescale Q-learning dynamics is proved to be both rational and convergent: if the opponent follows an asymptotically stationary strategy, each agent converges to a best response; if both agents use the dynamics, they converge to the Nash equilibrium of the game. The construction uses local Q-function updates on a fast timescale and value-function updates on a slower timescale to handle the non-stationarity induced by simultaneous adaptation (Sayin et al., 2021).

A separate direction treats equilibrium selection by reward design. In a zero-sum Markov game, reward perturbations can be chosen by solving a bilevel problem whose lower level is a regularized Nash equilibrium and whose upper level encodes an arbitrating objective. The proposed framework differentiates through the regularized equilibrium by implicit differentiation, requires only a black-box solver for the regularized Nash equilibrium, and applies directly to symmetric zero-sum Markov games when policy parameterization and reward perturbations are constrained to respect the underlying symmetry (Wang et al., 2023).

5. Symmetry-specific reductions and information structures

The one-shot theory of symmetric zero-sum matrix games remains informative because every state of a simultaneous-move Markov game induces a local matrix game. For a skew-symmetric payoff matrix $R(s,a,o)$8, the value is $R(s,a,o)$9, and under symmetric and regular distributions over such matrices, the support of the unique optimal strategy has odd cardinality; moreover, every odd support set is equally likely, with probability $\gamma \in (0,1]$0 (Brandl, 2016). This suggests that per-state equilibrium supports in symmetric zero-sum Markov games are generically mixed and combinatorially constrained, even when the global dynamics are more complex.

Recent work on learning without payoff observations makes the symmetry structure itself algorithmically central. In finite-horizon zero-sum symmetric Markov games with known transition dynamics and observed opponent actions, but without observed payoffs, the learner can still compete with an adversary by reducing the problem to online linear optimization over a convex class of symmetric payoff functions. The three symmetry classes SSG, MSG, and HSG correspond to increasingly strong invariances of the value function under role exchange. For SSG and MSG, the algorithms operate directly on the Markov game; for HSG, the symmetry is strong enough that the game collapses to an equivalent symmetric zero-sum matrix game at the first stage (Ablin et al., 29 Aug 2025).

Not every symmetry assumption has the same dynamical consequences. HSG effectively eliminates strategic dependence after the first move, but SSG and MSG preserve a genuinely Markovian interaction. This distinction is essential when interpreting statements about “symmetric” zero-sum Markov games: some symmetry classes preserve the full recursive structure, while others reduce it.

A different information structure appears in continuous-time games with vanishing stage duration. In two-person zero-sum games where players control a continuous-time Markov state process at discrete times induced by a partition of $\gamma \in (0,1]$1, the value $\gamma \in (0,1]$2 converges as the mesh of the partition tends to $\gamma \in (0,1]$3. The analysis covers both the observed-state case and a symmetric no-information case in which neither player observes the current state but both know the law of the state process. In the symmetric no-information model, the effective state variable is the law $\gamma \in (0,1]$4, and the limit value is characterized as the unique viscosity solution of a Hamilton–Jacobi–Isaacs equation on the simplex (Sorin, 2016).

6. Variants, examples, and scope

Semi-Markov and risk-sensitive extensions preserve much of the zero-sum structure while changing the dynamic programming equations. For a two-player zero-sum risk-sensitive semi-Markov game with finite state space, compact action sets, and suitable continuity and irreducibility conditions, the average criterion

$\gamma \in (0,1]$5

has a value, and there exists a stationary saddle point equilibrium. The associated optimality equation is multiplicative rather than additive, reflecting the exponential risk-sensitive criterion (Bhabak et al., 2021).

Perfect-information semi-Markov games admit sharper structural results. Under limiting ratio average payoff, a zero-sum two-person perfect information semi-Markov game has a value and both the maximiser and the minimiser have optimal pure semi-stationary strategies. The proof proceeds by fixing the initial state, forming the matrix of undiscounted payoffs indexed by pairs of pure stationary strategies, and showing that every $\gamma \in (0,1]$6 submatrix has a saddle point, hence the whole matrix has a pure saddle point (Sinha et al., 2022). This is the semi-Markov analogue of turn-based perfect-information stochastic-game structure.

On the approximation side, value-function methods transfer from MDPs to Markov games. In particular, value function approximation with LSPI is shown to learn good policies in a soccer domain and in a flow control problem, and stronger convergence guarantees are obtained for the two-player generalization of optimal stopping with linear function approximation (Lagoudakis et al., 2012). These results do not require symmetry, but symmetric domains are a natural special case because the two players share state dynamics and typically comparable action spaces.

Search-based planning has also been extended to simultaneous-action zero-sum Markov games. Simultaneous AlphaZero adapts AlphaZero-style MCTS to deterministic two-player zero-sum Markov games with simultaneous actions by resolving each node as a matrix game whose entries combine immediate reward and future value estimates. To handle bandit feedback during tree search, the method incorporates a regret-optimal matrix-game solver, and it demonstrates robust strategies in a continuous-state discrete-action pursuit-evasion game and in satellite custody maintenance scenarios, including evaluation against maximally exploitative opponents (Becker et al., 13 Dec 2025).

Taken together, these strands fix the scope of the subject. Zero-sum symmetric Markov games include discounted and average-payoff stochastic games, simultaneous-move finite-horizon games, continuous-time limits, and semi-Markov variants. Their theory combines minimax dynamic programming, symmetry constraints, ergodic analysis, and increasingly sophisticated learning algorithms. The recurring theme is that symmetry sharpens both interpretation and computation, but it does not trivialize the underlying control problem: the decisive objects remain the Shapley operator, the induced local matrix games, and the long-run structure of the controlled state process.