---
title: Distributionally Robust Markov Games
url: https://www.emergentmind.com/topics/distributionally-robust-markov-games-drmgs
type: topic
---

# Distributionally Robust Markov Games

Searching arXiv for recent papers on distributionally robust Markov games and related robust multi-agent RL.
Distributionally Robust Markov Games (DRMGs) are Markov games in which the transition kernel is not known exactly and is instead assumed to lie in an ambiguity set of plausible kernels; each agent then evaluates policies by worst-case performance over that set, rather than under a single nominal model. In the formulations developed recently, this framework supports both average-reward and finite-horizon analyses, robust analogues of Nash-type solution concepts, and algorithmic procedures for equilibrium computation and learning under model uncertainty [2508.03136], [2404.18909], [2508.02948].

## 1. Formal model and uncertainty structure

A non-robust Markov game with \(N\) agents is specified by
\[
(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),
\]
where \(S\) is the common state space, \(A_i\) is the action space of agent \(i\), \(A=\otimes_i A_i\) is the joint action space, \(r_i:S\times A\to\mathbb{R}\) is agent \(i\)’s reward, and \(\mathsf P=\{p_s^a:s\in S,a\in A\}\) is the transition kernel [2508.03136]. A stationary policy for agent \(i\) is \(\pi_i:S\to\Delta(A_i)\), and a joint policy is \(\pi=(\pi_1,\dots,\pi_N)\) [2508.03136].

In a distributionally robust Markov game, the kernel is uncertain. Instead of a single \(\mathsf P\), one works with an ambiguity set
\[
P=\bigotimes_{(s,a)\in S\times A} P_s^a,
\]
where each \(P_s^a\subset\Delta(S)\) is nonempty, convex, and compact [2508.03136]. This is the standard \((s,a)\)-rectangular structure: the adversary chooses transitions independently per \((s,a)\) [2508.03136]. Compatible examples include KL balls and Wasserstein balls; in the average-reward experiments of [2508.03136], the local set is instantiated as
\[
\mathcal{P}^a_s = \{ q\in\Delta(S) \mid D_{KL}(q \,\|\, (P_0)^a_s) \leq \theta \}.
\]

Several recent finite-horizon formulations refine this generic ambiguity model. One line uses agent-wise \(f\)-divergence uncertainty sets
\[
\mathcal{U}_i = \bigotimes_{(h,s,\mathbf a)\in[H]\times\mathcal{S}\times\mathcal{A}} \mathcal{U}^{\sigma_i}_{i,h,f}(s,\mathbf a),
\]
with Total Variation or Kullback–Leibler divergence balls around the nominal transition \(P^\star_h(\cdot\mid s,\mathbf a)\) [2508.02948]. Another line proposes policy-induced \((s,a_i)\)-rectangular “fictitious” uncertainty sets centered at the transition obtained by averaging the nominal kernel over other agents’ behavior,
\[
U_\rho^{\sigma_i}(P^0,\pi) = \bigotimes_{h,s,a_i} U_\rho^{\sigma_i}(P^{\pi_{-i}}_{h,s,a_i}),
\]
which reduces each agent’s problem, for fixed \(\pi_{-i}\), to a robust MDP indexed by the agent’s own action \(a_i\) rather than by the full joint action \(\mathbf a\) [2409.20067]. This suggests that ambiguity-set design is not merely technical; it determines whether the multi-agent robust problem inherits the decomposability of robust MDPs or retains the full curse of joint action dimensionality.

A complementary line of work on distributionally robust MDPs emphasizes that rectangularity governs the relation between static robust formulations and dynamic game formulations, and delineates when strong duality and Bellman-style recursion remain valid [2308.11139]. That discussion is single-controller, but it directly informs DRMG modeling because the nature player in a robust Markov game plays the same role as the adversarial transition selector in a robust MDP [2308.11139].

## 2. Robust performance criteria and solution concepts

For a stationary joint policy \(\pi\), the average reward of agent \(i\) in a non-robust Markov game is
\[
g^\pi_{\,\mathsf P,i}(s)\triangleq \liminf_{n\to\infty} \mathbb{E}_{\pi,\mathsf P}\Big[\frac{1}{n}\sum^{n-1}_{t=0} r_i(s_t,a_t)\mid S_0=s \Big],
\]
and the worst-case robust average reward is
\[
g^\pi_{P,i}(s) \;:=\; \min_{\kappa\in\bigotimes_{t\ge0}P}\; \liminf_{n\to\infty} \mathbb{E}_{\pi,\kappa}\Big[\frac{1}{n}\sum_{t=0}^{n-1} r_i(s_t,a_t)\mid S_0=s\Big].
\]
Under irreducibility, this robust average reward is independent of the initial state [2508.03136].

The average-reward robust Nash equilibrium is then defined by the inequality
\[
g^{\pi^*}_{P,i}(s)\;\ge\; g^{(\pi^*_{-i},\pi_i)}_{P,i}(s),
\quad\forall i\in N,\;\forall \pi_i,\;\forall s\in S,
\]
so each agent maximizes its own worst-case long-run average reward against adversarial transition selection from the ambiguity set [2508.03136]. The corresponding robust best-response set is
\[
\text{BR}_i(\pi_{-i}) \triangleq \left\{\pi_i \;:\; g^{(\pi_{-i},\pi_i)}_{P,i} = \max_{\mu_i} g^{(\pi_{-i},\mu_i)}_{P,i} \right\},
\]
and a robust equilibrium is precisely a fixed point of the product correspondence \(\text{BR}(\pi)=\bigotimes_i \text{BR}_i(\pi_{-i})\) [2508.03136].

In finite-horizon robust multi-agent reinforcement learning, the central quantity is the robust value
\[
V^{\pi,\sigma_i}_{i,h}(s) \triangleq \inf_{\tilde{P}\in\mathcal{U}_i}\; \mathbb{E}_{\pi,\tilde P} \Big[ \sum_{t=h}^H r_{i,t}(s_t,\mathbf a_t)\,\Big|\, s_h=s\Big],
\]
with robust \(Q\)-value defined analogously [2508.02948]. The robust best-response value is
\[
V^{\dagger,\pi_{-i},\sigma_i}_{i,h}(s) \triangleq \max_{\pi'_i\in\Delta(\mathcal{A}_i)} V^{\pi'_i\times\pi_{-i},\sigma_i}_{i,h}(s),
\]
and several robust equilibrium notions follow by replacing ordinary values with robust ones [2508.02948].

The most common finite-horizon notions are the robust \(\varepsilon\)-Nash equilibrium, robust \(\varepsilon\)-coarse correlated equilibrium, and robust \(\varepsilon\)-correlated equilibrium. For example, a product policy \(\pi\) is an \(\varepsilon\)-robust NE if
\[
\text{gap}_{\text{NE}}(\pi,s)
\triangleq
\max_{i\in\mathcal{M}}
\big\{ V^{\dagger,\pi_{-i},\sigma_i}_{i,1}(s) - V^{\pi,\sigma_i}_{i,1}(s)\big\}
\le \varepsilon
\]
for all states \(s\) [2508.02948]. Robust CCE and CE are defined similarly, with the CE notion using modification operators \(\phi\in\Phi_i\) applied to the recommended action [2508.02948].

The literature also contains static antecedents that clarify the conceptual role of “nature” in distributional robustness. Static distributionally robust games treat nature as a virtual player choosing an adversarial distribution from an ambiguity set, and define equilibrium by worst-case expected loss or by coherent utility measures such as CVaR and mean-semideviation [1702.05371], [1610.00651], [2605.19302]. This suggests a useful interpretation of DRMGs: they are dynamic games in which the transition law itself is part of the strategic uncertainty set, and the environment acts as an adversarial player constrained by rectangularity and divergence geometry.

## 3. Bellman equations and structural assumptions

A central technical object is the robust Bellman equation. In the single-agent average-reward robust MDP, for a fixed policy \(\pi\),
\[
V(s)+g = \sum_a \pi(a|s)\big(r(s,a)+\sigma_{P^a_s}(V)\big),
\]
where
\[
\sigma_{P^a_s}(V) \triangleq \min_{P\in P^a_s} \sum_{s'} P(s'|s,a)\,V(s').
\]
The optimal robust Bellman equation is
\[
V(s) = \max_a \big\{ r(s,a) - g + \sigma_{P^a_s}(V)\big\}.
\]
A key result is that, under irreducibility, both the fixed-policy and optimal robust Bellman equations have solutions; moreover, unlike the non-robust case, the solution \((g,V)\) is not unique up to a constant vector in general [2508.03136].

This single-agent analysis extends to Markov games through induced robust MDPs. Fixing player \(i\) and opponents’ policy \(\pi_{-i}\), one defines an induced robust MDP
\[
\mathcal{M}(MG,i,\pi_{-i}),
\]
whose transition ambiguity set averages the original joint-action uncertainty over \(\pi_{-i}\), and whose reward is the corresponding expected stage reward to player \(i\) [2508.03136]. The induced fixed-policy Bellman equation becomes
\[
V_i(s) + g_{P,i}^{(\pi_{-i},\pi_i)} = \sum_{a_i}\pi_i(a_i\mid s) \big( r^{\pi_{-i}}(s,a_i) + \sigma_{(P^{\pi_{-i}})^a_s}(V_i) \big),
\]
with a corresponding optimal robust Bellman equation for the best response [2508.03136]. This induced-MDP construction is the main bridge from robust dynamic programming to multi-agent equilibrium theory.

For finite-horizon robust Markov games, robust Bellman operators take the form of worst-case expectations of continuation values. In the online-learning formulation,
\[
E_{\mathcal{U}}[V] \triangleq \inf_{P\in\mathcal{U}} \mathbb{E}_P[V],
\]
and the robust Bellman step uses these support-function-like operators inside optimistic or pessimistic value iteration [2508.02948]. In the linear-function-approximation setting with \(d\)-rectangular TV uncertainty, the robust Bellman backup admits a coordinatewise dual form:
\[
\RQ[h](s,a) = r_{i,h}(s,a)+ \Big\langle \phi(s,a),\ \big[\max_\alpha \{\nu_{i,h,j}(\alpha)-\sigma_i\alpha\}\big]_{j=1}^d \Big\rangle,
\]
which preserves linearity of the robust value function in the feature map \(\phi\) [2511.07831]. That preservation is essential for large-state-space learning.

These Bellman constructions rely on structural assumptions. The average-reward existence and solvability results assume irreducibility: for any deterministic policy and any transition kernel in the ambiguity set, the induced Markov chain is irreducible; compactness and convexity of the ambiguity set are also assumed [2508.03136]. Finite-horizon online-learning results often require either support-preserving uncertainty, such as KL ambiguity under a positive minimum transition probability assumption, or additional “failure state” or “minimum value” conditions to neutralize support-shift pathologies under TV ambiguity [2508.02948], [2511.07831]. The underlying single-agent robust-MDP theory emphasizes that rectangularity is what links dynamic consistency, Bellman recursion, and strong duality [2308.11139].

## 4. Existence theory and relations among formulations

The average-reward theory in [2508.03136] establishes existence of stationary robust Nash equilibria by applying Kakutani’s fixed point theorem to the robust best-response correspondence. The proof verifies three properties: nonemptiness of \(\text{BR}_i(\pi_{-i})\), convexity of the optimal robust policy set in induced robust MDPs, and upper semicontinuity of the best-response map under convergence of opponents’ policies [2508.03136]. The resulting theorem states that for an average-reward distributionally robust Markov game satisfying irreducibility, there exists a stationary robust Nash equilibrium [2508.03136].

Finite-horizon robust games admit parallel well-posedness results. In the policy-induced \((s,a_i)\)-rectangular model, robust best responses exist for every fixed \(\pi_{-i}\), and a robust NE exists for the full game by recasting the problem as a hierarchical game with the original agents and local adversarial transition selectors, then invoking Kakutani’s theorem on the combined best-response correspondence [2409.20067]. Robust CCE existence follows because every NE is also a CCE in that framework [2409.20067].

The static literature adds another layer of perspective. Distributionally robust games with ambiguity sets over payoff distributions reduce, under special cases, to complete-information Nash games, Bayesian games, or robust games, depending on the structure of the ambiguity set and whether players are risk-neutral [1610.00651], [1512.03253]. In particular, when the ambiguity collapses to a singleton or to a fixed mean payoff matrix, the equilibrium set coincides with that of an ordinary Nash game [1610.00651]. This suggests that DRMGs similarly interpolate between ordinary Markov games, Bayesian or partially specified dynamic games, and robust control/game formulations, depending on whether uncertainty is degenerate, probabilistically specified, or only set-valued.

The relation between static and dynamic robust formulations is especially sharp in robust MDP theory. Under rectangularity and suitable convexity of state-wise marginals, the dynamic game formulation and the static robust formulation are equivalent and satisfy strong duality; without such structure, duality gaps can appear and dynamic programming can cease to reflect the static problem faithfully [2308.11139]. A plausible implication is that DRMG formulations inherit the same sensitivity: equilibrium existence alone does not guarantee that a static worst-case kernel interpretation, a Bellman recursion, and a learning objective all coincide unless the ambiguity set is dynamically consistent.

## 5. Algorithms for equilibrium computation and learning

The average-reward setting in [2508.03136] introduces **Robust Nash-Iteration**, a value-iteration-like algorithm for computing a stationary robust NE. The algorithm maintains bias estimates \(h_i:S\to\mathbb{R}\) for each player, repeatedly computes robust state-action values
\[
Q_i(s,a,h_i^0) \leftarrow r_i(s,a) + \sigma_{\mathcal{P}^a_s}(h_i^0),
\]
solves at each state a stage game with payoffs \(\{Q_i(s,a,h_i)\}_i\), and updates the bias by the expected equilibrium payoff
\[
h_i(s) \leftarrow \mathbb{E}_{a\sim\pi(s)}[Q_i(s,a,h_i^0)].
\]
The stopping rule is
\[
\max_i\left\{\text{sp}(h_i - h^0_i)\right\} = 0,
\]
where \(\text{sp}(v)=\max_s v(s)-\min_s v(s)\) is the span seminorm [2508.03136]. Under irreducibility and an equilibrium-selection rule ensuring either global optimality or a saddle-point condition in each stage game, the associated operator is a contraction in span seminorm and the algorithm converges to a stationary robust NE [2508.03136].

In the finite-horizon, generative-model setting, **DRNVI** learns robust equilibrium notions such as robust NE, CCE, and CE with finite-sample guarantees [2404.18909]. The broad pattern is model-based robust Nash value iteration: estimate the nominal transition model from a non-adaptive generative sampling scheme, form empirical robust Bellman operators, and compute equilibrium policies by backward induction under those robust operators [2404.18909]. The paper also provides an information-theoretic lower bound and states that the finite-sample complexity of DRNVI is near-optimal with respect to problem-dependent factors such as the size of the state space, the target accuracy, and the horizon length [2404.18909].

In the online interactive setting, **RONAVI** maintains empirical kernels
\[
\widehat{P}_h^k(s'|s,{\bf a}) = \frac{N_h^k(s,{\bf a},s')}{N_h^k(s,{\bf a})\vee 1},
\]
then computes optimistic and pessimistic robust \(Q\)-values
\[
\overline{Q}_{i,h}^{k,\sigma_i}(s, \bm{a}) = \min\Big\{ r_{i,h}(s,\bm{a}) + E_{\widehat{\mathcal{U}_{i,h,f}^{\sigma_i}(s, \bm{a})}} [\overline{V}_{i,h+1}^{k, \sigma_i}] + \beta^k_{i,h,f}(s,\bm{a}),\ H \Big\},
\]
\[
\underline{Q}_{i,h}^{k, \sigma_i}(s, \bm{a}) = \max\Big\{ r_{i,h}(s,\bm{a}) + E_{\widehat{\mathcal{U}_{i,h,f}^{\sigma_i}(s, \bm{a})}}[\underline{V}_{i,h+1}^{k, \sigma_i}] - \beta^k_{i,h,f}(s,\bm{a}),\ 0 \Big\},
\]
selects a stage-game equilibrium of the optimistic payoffs, and executes the resulting joint policy in the nominal environment to gather data [2508.02948]. The bonuses are tailored to the geometry of TV or KL uncertainty. For TV, the paper gives a variance-adaptive Bernstein-type bonus; for KL, it gives a bonus depending on the smallest positive empirical transition probability and the uncertainty radius [2508.02948]. Under the relevant assumptions, RONAVI achieves sublinear regret and returns \(\varepsilon\)-robust NE, CCE, or CE via online-to-batch conversion [2508.02948].

For large state spaces, **DR-CCE-LSI** develops an online least-squares value-iteration method with linear function approximation. It assumes a linear Markov game with feature map \(\phi:\mathcal{S}\times\mathcal{A}\to\mathbb{R}^d\), \(d\)-rectangular TV uncertainty in feature space, and defines robust values by taking the infimum over the uncertainty set [2511.07831]. The robust Bellman dual is estimated by ridge regression for each clipping level \(\alpha\), followed by an exploration bonus
\[
\Gamma^i_{h,k}(s,a) = \beta_i \sum_{j=1}^d \sqrt{\phi_j(s,a)\,\mathbf{1}_j^\top (\Lambda_h^k)^{-1} \mathbf{1}_j\,\phi_j(s,a)},
\]
and a **Find-CCE** subroutine that handles the non-Lipschitz dependence of CCE on payoffs through an \(\epsilon\)-cover argument [2511.07831].

A related large-state-space development studies robust linear Markov games with per-agent independent features for the nominal model and proposes generative-model and online algorithms based on ridge regression, supporting-set sampling, robust \(Q\)-updates under TV ambiguity, and FTRL policy updates [2605.03125]. In the generative-model case, the algorithm outputs an \(\varepsilon\)-robust CCE with total sample complexity
\[
N_{\mathrm{all}} = \tilde{\mathcal{O}\big(H^9 d^3/\varepsilon^4\big),
\]
while in the online adversarial setting the regret bound is
\[
\mathrm{Regret}(T) \le \tilde{\mathcal{O}\big(H^2 d \,\max_{i\in[n]} A_i\, \sqrt{T}\big),
\]
and the uniform mixture of the policies is an \(\varepsilon\)-robust CCE once \(T\) is large enough [2605.03125]. These results explicitly target robust Markov games with large or infinite state spaces.

## 6. Learnability, hardness, limitations, and broader context

A recurring theme is that robustness amplifies both statistical and computational difficulty. In online robust multi-agent RL with agent-wise \(f\)-divergence uncertainty sets, the paper first proves hardness results. With support shift, such as TV ambiguity, there exists a DRMG for which any online algorithm incurs linear regret
\[
\Omega\Big(\sigma\, K \cdot \min\{H,\ \prod_{i\in\mathcal M} A_i\}\Big),
\]
and even without support shift, as in the KL case, there is an \(\Omega(\sqrt{K\prod_i A_i})\) lower bound [2508.02948]. These results expose a curse of multiagency in the online robust setting and motivate the structural restrictions used by later curse-breaking methods.

The large-state-space line sharpens this further. In online robust Markov games with linear function approximation and \(d\)-rectangular TV uncertainty, there is a fundamental regret lower bound of \(\Omega(\sigma H K)\) unless one imposes a vanishing minimal value assumption, implemented by adding an absorbing fail state with zero robust value [2511.07831]. Under that assumption, worst-case kernels become absolutely continuous with respect to the nominal support, which eliminates the support-shift pathology and restores learnability [2511.07831]. This suggests that certain impossibility results are not merely algorithmic artifacts but stem from the interaction between adversarial transition reweighting and partial observability of low-probability states under nominal dynamics.

At the same time, recent work shows that the curse of multiagency is not intrinsic to all DRMG formulations. In the policy-induced \((s,a_i)\)-rectangular model, the robust CCE sample complexity scales polynomially with \(\sum_i A_i\) rather than exponentially with \(\prod_i A_i\) [2409.20067]. In robust linear Markov games with per-agent features, both generative-model and online results replace \(\prod_i A_i\) dependence by polynomial dependence on \(\max_i A_i\) and the feature dimension \(d\) [2605.03125]. This indicates that the formulation of the ambiguity set is itself a statistical design choice: it determines whether robustness preserves or destroys the decomposability needed for scalable multi-agent learning.

Several limitations recur across the literature. Most results are tabular or linear; function approximation beyond linear structure is largely open [2511.07831], [2605.03125]. Many algorithms assume a generative model or access to robustified simulators [2404.18909], [2409.20067], [2605.03125]. Average-reward theory is currently established under irreducibility, with unichain extensions discussed but full generality still open [2508.03136]. Stage-game equilibrium computation remains a bottleneck: Nash equilibrium computation is PPAD-complete in general, and arbitrary equilibrium selection can destroy convergence, which is why average-reward Robust Nash-Iteration imposes a strong equilibrium-selection assumption [2508.03136]. More generally, static distributionally robust games with coherent risk measures are PPAD-complete in general and require continuous-game rather than finite-matrix-game equilibrium notions [2605.19302].

Finally, DRMGs sit at the intersection of several broader research lines: robust and distributionally robust MDPs, static distributionally robust games, stochastic games, and safety-oriented dynamic games with adversarial uncertainty. Static \(f\)-divergence and CVaR-based game models provide the duality, ambiguity-set, and coherent-risk vocabulary [1702.05371], [1610.00651], [2605.19302]. Robust MDP theory provides the rectangularity and duality principles linking Bellman recursion and game formulations [2308.11139]. Safety-oriented dynamic games with moment-based ambiguity over disturbances show how adversarial distribution selection can be reformulated via duality into tractable stagewise programs [1701.06260]. Within this landscape, DRMGs provide the dynamic, multi-agent counterpart: a framework for strategic decision making under transition uncertainty, in which each agent optimizes worst-case long-run or episodic performance and equilibrium analysis must coexist with adversarial dynamic programming [2508.03136], [2404.18909].

Source: https://www.emergentmind.com/topics/distributionally-robust-markov-games-drmgs