Distributionally Robust Markov Games

• Distributionally Robust Markov Games (DRMGs) are multi-agent frameworks that model uncertain transition kernels using ambiguity sets to optimize worst-case rewards.
• DRMGs employ robust performance criteria and adapted Bellman equations to establish equilibria, including robust Nash, coarse correlated, and correlated equilibria in both average-reward and finite-horizon settings.
• Advanced algorithms like Robust Nash-Iteration, DRNVI, and RONAVI are developed to compute equilibrium policies under uncertainty, addressing scalability and the curse of multiagency.

Searching arXiv for papers on distributionally robust Markov games and related robust multi-agent RL. Distributionally Robust Markov Games (DRMGs) are Markov games in which the transition kernel is not known exactly and is instead assumed to lie in an ambiguity set of plausible kernels; each agent then evaluates policies by worst-case performance over that set, rather than under a single nominal model. In the formulations developed recently, this framework supports both average-reward and finite-horizon analyses, robust analogues of Nash-type solution concepts, and algorithmic procedures for equilibrium computation and learning under model uncertainty (Roch et al., 5 Aug 2025, Shi et al., 2024, Farhat et al., 4 Aug 2025).

1. Formal model and uncertainty structure

A non-robust Markov game with $N$ agents is specified by

$$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$

where $S$ is the common state space, $A_i$ is the action space of agent $i$, $A=\otimes_i A_i$ is the joint action space, $r_i:S\times A\to\mathbb{R}$ is agent $i$’s reward, and $\mathsf P=\{p_s^a:s\in S,a\in A\}$ is the transition kernel (Roch et al., 5 Aug 2025). A stationary policy for agent $i$ is $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$0, and a joint policy is $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$1 (Roch et al., 5 Aug 2025).

In a distributionally robust Markov game, the kernel is uncertain. Instead of a single $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$2, one works with an ambiguity set

$$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$3

where each $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$4 is nonempty, convex, and compact (Roch et al., 5 Aug 2025). This is the standard $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$5-rectangular structure: the adversary chooses transitions independently per $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$6 (Roch et al., 5 Aug 2025). Compatible examples include KL balls and Wasserstein balls; in the average-reward experiments of (Roch et al., 5 Aug 2025), the local set is instantiated as

$$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$7

Several recent finite-horizon formulations refine this generic ambiguity model. One line uses agent-wise $$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$8-divergence uncertainty sets

$$(N,S,A=\otimes_{i\in N} A_i,\{r_i\}_{i\in N},\mathsf P),$$9

with Total Variation or Kullback–Leibler divergence balls around the nominal transition $S$0 (Farhat et al., 4 Aug 2025). Another line proposes policy-induced $S$1-rectangular “fictitious” uncertainty sets centered at the transition obtained by averaging the nominal kernel over other agents’ behavior,

$S$2

which reduces each agent’s problem, for fixed $S$3, to a robust MDP indexed by the agent’s own action $S$4 rather than by the full joint action $S$5 (Shi et al., 2024). This suggests that ambiguity-set design is not merely technical; it determines whether the multi-agent robust problem inherits the decomposability of robust MDPs or retains the full curse of joint action dimensionality.

A complementary line of work on distributionally robust MDPs emphasizes that rectangularity governs the relation between static robust formulations and dynamic game formulations, and delineates when strong duality and Bellman-style recursion remain valid (Li et al., 2023). That discussion is single-controller, but it directly informs DRMG modeling because the nature player in a robust Markov game plays the same role as the adversarial transition selector in a robust MDP (Li et al., 2023).

2. Robust performance criteria and solution concepts

For a stationary joint policy $S$6, the average reward of agent $S$7 in a non-robust Markov game is

$S$8

and the worst-case robust average reward is

$S$9

Under irreducibility, this robust average reward is independent of the initial state (Roch et al., 5 Aug 2025).

The average-reward robust Nash equilibrium is then defined by the inequality

$A_i$0

so each agent maximizes its own worst-case long-run average reward against adversarial transition selection from the ambiguity set (Roch et al., 5 Aug 2025). The corresponding robust best-response set is

$A_i$1

and a robust equilibrium is precisely a fixed point of the product correspondence $A_i$2 (Roch et al., 5 Aug 2025).

In finite-horizon robust multi-agent reinforcement learning, the central quantity is the robust value

$A_i$3

with robust $A_i$4-value defined analogously (Farhat et al., 4 Aug 2025). The robust best-response value is

$A_i$5

and several robust equilibrium notions follow by replacing ordinary values with robust ones (Farhat et al., 4 Aug 2025).

The most common finite-horizon notions are the robust $A_i$6-Nash equilibrium, robust $A_i$7-coarse correlated equilibrium, and robust $A_i$8-correlated equilibrium. For example, a product policy $A_i$9 is an $i$0-robust NE if

$i$1

for all states $i$2 (Farhat et al., 4 Aug 2025). Robust CCE and CE are defined similarly, with the CE notion using modification operators $i$3 applied to the recommended action (Farhat et al., 4 Aug 2025).

The literature also contains static antecedents that clarify the conceptual role of “nature” in distributional robustness. Static distributionally robust games treat nature as a virtual player choosing an adversarial distribution from an ambiguity set, and define equilibrium by worst-case expected loss or by coherent utility measures such as CVaR and mean-semideviation (Bauso et al., 2017, Loizou, 2016, Gangwani et al., 19 May 2026). This suggests a useful interpretation of DRMGs: they are dynamic games in which the transition law itself is part of the strategic uncertainty set, and the environment acts as an adversarial player constrained by rectangularity and divergence geometry.

3. Bellman equations and structural assumptions

A central technical object is the robust Bellman equation. In the single-agent average-reward robust MDP, for a fixed policy $i$4,

$i$5

where

$i$6

The optimal robust Bellman equation is

$i$7

A key result is that, under irreducibility, both the fixed-policy and optimal robust Bellman equations have solutions; moreover, unlike the non-robust case, the solution $i$8 is not unique up to a constant vector in general (Roch et al., 5 Aug 2025).

This single-agent analysis extends to Markov games through induced robust MDPs. Fixing player $i$9 and opponents’ policy $A=\otimes_i A_i$0, one defines an induced robust MDP

$A=\otimes_i A_i$1

whose transition ambiguity set averages the original joint-action uncertainty over $A=\otimes_i A_i$2, and whose reward is the corresponding expected stage reward to player $A=\otimes_i A_i$3 (Roch et al., 5 Aug 2025). The induced fixed-policy Bellman equation becomes

$A=\otimes_i A_i$4

with a corresponding optimal robust Bellman equation for the best response (Roch et al., 5 Aug 2025). This induced-MDP construction is the main bridge from robust dynamic programming to multi-agent equilibrium theory.

For finite-horizon robust Markov games, robust Bellman operators take the form of worst-case expectations of continuation values. In the online-learning formulation,

$A=\otimes_i A_i$5

and the robust Bellman step uses these support-function-like operators inside optimistic or pessimistic value iteration (Farhat et al., 4 Aug 2025). In the linear-function-approximation setting with $A=\otimes_i A_i$6-rectangular TV uncertainty, the robust Bellman backup admits a coordinatewise dual form: $A=\otimes_i A_i$7 which preserves linearity of the robust value function in the feature map $A=\otimes_i A_i$8 (Zheng et al., 11 Nov 2025). That preservation is essential for large-state-space learning.

These Bellman constructions rely on structural assumptions. The average-reward existence and solvability results assume irreducibility: for any deterministic policy and any transition kernel in the ambiguity set, the induced Markov chain is irreducible; compactness and convexity of the ambiguity set are also assumed (Roch et al., 5 Aug 2025). Finite-horizon online-learning results often require either support-preserving uncertainty, such as KL ambiguity under a positive minimum transition probability assumption, or additional “failure state” or “minimum value” conditions to neutralize support-shift pathologies under TV ambiguity (Farhat et al., 4 Aug 2025, Zheng et al., 11 Nov 2025). The underlying single-agent robust-MDP theory emphasizes that rectangularity is what links dynamic consistency, Bellman recursion, and strong duality (Li et al., 2023).

4. Existence theory and relations among formulations

The average-reward theory in (Roch et al., 5 Aug 2025) establishes existence of stationary robust Nash equilibria by applying Kakutani’s fixed point theorem to the robust best-response correspondence. The proof verifies three properties: nonemptiness of $A=\otimes_i A_i$9, convexity of the optimal robust policy set in induced robust MDPs, and upper semicontinuity of the best-response map under convergence of opponents’ policies (Roch et al., 5 Aug 2025). The resulting theorem states that for an average-reward distributionally robust Markov game satisfying irreducibility, there exists a stationary robust Nash equilibrium (Roch et al., 5 Aug 2025).

Finite-horizon robust games admit parallel well-posedness results. In the policy-induced $r_i:S\times A\to\mathbb{R}$0-rectangular model, robust best responses exist for every fixed $r_i:S\times A\to\mathbb{R}$1, and a robust NE exists for the full game by recasting the problem as a hierarchical game with the original agents and local adversarial transition selectors, then invoking Kakutani’s theorem on the combined best-response correspondence (Shi et al., 2024). Robust CCE existence follows because every NE is also a CCE in that framework (Shi et al., 2024).

The static literature adds another layer of perspective. Distributionally robust games with ambiguity sets over payoff distributions reduce, under special cases, to complete-information Nash games, Bayesian games, or robust games, depending on the structure of the ambiguity set and whether players are risk-neutral (Loizou, 2016, Loizou, 2015). In particular, when the ambiguity collapses to a singleton or to a fixed mean payoff matrix, the equilibrium set coincides with that of an ordinary Nash game (Loizou, 2016). This suggests that DRMGs similarly interpolate between ordinary Markov games, Bayesian or partially specified dynamic games, and robust control/game formulations, depending on whether uncertainty is degenerate, probabilistically specified, or only set-valued.

The relation between static and dynamic robust formulations is especially sharp in robust MDP theory. Under rectangularity and suitable convexity of state-wise marginals, the dynamic game formulation and the static robust formulation are equivalent and satisfy strong duality; without such structure, duality gaps can appear and dynamic programming can cease to reflect the static problem faithfully (Li et al., 2023). A plausible implication is that DRMG formulations inherit the same sensitivity: equilibrium existence alone does not guarantee that a static worst-case kernel interpretation, a Bellman recursion, and a learning objective all coincide unless the ambiguity set is dynamically consistent.

5. Algorithms for equilibrium computation and learning

The average-reward setting in (Roch et al., 5 Aug 2025) introduces Robust Nash-Iteration, a value-iteration-like algorithm for computing a stationary robust NE. The algorithm maintains bias estimates $r_i:S\times A\to\mathbb{R}$2 for each player, repeatedly computes robust state-action values

$r_i:S\times A\to\mathbb{R}$3

solves at each state a stage game with payoffs $r_i:S\times A\to\mathbb{R}$4, and updates the bias by the expected equilibrium payoff

$r_i:S\times A\to\mathbb{R}$5

The stopping rule is

$r_i:S\times A\to\mathbb{R}$6

where $r_i:S\times A\to\mathbb{R}$7 is the span seminorm (Roch et al., 5 Aug 2025). Under irreducibility and an equilibrium-selection rule ensuring either global optimality or a saddle-point condition in each stage game, the associated operator is a contraction in span seminorm and the algorithm converges to a stationary robust NE (Roch et al., 5 Aug 2025).

In the finite-horizon, generative-model setting, DRNVI learns robust equilibrium notions such as robust NE, CCE, and CE with finite-sample guarantees (Shi et al., 2024). The broad pattern is model-based robust Nash value iteration: estimate the nominal transition model from a non-adaptive generative sampling scheme, form empirical robust Bellman operators, and compute equilibrium policies by backward induction under those robust operators (Shi et al., 2024). The paper also provides an information-theoretic lower bound and states that the finite-sample complexity of DRNVI is near-optimal with respect to problem-dependent factors such as the size of the state space, the target accuracy, and the horizon length (Shi et al., 2024).

In the online interactive setting, RONAVI maintains empirical kernels

$r_i:S\times A\to\mathbb{R}$8

then computes optimistic and pessimistic robust $r_i:S\times A\to\mathbb{R}$9-values

$i$0

$i$1

selects a stage-game equilibrium of the optimistic payoffs, and executes the resulting joint policy in the nominal environment to gather data (Farhat et al., 4 Aug 2025). The bonuses are tailored to the geometry of TV or KL uncertainty. For TV, the paper gives a variance-adaptive Bernstein-type bonus; for KL, it gives a bonus depending on the smallest positive empirical transition probability and the uncertainty radius (Farhat et al., 4 Aug 2025). Under the relevant assumptions, RONAVI achieves sublinear regret and returns $i$2-robust NE, CCE, or CE via online-to-batch conversion (Farhat et al., 4 Aug 2025).

For large state spaces, DR-CCE-LSI develops an online least-squares value-iteration method with linear function approximation. It assumes a linear Markov game with feature map $i$3, $i$4-rectangular TV uncertainty in feature space, and defines robust values by taking the infimum over the uncertainty set (Zheng et al., 11 Nov 2025). The robust Bellman dual is estimated by ridge regression for each clipping level $i$5, followed by an exploration bonus

$i$6

and a Find-CCE subroutine that handles the non-Lipschitz dependence of CCE on payoffs through an $i$7-cover argument (Zheng et al., 11 Nov 2025).

A related large-state-space development studies robust linear Markov games with per-agent independent features for the nominal model and proposes generative-model and online algorithms based on ridge regression, supporting-set sampling, robust $i$8-updates under TV ambiguity, and FTRL policy updates (Gai et al., 4 May 2026). In the generative-model case, the algorithm outputs an $i$9-robust CCE with total sample complexity

$\mathsf P=\{p_s^a:s\in S,a\in A\}$0

while in the online adversarial setting the regret bound is

$\mathsf P=\{p_s^a:s\in S,a\in A\}$1

and the uniform mixture of the policies is an $\mathsf P=\{p_s^a:s\in S,a\in A\}$2-robust CCE once $\mathsf P=\{p_s^a:s\in S,a\in A\}$3 is large enough (Gai et al., 4 May 2026). These results explicitly target robust Markov games with large or infinite state spaces.

6. Learnability, hardness, limitations, and broader context

A recurring theme is that robustness amplifies both statistical and computational difficulty. In online robust multi-agent RL with agent-wise $\mathsf P=\{p_s^a:s\in S,a\in A\}$4-divergence uncertainty sets, the paper first proves hardness results. With support shift, such as TV ambiguity, there exists a DRMG for which any online algorithm incurs linear regret

$\mathsf P=\{p_s^a:s\in S,a\in A\}$5

and even without support shift, as in the KL case, there is an $\mathsf P=\{p_s^a:s\in S,a\in A\}$6 lower bound (Farhat et al., 4 Aug 2025). These results expose a curse of multiagency in the online robust setting and motivate the structural restrictions used by later curse-breaking methods.

The large-state-space line sharpens this further. In online robust Markov games with linear function approximation and $\mathsf P=\{p_s^a:s\in S,a\in A\}$7-rectangular TV uncertainty, there is a fundamental regret lower bound of $\mathsf P=\{p_s^a:s\in S,a\in A\}$8 unless one imposes a vanishing minimal value assumption, implemented by adding an absorbing fail state with zero robust value (Zheng et al., 11 Nov 2025). Under that assumption, worst-case kernels become absolutely continuous with respect to the nominal support, which eliminates the support-shift pathology and restores learnability (Zheng et al., 11 Nov 2025). This suggests that certain impossibility results are not merely algorithmic artifacts but stem from the interaction between adversarial transition reweighting and partial observability of low-probability states under nominal dynamics.

At the same time, recent work shows that the curse of multiagency is not intrinsic to all DRMG formulations. In the policy-induced $\mathsf P=\{p_s^a:s\in S,a\in A\}$9-rectangular model, the robust CCE sample complexity scales polynomially with $i$0 rather than exponentially with $i$1 (Shi et al., 2024). In robust linear Markov games with per-agent features, both generative-model and online results replace $i$2 dependence by polynomial dependence on $i$3 and the feature dimension $i$4 (Gai et al., 4 May 2026). This indicates that the formulation of the ambiguity set is itself a statistical design choice: it determines whether robustness preserves or destroys the decomposability needed for scalable multi-agent learning.

Several limitations recur across the literature. Most results are tabular or linear; function approximation beyond linear structure is largely open (Zheng et al., 11 Nov 2025, Gai et al., 4 May 2026). Many algorithms assume a generative model or access to robustified simulators (Shi et al., 2024, Shi et al., 2024, Gai et al., 4 May 2026). Average-reward theory is currently established under irreducibility, with unichain extensions discussed but full generality still open (Roch et al., 5 Aug 2025). Stage-game equilibrium computation remains a bottleneck: Nash equilibrium computation is PPAD-complete in general, and arbitrary equilibrium selection can destroy convergence, which is why average-reward Robust Nash-Iteration imposes a strong equilibrium-selection assumption (Roch et al., 5 Aug 2025). More generally, static distributionally robust games with coherent risk measures are PPAD-complete in general and require continuous-game rather than finite-matrix-game equilibrium notions (Gangwani et al., 19 May 2026).

Finally, DRMGs sit at the intersection of several broader research lines: robust and distributionally robust MDPs, static distributionally robust games, stochastic games, and safety-oriented dynamic games with adversarial uncertainty. Static $i$5-divergence and CVaR-based game models provide the duality, ambiguity-set, and coherent-risk vocabulary (Bauso et al., 2017, Loizou, 2016, Gangwani et al., 19 May 2026). Robust MDP theory provides the rectangularity and duality principles linking Bellman recursion and game formulations (Li et al., 2023). Safety-oriented dynamic games with moment-based ambiguity over disturbances show how adversarial distribution selection can be reformulated via duality into tractable stagewise programs (Yang, 2017). Within this landscape, DRMGs provide the dynamic, multi-agent counterpart: a framework for strategic decision making under transition uncertainty, in which each agent optimizes worst-case long-run or episodic performance and equilibrium analysis must coexist with adversarial dynamic programming (Roch et al., 5 Aug 2025, Shi et al., 2024).