Robust Optimistic Nash Value Iteration (RONAVI)

• The paper introduces RONAVI, an algorithm that combines robust dynamic programming, optimism, and equilibrium computation for multi-agent learning under uncertainty.
• It leverages UCB-style bonuses and empirical model estimation to address distributional robustness using TV and KL divergence measures.
• RONAVI achieves provable regret guarantees and sample complexity bounds, highlighting its theoretical impact on robust multi-agent reinforcement learning.

to=sh code: $\mathcal{S} = \{1,\dots,S\},$18 to=shell code: $\mathcal{S} = \{1,\dots,S\},$19 to=shell code: $\mathcal{S} = \{1,\dots,S\},$20 to=shell code: $\mathcal{S} = \{1,\dots,S\},$21 Robust Optimistic Nash Value Iteration (RONAVI) is a model-based online algorithm for learning robust equilibria in finite-horizon distributionally robust Markov games (DRMGs) under model uncertainty. It is introduced in “Online Robust Multi-Agent Reinforcement Learning under Model Uncertainties” (Farhat et al., 4 Aug 2025), which studies multi-agent systems that must learn directly from environmental interaction, without a generative model and without large offline datasets, while optimizing worst-case performance over uncertainty sets defined around an unknown nominal transition kernel. In that formulation, RONAVI combines robust dynamic programming, optimism-in-the-face-of-uncertainty, and equilibrium computation to obtain provable regret guarantees for uncertainty sets measured by Total Variation divergence and Kullback–Leibler divergence.

1. Problem formulation in distributionally robust Markov games

The underlying environment is a standard finite-horizon Markov game with multiple agents. The agent set is

$\mathcal{M} = \{1,\dots,m\},$

the finite state space is

$\mathcal{S} = \{1,\dots,S\},$

and each agent $i$ has action space

$\mathcal{A}_i = \{1,\dots,A_i\},$

with joint action space

$$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$

A joint action is written ${\bf a} = (a_1,\dots,a_m)$. The horizon is finite, $H \in \mathbb{N}$, with stages $h=1,\dots,H$. Rewards are possibly non-stationary and deterministic,

$$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$

The nominal transition kernel is unknown: $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$

The paper frames model mismatch as an explicit uncertainty model. In applications such as robotics, autonomous driving, and multi-robot teams, the training and deployment environments may differ because of unmodeled dynamics, sensor noise, adversarial perturbations, or domain shift. In the multi-agent setting, the paper emphasizes that such perturbations can be amplified because one agent’s deviation changes the environment experienced by the others (Farhat et al., 4 Aug 2025).

A DRMG augments the Markov game with agent-specific uncertainty sets of transition kernels. The robust game is written as

$\mathcal{S} = \{1,\dots,S\},$0

For each agent $\mathcal{S} = \{1,\dots,S\},$1, the uncertainty set is agent-wise $\mathcal{S} = \{1,\dots,S\},$2-rectangular and defined through an $\mathcal{S} = \{1,\dots,S\},$3-divergence: $\mathcal{S} = \{1,\dots,S\},$4 where

$\mathcal{S} = \{1,\dots,S\},$5

with

$\mathcal{S} = \{1,\dots,S\},$6

The paper specializes to two divergences: $\mathcal{S} = \{1,\dots,S\},$7 and

$\mathcal{S} = \{1,\dots,S\},$8

The corresponding local TV and KL balls are

$\mathcal{S} = \{1,\dots,S\},$9

and

$i$0

Here $i$1 is the uncertainty radius of agent $i$2, and $i$3 enters the regret and sample-complexity bounds.

2. Robust values, best responses, and equilibrium notions

The robust objective is max–min. For a possibly stochastic, time-dependent policy

$i$4

and joint policy

$i$5

agent $i$6’s robust value from stage $i$7 and state $i$8 is

$i$9

Equivalently, agent $\mathcal{A}_i = \{1,\dots,A_i\},$0 solves

$\mathcal{A}_i = \{1,\dots,A_i\},$1

Given opponents’ policy $\mathcal{A}_i = \{1,\dots,A_i\},$2, the robust best response is

$\mathcal{A}_i = \{1,\dots,A_i\},$3

with corresponding robust best-response value

$\mathcal{A}_i = \{1,\dots,A_i\},$4

The paper defines three robust equilibrium concepts through gap functions. A product policy $\mathcal{A}_i = \{1,\dots,A_i\},$5 is an $\mathcal{A}_i = \{1,\dots,A_i\},$6-robust Nash equilibrium if, for any state $\mathcal{A}_i = \{1,\dots,A_i\},$7,

$\mathcal{A}_i = \{1,\dots,A_i\},$8

A possibly correlated joint policy is an $\mathcal{A}_i = \{1,\dots,A_i\},$9-robust coarse correlated equilibrium when the same best-response gap criterion is imposed but $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$0 may be correlated. A policy $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$1 is an $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$2-robust correlated equilibrium if

$$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$3

where $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$4 is a strategy-modification map from recommended actions to alternative actions.

The online setting proceeds over episodes $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$5. In episode $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$6, the environment samples $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$7, agents choose a joint policy $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$8 based on past data, and then execute that policy in the nominal environment: $$\mathcal{A} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_m.$$9 After observing rewards and transitions, the agents update their estimates. Robust regret is defined as

${\bf a} = (a_1,\dots,a_m)$0

If regret is ${\bf a} = (a_1,\dots,a_m)$1, the learning dynamics converge to a robust equilibrium notion of the corresponding type (Farhat et al., 4 Aug 2025).

3. Robust Bellman structure and the RONAVI update mechanism

The robust ${\bf a} = (a_1,\dots,a_m)$2-value for agent ${\bf a} = (a_1,\dots,a_m)$3 under joint policy ${\bf a} = (a_1,\dots,a_m)$4 is

${\bf a} = (a_1,\dots,a_m)$5

The paper expresses the robust Bellman recursion through the robust support functional

${\bf a} = (a_1,\dots,a_m)$6

Then

${\bf a} = (a_1,\dots,a_m)$7

and

${\bf a} = (a_1,\dots,a_m)$8

RONAVI does not introduce a new explicit Bellman operator form in the main text; instead, it maintains optimistic and pessimistic estimates of robust values and performs a value-iteration-like backward sweep. The high-level structure is:

1. Learn an empirical nominal model ${\bf a} = (a_1,\dots,a_m)$9 from online trajectory data.
2. Maintain upper and lower confidence bounds

$H \in \mathbb{N}$0

and $H \in \mathbb{N}$1.

1. At each state and stage, form a finite normal-form stage game with optimistic payoffs $H \in \mathbb{N}$2.
2. Compute an equilibrium of that stage game—NE, CCE, or CE—and use it as the episode policy.
3. Execute the policy, collect additional transitions, and repeat.

The algorithm is explicitly model-based, tabular, and purely online. It assumes no generative model and no large offline dataset. Empirical transitions are formed from counts

$H \in \mathbb{N}$3

$H \in \mathbb{N}$4

and

$H \in \mathbb{N}$5

The backward induction initializes

$H \in \mathbb{N}$6

For each $H \in \mathbb{N}$7, the estimated robust expectation under the empirical uncertainty set is

$H \in \mathbb{N}$8

The optimistic and pessimistic robust $H \in \mathbb{N}$9-updates are

$h=1,\dots,H$0

$h=1,\dots,H$1

For each state $h=1,\dots,H$2, RONAVI calls a black-box equilibrium solver: $h=1,\dots,H$3 and updates the state values by

$h=1,\dots,H$4

$h=1,\dots,H$5

The role of optimism is explicit: the bonus $h=1,\dots,H$6 is chosen so that, with high probability,

$h=1,\dots,H$7

This produces exploration by upper-confidence planning while preserving the robust max–min structure (Farhat et al., 4 Aug 2025).

4. TV and KL instantiations

The paper presents RONAVI as a meta-framework for general $h=1,\dots,H$8-divergence uncertainty sets, but gives explicit instantiations for TV and KL uncertainty. The two cases differ in their assumptions and bonus design.

For TV-divergence, under a standard failure-state assumption used to handle support shift, the exploration bonus is

$h=1,\dots,H$9

where

$$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$0

and $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$1 are absolute constants. The paper characterizes this as a variance-aware empirical Bernstein style bonus plus a term depending on the width $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$2, together with a $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$3 term.

For KL-divergence, the paper assumes a minimal-mass condition: there exists $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$4 such that whenever $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$5, it is at least $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$6. The bonus becomes

$$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$7

where $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$8 depends on $$r = \{r_{i,h}\}_{1\le i \le m,\,1\le h \le H}, \qquad r_{i,h} : \mathcal{S}\times\mathcal{A} \to [0,1].$$9, and

$$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$0

The paper notes that this bonus grows with $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$1 and inversely with sample size and minimal probability mass.

Instantiation	Assumption	Distinctive feature
RONAVI-TV	Failure states assumption (Assumption 1)	Variance-aware empirical Bernstein style bonus
RONAVI-KL	Minimal mass assumption (Assumption 2)	Bonus depends on $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$2 and $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$3

At each state and stage, RONAVI must solve a finite normal-form game over the joint action space $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$4, with payoffs $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$5. The paper states that NE computation may be PPAD-hard in general, while CE and CCE are computable via linear programming in polynomial time. As a result, CE and CCE are the more tractable equilibrium notions from a computational perspective in this framework (Farhat et al., 4 Aug 2025).

5. Hardness results, regret bounds, and sample complexity

The theoretical contribution begins with lower bounds. With support shift—for example, under TV uncertainty sets that can admit worst-case kernels whose support is not fully covered by the nominal kernel—the paper states that there exists a DRMG such that any online algorithm suffers

$$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$6

Thus, in that setting, regret can be necessarily linear in $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$7.

Even without support shift—for example, under KL uncertainty—there exist DRMGs such that any algorithm has

$$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$8

These lower bounds isolate an inherent dependence on the joint action space size $$P^\star : [H]\times \mathcal{S}\times\mathcal{A} \to \Delta(\mathcal{S}), \qquad P_h^\star(s' \mid s,{\bf a}).$$9 in online robust multi-agent learning.

Under the failure-states assumption, Theorem 1 gives the regret bound for RONAVI-TV. For any $\mathcal{S} = \{1,\dots,S\},$00, with probability at least $\mathcal{S} = \{1,\dots,S\},$01,

$\mathcal{S} = \{1,\dots,S\},$02

Equivalently, the per-episode regret is

$\mathcal{S} = \{1,\dots,S\},$03

Under the minimal-mass assumption, Theorem 2 gives the regret bound for RONAVI-KL. For any $\mathcal{S} = \{1,\dots,S\},$04, with probability at least $\mathcal{S} = \{1,\dots,S\},$05,

$\mathcal{S} = \{1,\dots,S\},$06

The paper notes that the exponential in $\mathcal{S} = \{1,\dots,S\},$07 is typical in KL-robustness analyses.

The online-to-batch conversion yields sample-complexity corollaries. To obtain an $\mathcal{S} = \{1,\dots,S\},$08-equilibrium of type NE, CCE, or CE with probability at least $\mathcal{S} = \{1,\dots,S\},$09, the required number of episodes is

$\mathcal{S} = \{1,\dots,S\},$10

and

$\mathcal{S} = \{1,\dots,S\},$11

The paper further summarizes the proof strategy through four components: concentration of empirical robust Bellman operators, confidence intervals induced by the bonuses, optimism combined with equilibrium computation, and backward propagation of value-estimation error along the horizon. The analysis is technically nontrivial because robust expectations are nonlinear in $\mathcal{S} = \{1,\dots,S\},$12, each agent has its own uncertainty set, and equilibrium computation must be integrated into the dynamic-programming argument (Farhat et al., 4 Aug 2025).

6. Position within the literature and practical scope

The paper places RONAVI at the intersection of distributionally robust reinforcement learning and online multi-agent reinforcement learning. Earlier work on DRMDPs and DRMGs, as described in the paper, mostly assumes either access to a generative model or the availability of offline data with coverage coefficients. Relative to that literature, the authors state that RONAVI is the first provably efficient online learning algorithm for general-sum DRMGs learned directly from interactive data, without simulators or large offline datasets (Farhat et al., 4 Aug 2025).

The paper also compares RONAVI with prior robust RL formulations. It states that works such as Shi et al. (2024) and Jiao et al. (2024) consider DRMGs with a generative model, and that offline DRRL with TV or KL uncertainty, including Blanchet et al. (2023) and Li et al. (2025), relies on coverage coefficients of an offline dataset. Within that comparison, RONAVI’s complexity is described as comparable to existing generative or offline results in all parameters except the explicit $\mathcal{S} = \{1,\dots,S\},$13 dependence, which the paper argues may be fundamental in the online robust multi-agent setting.

Several practical limitations are explicit. RONAVI is model-based and tabular: it estimates the full transition kernel $\mathcal{S} = \{1,\dots,S\},$14 and maintains value estimates for all $\mathcal{S} = \{1,\dots,S\},$15. The framework therefore applies directly only to moderate-size tabular problems. Equilibrium computation is itself nontrivial: NE computation is PPAD-hard in general, whereas CE and CCE admit polynomial-time linear-programming formulations, although the joint action space $\mathcal{S} = \{1,\dots,S\},$16 still scales exponentially in the number of agents. The algorithm also assumes that the uncertainty radii $\mathcal{S} = \{1,\dots,S\},$17 and the divergence type—TV or KL—are known in advance. Finally, the paper contains no experiments and is primarily theoretical.

Within those boundaries, the significance of RONAVI lies in its synthesis of three ingredients: explicit distributional robustness through agent-specific uncertainty sets, UCB-style optimism for online exploration, and equilibrium selection at each state of a stochastic game. A plausible implication is that the framework serves as a template for future work on online robust MARL with richer function classes, although the paper itself confines its formal treatment to the tabular setting.