Supremal Visitation Ratio in Robust RL

• Supremal Visitation Ratio is a complexity parameter that quantifies how worst-case dynamics can amplify state visitations relative to nominal training distributions in RL.
• It underpins the analysis of robust reinforcement learning by linking adversarial state concentration to statistical estimation challenges and regret bounds.
• The parameter distinguishes between bounded regimes, where standard exploration suffices, and unbounded scenarios that render learning exponentially hard.

The supremal visitation ratio is a complexity parameter introduced in the analysis of robust and off-dynamics reinforcement learning (RL) in Markov decision processes (MDPs) with mismatches between training and deployment dynamics. It quantifies the maximal amplification that adversarial or worst-case transition dynamics can effect in state visitation, relative to the corresponding nominal distribution, under any policy. This ratio arises as a sharp threshold in characterizing the statistical and algorithmic hardness of robust RL when exploration is limited to online interaction with the training environment, and the goal is to perform reliably under a family of perturbed deployment environments (He et al., 7 Nov 2025).

1. Formal Definition

Let $(\mathcal{S}, \mathcal{A}, H)$ denote the finite state, action, and horizon spaces. For any (possibly history-dependent) policy $\pi$, define the nominal state-visitation measure at step $h$ under the training transition kernel $P^o$ by

$d_h^\pi(s) = \Pr\big(s_h = s \mid \pi, P^o\big).$

Let $q_h^\pi(s)$ denote the counterpart under a worst-case kernel $P^{w,\pi}$ chosen adversarially from an uncertainty set $\mathcal{U}^\rho(P^o)$ or regularized via a parameter $\beta$: $$q_h^\pi(s) = \Pr\big(s_h = s \mid \pi, P^{w,\pi}\big).$$ The supremal visitation ratio is defined as

$$\kappa := \sup_{\pi \in \Pi} \;\sup_{1 \leq h \leq H}\; \sup_{s \in \mathcal{S}}\; \frac{q_h^\pi(s)}{d_h^\pi(s) \vee \epsilon}$$

for arbitrarily small $\epsilon > 0$ to avoid division by zero. In the absence of robustness requirements ($\rho=0$ or $\beta \to \infty$), $q_h^\pi \equiv d_h^\pi$ and thus $\kappa = 1$ (He et al., 7 Nov 2025).

2. Interpretive Significance and Intuition

The supremal visitation ratio $\kappa$ reflects the worst-case factor by which an adversarial transition model can concentrate probability mass on any state $s$ at any step $h$ relative to how frequently $s$ is visited under the nominal dynamics. If states critical for the robust value objective are rarely encountered in training (small $d_h^\pi(s)$), but can be forced to high visitation in deployment (large $q_h^\pi(s)$), there is a severe information deficit: learning robust policies at those states becomes statistically difficult or infeasible.

The ratio $\frac{q_h^\pi(s)}{d_h^\pi(s)}$ forms the basic control for estimation error and regret in robust RL. The maximization over all policies, steps, and states ensures that $\kappa$ captures the most severe case across the entire interaction protocol (He et al., 7 Nov 2025).

3. Regimes: Boundedness and Implications

The statistical and computational tractability of robust RL is sharply controlled by the magnitude of $\kappa$:

• Bounded regime ($\kappa \leq \mathrm{poly}(S,A,H)$): In structured robust MDPs, such as those with total-variation uncertainty and fail-states (where worst-case dynamics can only redirect transitions among low-reward states), it is often the case that $q_h^\pi(s) \leq d_h^\pi(s)$ so $\kappa \leq 1$. This implies standard RL exploration suffices for robustness.
• Unbounded or exponential regime ($\kappa \to \infty$): There exist worst-case constructions (e.g., critical states with exponentially small nominal visitation and large adversarial boosts) where $\kappa \approx e^{cA}$. In such cases, no algorithm can achieve sublinear regret; learning is exponentially hard in $A$ (He et al., 7 Nov 2025).

Illustrative examples include:

• Fail-state TV-CRMDP: $\kappa=1$, learning complexity matches standard RL.
• Toy exponential gap: $A$ actions, nominal probability $p=2^{-2A}$ to critical state, but worst-case can push up to $\widetilde{p}\approx \frac12$; thus $\kappa \approx 2^{2A}$, and polynomial-sample RL is infeasible.

4. Appearance in Regret Bounds

The supremal visitation ratio $\kappa$ arises as the central complexity parameter in non-asymptotic upper and lower bounds for regret in robust RL with online exploration: $$\mathrm{Regret}(K) = \widetilde{O}\big( \kappa\, \mathrm{poly}(S,A,H) + \sqrt{\kappa}\,\mathrm{poly}(S,A,H)\, \sqrt{K} \big)$$ where $K$ is the number of episodes, and the precise polynomial in $(S, A, H)$ depends on the divergence and uncertainty model (e.g., total variation, KL, $\chi^2$). For constrained robust MDPs (CRMDP) with TV divergence,

$$\mathrm{Regret}(K) = \widetilde{O}\bigl( \kappa S^2A H^2 + \sqrt{\kappa} S^{3/2}A^{1/2} H^2\sqrt{K}\bigr).$$

Matching information-theoretic lower bounds (based on bandit-style constructions) show

$$\inf_\xi\;\sup_{\mathcal{M}: \kappa(\mathcal{M}) \leq \kappa} \mathbb{E}[\mathrm{Regret}_\mathcal{M}(\xi, K)] = \Omega\big(\sqrt{\kappa K}\big),$$

and thus the dependence on $\kappa$ is unavoidable in general (He et al., 7 Nov 2025).

5. Relation to Other Coverage and Divergence Measures

The supremal visitation ratio $\kappa$ is analogous to the concentrability coefficient in batch or offline RL, where coverage is measured with respect to a static data-collection distribution $\mu$: $C = \sup_{\pi, h} \frac{d_h^\pi(s)}{\mu(s)}.$ In robust RL, $\kappa$ generalizes this to an online, worst-case setting, with the roles of $\mu$ and $d_h^\pi$ replaced by $d_h^\pi$ and $q_h^\pi$, respectively. Divergence-based uncertainty set radii (e.g., total variation $\rho$, regularization parameter $\beta$) further enter regret bounds as multiplicative or additive factors (He et al., 7 Nov 2025).

6. Proof Framework and Analytical Role

Analysis of algorithms attaining optimal dependence on $\kappa$ uses:

• Dual-form Bellman operators and optimistic value iteration over uncertainty sets for closed-form minimax solutions.
• Conditional bonus-based estimation of $Q$-values, with sample counts $n_h(s,a)$ and weighting by $q_h^\pi(s)$.
• Martingale-based bounds linking empirical counts in the nominal environment to required sample support for robust deployment, with occurrences of $\kappa$ arising when worst-case visitation outstrips nominal sample coverage.
• Lower bounds via change-of-measure and information-theoretic arguments (Bretagnolle–Huber lemma), ensuring tightness of $\sqrt{\kappa K}$ scaling in the worst case.

7. Research Context and Further Directions

The supremal visitation ratio was introduced to address the fundamental challenge of robust online RL where neither full generative access nor broad state coverage can be assumed. Unlike prior work, which often relegated robust exploration issues by such assumptions, this parameter demarcates the frontier between tractable and intractable regimes for sublinear regret. When $\kappa$ is bounded by a polynomial in problem parameters, robust RL is statistically and computationally feasible; when it is exponential, exploration becomes bottlenecked and no efficient learning is generally possible (He et al., 7 Nov 2025). This suggests further investigation into structure-exploiting exploration and efficient estimation in models where $\kappa$ is neither globally bounded nor globally large.