Variance-Regularized Pessimistic Off-Policy Learning

• The paper introduces a variance-regularized pessimistic off-policy objective that incorporates a data-driven variance penalty to reliably control risk under adaptive data collection.
• It leverages self-normalized maximal inequalities to yield uniform high-probability bounds and fast convergence rates, even with dependent and sequential data.
• The approach enhances off-policy reinforcement learning by automatically adjusting to local variance and model complexity, leading to more conservative and stable policy evaluation.

A variance-regularized pessimistic off-policy learning objective is a principled approach in statistical learning and reinforcement learning (RL) that integrates control of empirical variance with explicit pessimism to produce reliable, conservative policy evaluation and selection, especially under distribution shift, insufficient exploration, or adaptively collected data. The defining characteristic is the addition of an explicit, data-dependent variance penalty to the empirical risk or loss, resulting in learning objectives that adapt to sample instability and yield performance guarantees that are more robust than classical empirical risk minimization (ERM) or naive off-policy estimators. This construct is grounded in nonasymptotic concentration theory for empirical processes, with recent advances leveraging self-normalized inequalities for martingales to yield high-probability uniform deviations for general (including sequential) data and enabling excess risk guarantees and even fast rates of convergence under suitable low-variance or margin-type conditions.

1. Definition and Structure of the Variance-Regularized Pessimistic Objective

Given observed data $\{(x_t, a_t, Y_t, \varpi_t(a_t|x_t))\}_{t=1}^T$ collected under a (possibly adaptive) behavior policy $\varpi_t$, with possibly dependent $(x_t, a_t)$, and a function class $\mathcal{F}$ (e.g., stochastic policies, scoring functions), the standard (inverse propensity scored) empirical risk for $f \in \mathcal{F}$ is

$$\hat{R}_T(f) = \frac{1}{T} \sum_{t=1}^T \ell_t(f), \qquad \text{where} \quad \ell_t(f) = \frac{f(a_t \mid x_t)}{\varpi_t(a_t \mid x_t)} Y_t.$$

The variance-regularized pessimistic objective augments this empirical risk with a data-driven sample variance penalty: $$\hat{f}_T^\lambda = \arg\min_{f \in \mathcal{F}} \left\{ \hat{R}_T(f) + \lambda \left( \frac{\hat{\sigma}_T(f)^{1-p/2}}{\sqrt{T}} + \frac{\hat{\sigma}_T(f)^{-p}}{T} \right) \right\}$$ where $\hat{\sigma}_T(f)^2$ is the empirical variance of the loss sequence, $\lambda > 0$ is a regularization parameter set in accord with the maximal inequality, and $p \ge 0$ is a complexity exponent reflecting the effective sequential bracketing entropy of the function class (Girard et al., 17 Oct 2025).

This structure implements pessimism—minimizing a risk estimate uniformly upper-bounded with high probability over adaptively collected data—while adapting to empirical variance and class complexity. In off-policy RL, this typically involves similar variance penalization applied to importance-weighted temporal difference errors, value-function residuals, or counterfactual risk estimates.

2. Theoretical Principles: Self-Normalized Maximal Inequalities

Classical confidence bounds (e.g., empirical Bernstein) become invalid or loose when losses are adapted to previous feedback, as in adaptive experiments, reinforcement learning, or contextual bandits with adaptive exploration. The key innovation underlying modern variance-regularized pessimistic objectives is the development of self-normalized maximal inequalities for martingale empirical processes (Girard et al., 17 Oct 2025):

• For a potentially dependent sequence, given losses $\ell_t(f)$, the deviation $M_T(f) = \hat{R}_T(f) - R_T(f)$ admits a high-probability bound (for all $f \in \mathcal{F}$) of the form

$$|M_T(f)| \leq O\left( \frac{\hat{\sigma}_T(f)^{1-p/2}}{\sqrt{T}} + \frac{\hat{\sigma}_T(f)^{-p}}{T} \right),$$

up to polylogarithmic factors in $T$ and $1/\delta$, where $R_T(f)$ is the true (conditional) risk and $p$ quantifies sequential bracketing entropy.

• This bound is "self-normalized": it scales with the empirical variance $\hat{\sigma}_T(f)$, thus adapting to both the noisiness of the policy and the stability of the target.

As a result, the optimal choice of variance penalty in the pessimistic objective mirrors the rate allowed by the inequality, bestowing automatic calibration to local data conditions and effective model complexity.

3. Algorithmic Implementation and Off-Policy Learning

In off-policy learning, especially for contextual bandits and RL, the variance-regularized pessimistic objective admits several concrete instantiations:

• The "Adaptive Sample Variance Penalization" (ASVP) algorithm (Girard et al., 17 Oct 2025):

$$\hat{f}_T^\lambda = \arg\min_{f\in\mathcal{F}} \left\{ \hat{R}_T(f) + \lambda \left(\frac{\hat{\sigma}_T(f)^{1-p/2}}{\sqrt{T}} + \frac{\hat{\sigma}_T(f)^{-p}}{T} \right) \right\},$$

where $\hat{\sigma}_T(f)$ is computed over the accumulated (possibly importance-weighted) sample losses.

• In the off-policy RL context, the losses are typically inverse-propensity-weighted returns or temporal difference errors, and the method may stabilize further by implicit exploration (e.g., adding minimum probability mass to the denominator).
• For the online, sequential setting, the procedure may run the update iteratively as new data arrives, in analogy to online empirical risk minimization.

4. Statistical Guarantees and Fast-Rate Regimes

The variance-regularized pessimistic objective yields nonasymptotic excess risk bounds: $$R_T(\hat{f}_T^\lambda) - R_T(f^*) = \widetilde{O}\left( \frac{\sigma_T(f^*)^{1-p/2}}{\sqrt{T}} + \frac{1}{T^{2/(2+p)}} \right),$$ where $f^*$ is a minimizer of the true risk, and $\sigma_T(f^*)$ its standard deviation (Girard et al., 17 Oct 2025).

Key properties:

• If the optimal $f^*$ (e.g., optimal policy) has low variance, the excess risk decays faster than $1/\sqrt{T}$, with $O(1/T)$ achievable if $\sigma_T(f^*) = 0$ (realizable, well-separated setting).
• Under margin/Hölder-type conditions relating the variance to the risk gap, the algorithm adjusts automatically, achieving parametric or accelerated rates.

5. Comparison to Classical and Alternative Variance Regularization Techniques

The variance-regularized pessimistic objective distinguishes itself from other regularization approaches:

Method	Adapts to Empirical Variance	Handles Dependent Data	Uniform High-Probability Bound	Fast Rate Possible	Principal Reference
Empirical Risk Minimization (ERM)	No	No	Only via standard VC/class entropy	$\sim 1/\sqrt{T}$ always	Classical theory
Empirical Bernstein (i.i.d.)	Yes (via sample variance)	No	Yes (in i.i.d.)	Sometimes	(Girard et al., 17 Oct 2025)
ASVP / Self-Normalized Martingale	Yes	Yes	Yes	Yes (low-variance regimes)	(Girard et al., 17 Oct 2025)
Explicit Lipschitz/Norm Regularization	No	No	No	No	Various

The self-normalized approach offers both data-adaptivity and the robustness of high-probability uniform control under dependence.

6. Practical Considerations and Simulations

Empirical results confirm the efficacy of variance-regularized pessimistic objectives:

• In i.i.d. and adaptive data regimes, ASVP-type algorithms decisively outperform classical ERM and standard IPS minimization when loss variance is heterogeneous or exploration is non-uniform.
• When integrated with implicit exploration or clipping, the variance penalty further enhances performance in off-policy RL benchmarks, notably by avoiding catastrophic overestimation when importance weights become large.
• Online versions (OSVP-PL) yield lower regret and more stable learning than sequential batch/baseline methods, especially in non-stationary or temporally correlated environments.

7. Implications and Significance

Variance-regularized pessimistic off-policy objectives provide a unified framework for compensating empirical loss instability and distributional shift in adaptive or offline learning scenarios. They deliver the following advantages:

• Uniform performance guarantees, valid for general martingale/adaptive data and arbitrary function classes with quantifiable complexity.
• Automatic interpolation between slow and fast rates of learning depending on the empirical variance of the optimal function.
• Applicability across contextual bandits, off-policy RL, and online learning with minor adaptation.

This methodology generalizes and subsumes previous approaches, setting a rigorous foundation for robust, conservative policy learning and evaluation in high-variance, data-adaptive settings (Girard et al., 17 Oct 2025).