---
title: On-Policyness Weighting for RL and Bandits
url: https://www.emergentmind.com/topics/on-policyness-based-weighting
type: topic
---

# On-Policyness Weighting for RL and Bandits

On-policyness-based weighting encompasses modern methodologies for stabilizing and improving off-policy evaluation and learning from adaptively collected or offline data in contextual bandits and reinforcement learning. These approaches select or learn weights for each data point according to how closely its context and action match the distribution induced by a target policy—rather than relying on direct propensity ratios alone—thereby optimizing the so-called “on-policyness” of the weighted sample. This weighting paradigm reduces estimator variance, improves support coverage, and enables minimax mean-square-error optimization, as formalized in recent work [2106.02029], [1705.07384], [2109.04640].

## 1. Problem Formulation and the Need for On-Policyness-Based Weighting

Off-policy evaluation (OPE) refers to estimating the value of a target policy $\pi$ given data generated by a possibly different policy (logging or behavior policy). In contextual bandits, one observes data $(x_t, w_t, y_t)$ with context $x_t$, action $w_t \sim \pi_0(\cdot|x_t)$, and reward $y_t$. The analogous RL setting concerns discounted visitation distributions over state-action pairs. Naive inverse-propensity weighting (IPW) or doubly robust (DR) estimators normalize each sample via ratios such as $\frac{\pi(w_t|x_t)}{\pi_0(w_t|x_t)}$, which can introduce severe variance and instability if the target and logging policies have poor overlap. High-variance weights arise particularly when the logging probabilities $e_t(x,w)$ are small relative to the target, yielding “exploding” importance ratios in finite samples [2106.02029], [1705.07384].

On-policyness-based weighting directly addresses this pathology by constructing weights that favor data points most consistent with the target policy, either through adaptive variance-stabilizing transformations [2106.02029], direct balance optimization [1705.07384], or state-action marginal balancing [2109.04640].

## 2. Methodologies for Weight Construction

### Adaptive Weighting in Contextual Bandits

Zhan et al. [2106.02029] develop an adaptive weighting scheme atop the DR estimator, using data-driven proxies for per-sample variance. For a target policy $\pi$, the weighting factor is given by $h_t(x) = \phi\left(\sum_w \frac{\pi^2(w|x)}{e_t(x,w)}\right)$, with $\phi$ chosen to be either inverse standard deviation ($1/\sqrt{v}$, "StableVar") or inverse variance ($1/v$, "MinVar"). The improved estimator

$$
\widehat V^{AWDR}_T(\pi) = \sum_{t=1}^T \frac{h_t(x_t)}{\sum_{s=1}^T h_s(x_t)} \Gamma_t(x_t, \pi)
$$

sharply downweights high-variance, low-overlap samples, directly reducing estimator MSE and yielding stable inference.

### Balanced Weight Optimization

Kallus [1705.07384] formulates weight selection as a convex optimization minimizing worst-case or posterior conditional mean squared error (CMSE), subject to sample balance between the target and historical distribution. For i.i.d. data $(X_i, T_i, Y_i)$ and policy $\pi$, the optimal weight vector solves

$$
W^*(\pi) = \arg\min_{W\geq 0, \sum W_i = n}\; \sup_{\|f\|\leq 1} B^2(W,\pi;f) + \frac{1}{n^2} W^\top \Lambda W
$$

where $B(W, \pi; f)$ measures discrepancies in function moments between weighted empirical data and target policy, and $\Lambda$ penalizes variance. These “balance-based weights” expand support beyond IPW and yield lower finite-sample variance.

### Projected State-Action Balancing in RL

Wang et al. [2109.04640] advance a projected balancing approach in RL, where the optimal weights $w_{i,t}$ approximately satisfy moment-matching equations motivated by the Bellman backward recursion for discounted visitation. The optimization enforces, for each chosen basis function $B_k$,

$$
\left| \frac{1}{nT} \sum_{i,t} w_{i,t} \hat{L}_k(S_{i,t},A_{i,t}) - \ell_k \right| \leq \delta_k
$$

where $\hat{L}_k$ encodes local Bellman error and $\ell_k$ is the expected moment under $\pi$. This yields a quadratic program whose solution smoothly approximates the ideal marginal ratio $\omega^\pi(s,a)$, further regularized to prevent extreme weights.

## 3. Theoretical Guarantees and Finite-Sample Behavior

On-policyness-based weighting frameworks possess robust theoretical guarantees. In contextual bandits, adaptive weighting yields central limit theorems (CLT) for the studentized estimator under regularity conditions (bounded rewards, sufficient exploration, regression consistency, stability of adaptive weights). One obtains asymptotically normal t-statistics and valid confidence intervals of the form

$$
\frac{\widehat V_T - V(\pi)}{\widehat{\sigma}_T} \xrightarrow{d} \mathcal{N}(0,1)
$$

enabling inferential validity even under adaptive data collection [2106.02029]. Balanced weighting possesses minimax consistency and uniform regret bounds, with error rates scaling as $\mathfrak{R}_n(\Pi) + n^{-1/2}$ in policy learning (where $\mathfrak{R}$ is Rademacher complexity) [1705.07384]. In RL, projected balancing yields semiparametric efficient estimates, with convergence rates depending on basis representation and the minimal eigenvalue of the balancing Gram matrix [2109.04640]. Coverage and absolute continuity conditions ($\sup_{s,a} \bar d^\pi(s,a)/\bar p^b_T(s,a)<\infty$) are necessary for well-posedness of the off-policy evaluation problem.

## 4. Comparative Analysis: On-Policyness vs. Classical Weighting

Classical IPW can discard large portions of data when target policy support does not align and can suffer explosive variance; balanced and adaptive weighting, by contrast, constructs weights with wider support and explicitly penalizes variance rather than resorting to ad hoc clipping [1705.07384]. In RL, marginal importance ratios can be less extreme than per-step product ratios; projected balancing shrinks outlier weights by moment-matching on rich function classes, preventing instability [2109.04640].

The following table contrasts salient features of three weighting paradigms:

| Weighting Approach                 | Support Coverage     | Variance Control         |
|------------------------------------|---------------------|-------------------------|
| Inverse-Propensity (IPW/DR)        | Sparse, often drops | None (ad hoc clipping)  |
| Adaptive/Balance-based             | Wide, all samples   | Explicit via objective  |
| Projected State-Action Balancing   | Wide, smooth        | Moment-matching penalty |

## 5. Algorithmic Implementation

All major schemes can be cast as tractable convex programs:

- **Adaptive weighting**: $O(K)$ per step to compute variance proxies, with a single pass over the data [2106.02029].
- **Balance-based weights**: Inner QP in $n$ variables, outer policy optimization via quasi-Newton iteration [1705.07384].
- **Projected balancing RL**: Small-scale QP in $nT$ variables, with basis expansion and regularized regression steps [2109.04640].

Pseudocode is provided for each method, with key steps involving proxy/constraint computation, QP or convex dual solvers, and output value/inference statistics.

## 6. Empirical Performance and Applications

Across diverse domains (synthetic contextual bandit experiments, UCI datasets, RL benchmarks such as CartPole and mHealth), on-policyness-based weighting consistently yields lower RMSE, tighter standard errors, and improved inferential coverage compared to naive DR or IPW. For example, contextual MinVar AWDR cuts RMSE by 30–50% and yields near-nominal coverage [2106.02029]; balance-based learners outperform IPW and POEM methods in regret and support coverage by 5–15% [1705.07384]; projected balancing yields the lowest root-MSE and most plausible biological ordering in mHealth policy ranking [2109.04640]. Contextual versions outperform aggregate approaches, and more aggressive variance penalization (MinVar) trades off slight bias for further error reduction.

## 7. Significance, Current Limitations, and Future Directions

The “on-policyness” paradigm leverages data-driven estimates of policy alignment to produce more statistically efficient, stable, and widely applicable policy evaluation and learning procedures. Such methods are central as datasets grow larger, policies more adaptive, and off-policy deployment more routine. Current challenges include basis selection for balancing, scaling optimization to extreme sample sizes, and optimal regularization for finite-sample trade-offs. Theoretical boundaries of estimator efficiency, coverage, and regret under worst-case overlap remain active areas of investigation. Approaches integrating balancing, adaptive variance control, and nonparametric estimation are likely to expand in importance as both contextual bandit and RL evaluation frameworks evolve.

Source: https://www.emergentmind.com/topics/on-policyness-based-weighting