Residual-Based Exploration & Exploitation

• Residual-based exploration and exploitation is a method that uses model residuals to quantify uncertainty and balance exploration with exploitation in sequential decision-making.
• It applies in bandit and reinforcement learning settings by comparing tuned and overfit models to derive data-driven uncertainty measures for action selection.
• Techniques such as ROME and ReBoot demonstrate efficient performance with reduced computational overhead compared to full posterior sampling methods.

Residual-based exploration and exploitation refers to a class of methodologies in bandit and reinforcement learning that utilize residuals—quantitative measures of fitting error or model disagreement—to modulate the balance between exploration and exploitation. This paradigm systematically exploits discrepancies either between differently regularized models or between empirical means and resampled/pseudo samples, thereby providing state-dependent, data-driven uncertainty measures without recourse to computationally intensive posterior sampling or stringent parametric assumptions. Notable frameworks include the Residual Overfit Method of Exploration (ROME) for contextual bandits (McInerney et al., 2021) and Residual Bootstrap Exploration (ReBoot) for multi-armed bandits (Wang et al., 2020).

1. Mathematical Foundations

Residual-based exploration mechanisms fundamentally quantify epistemic uncertainty via residuals derived from alternative model fits or perturbations. The principal mathematical objects are as follows:

• ROME leverages two point-estimate reward models per context–action pair $(x, a)$:

$$\hat{r}_{\mathrm{tuned}}(x, a),\quad \hat{r}_{\mathrm{overfit}}(x, a)$$

with $\hat{r}_{\mathrm{tuned}}$ minimizing bias via regularization and $\hat{r}_{\mathrm{overfit}}$ approaching unbiased but high-variance fitting. Their residual,

$$\Delta(x, a) = \hat{r}_{\mathrm{overfit}}(x, a) - \hat{r}_{\mathrm{tuned}}(x, a),$$

serves as a data-adaptive exploration bonus. The composite action score is

$$\hat{y}(x, a) = \hat{r}_{\mathrm{tuned}}(x, a) + \beta\,\Delta(x, a)$$

for a tunable $\beta$.

• ReBoot creates a bootstrapped index per arm $k$ at time $t$:

$$\widetilde{\mu}_{k, t} = \bar{Y}_{k, s} + \frac{1}{s+2}\sum_{i=1}^{s+2}w_i e_{k, i}$$

where $e_{k, i}$ are centered residuals of the $k$-th arm’s observed rewards, augmented by pseudo-residuals for variance inflation, and $w_i$ are mean-zero, unit-variance random weights. The residual-based variance ensures empirically matched or inflated exploration, particularly when sample count $s$ is small.

This methodology encodes a local measure of uncertainty or fit instability, which is leveraged to select actions with greater epistemic ambiguity.

2. Theoretical Justification

The principal theoretical motivations are furnished by both frequentist and Bayesian frameworks:

• ROME (Contextual Bandits):
  • Frequentist rationale:
  • If $f$ and $g$ denote the tuned and overfit models, each trained on independent splits, then

  $$\mathbb{E}[(g(x, a) - f(x, a))^2] = \mathsf{MSE}[f(x, a)] + \mathrm{Var}[g(x, a)]$$

  making the residual $\Delta(x, a)$ an upper-bound for the pointwise standard deviation of estimation error, usable as a proxy for UCB-style confidence intervals. - Bayesian and Information-Theoretic:

  $\Delta(x, a)$ approximates posterior predictive variance and, under Gaussian likelihoods, is proportional to a single-sample Monte Carlo estimate of the expected KL divergence (Bayesian information gain). Sampling

  $$\hat{y}(x, a) \sim \mathcal{N}(f(x, a), \Delta(x, a)^2)$$

  yields Thompson sampling–style updates without explicit posterior sampling.

• ReBoot (Multi-armed Bandits):
  • The bootstrapped index has mean $\bar{Y}_{k, s}$ and variance

  $$\mathrm{Var}(\widetilde{\mu}_{k, t} \mid H_{k, s}) = \frac{\mathrm{RSS}_{k, s} + 8\sigma_a^2}{(s+2)^2}$$

  with $\mathrm{RSS}_{k, s}$ the empirical residual sum of squares and $\sigma_a$ the exploration-aid unit. For large $s$, the variance reflects data-driven uncertainty; for small $s$, the fixed pseudo-residuals guarantee sufficient exploration. Under Gaussian reward/noise models, this choice secures instance-dependent logarithmic regret.

3. Algorithmic Realization

Both approaches admit straightforward algorithmic formulations while avoiding the high computational burden of classical posterior sampling or full nonparametric bootstrapping.

ROME-UCB (Contextual Bandits):

D = D₀ for t = 1,2,…: f = TrainModel(D, regularization=λ) g = TrainModel(D, regularization≈0) for x in batch of B contexts: for a in actions: r_tuned = f(x, a) r_overfit = g(x, a) bonus = r_overfit - r_tuned score[a] = r_tuned + β * bonus a_star = argmax_a score[a] observe reward r D.append((x, a_star, r))

- Model choices: Random forests, neural nets, or boosted trees. - Regularization for $f$: Weight decay, early stopping, dropout, bagging. - Overfit model $g$: Remove/relax regularization, increase depth/capacity, train longer. - Hyperparameter $\beta$: $\approx1$ as a baseline; can be decayed over time.

ReBoot (Multi-armed Bandits):

for each arm k: s_k = len(H_k) mean = np.mean(H_k) residuals = [y - mean for y in H_k] e_ps = [2*sigma_a, -2*sigma_a] w = np.random.normal(0, 1, s_k + 2) # or other mean-0, var-1 mu_tilde = mean + (1/(s_k+2)) * sum(w_i * e_i for w_i, e_i in zip(w, residuals + e_ps)) choose arm with highest mu_tilde, pull and observe new reward, append to H_k

- Variance inflation: $\sigma_a = r\sigma$ with $r > 1.5$. - Computational cost: Per round is $O(K)$ for $K$ arms.

4. Empirical Evaluation

Empirical performance results are reported for both frameworks using established baselines on various synthetic and real datasets.

Method	Setting	Key Result
ROME-TS	Bandit, medium action (65)	Lowest regret ($0.657\pm0.012$) among practical methods—Bach Chorales
ROME-TS	Large/sparse (3,600 items)	Improved exploration and lower regret ($0.941\pm0.006$) vs. LinUCB ($0.967\pm0.005$)—MovieLens-depleting
ReBoot	Gaussian, K=10	Achieves $\mathcal{O}(\log T)$ regret with robust adaptation to mean and variance shifts (all reward types)

ROME is particularly effective in regimes with large, sparse action sets and limited positive samples per bootstrap replication, where single-split variance proxies outperform multi-resample schemes. ReBoot demonstrates robustness on both bounded and unbounded reward distributions, with computational efficiency matching that of Thompson Sampling and outperforming more memory- and compute-intensive bootstrap schemes such as Giro or PHE.

5. Comparison with Alternative Exploration Frameworks

Residual-based schemes are situated within a broader landscape of exploration strategies:

• Thompson Sampling: Bayesian posterior sampling (e.g., Normal-Gaussian), but variance shrinks as $s^{-1}$ and performance depends on prior specification; can under-explore if variance is underestimated.
• Giro: Nonparametric bootstrap plus deterministic pseudo-rewards; effective for bounded, $[0,1]$ rewards but not generalizable to unbounded cases.
• PHE: Adds i.i.d. Bernoulli-distributed noise to reward histories; similarly bounded to $[0,1]$ applications.
• ROME and ReBoot: Directly leverage empirical or model-based residuals; adapt to both bounded/unbounded rewards. They provide strong theoretical and empirical performance across diverse settings with minimal computational overhead (see table below):

Method	Reward Setting	Adaptivity	Computational Cost
TS	Gaussian/Bernoulli	Requires prior	$O(K)$ per round
Giro	Bounded ($[0,1]$)	No	$O(T^2)$, high
PHE	Bounded ($[0,1]$)	No	$O(K)$, simple
ReBoot	Any (bounded/unbounded)	Yes	$O(K)$, efficient
ROME	Contextual	Yes	$2\times$ standard model fit

6. Practical Considerations and Limitations

The deployment and tuning of residual-based exploration methods entail the following considerations:

• Model Class/Capacity: Flexibility is crucial; classifiers or regressors incapable of overfitting (e.g., shallow linear models) can compromise $\Delta(x,a)$'s informativeness. Function approximators must admit both bias–variance trade-off control and high-capacity regimes.
• Regularization Control: The strength and nature of regularization (e.g., $\ell_2$ for neural nets, tree-depth for forests) must be adjustable and validated.
• Variance Inflation Hyperparameters: Selection of exploration weight $\beta$ (ROME) or inflation ratio $r$ (ReBoot) is essential. Empirical guidance sets $r\approx1.5$ and sweeps $\beta$ to balance discovery against regret.
• Update Frequency and Data Efficiency: ROME retrains both models every $O(100)$ interactions; ReBoot updates streaming statistics per round.
• Independence of Model Fits: Whenever possible, $f$ and $g$ should be trained on independent data splits or with independent online updates to maximize residual informativeness.
• Computational Overhead: ROME incurs twice the cost of a single model fit per batch. ReBoot’s per-round complexity is similar to Thompson Sampling.

Empirical data suggest these schemes excel when standard bootstrap-based approaches are infeasible or ill-suited, particularly in high-dimensional, sparse-reward, or unbounded-outcome settings.

7. Extensions and Open Directions

Residual-based exploration methodologies highlight the utility of fit instability and empirical error as proxies for epistemic uncertainty in sequential decision-making problems. Their agnosticism to reward distributional assumptions (ReBoot) and model architecture (ROME), coupled with empirical robustness, positions them as practical alternatives when MCMC or full-posterior methods are prohibitive. Potential avenues for future research include:

• Automatic tuning or adaptive scheduling of $\beta$ and $r$;
• Application to structured, combinatorial, or nonstationary environments;
• Analysis of failure modes in heavily non-i.i.d. or adversarial scenarios;
• Extensions to deep reinforcement learning and non-tabular Markov decision processes.

These directions remain subjects of active theoretical and empirical investigation.