Bootstrapped UCB for Robust Exploration

• Bootstrapped UCB is a non-parametric, data-driven approach that quantifies uncertainty using ensemble resampling and bootstrapped estimators.
• It adapts classical UCB methods to complex environments, including heavy-tailed noise and nonlinear deep RL settings, by leveraging diverse model estimates.
• Empirical results show improved sample efficiency and lower regret compared to traditional techniques, with theoretical guarantees matching near-optimal bounds.

Bootstrapped Upper Confidence Bound (Bootstrapped UCB) is a non-parametric, data-driven methodology for constructing exploration bonuses in sequential decision-making problems, including stochastic bandits, linear bandits, and reinforcement learning (RL) with deep neural networks. By leveraging resampling or ensemble-based approaches, Bootstrapped UCB computes confidence bounds or uncertainty estimates that are robust to model mis-specification and heavy-tailed noise, thus extending classical UCB techniques beyond sub-Gaussian or linear environments (Bai et al., 2021, Chen et al., 2017, Hao et al., 2019, Sudarsanam et al., 2016).

1. Non-Parametric Bootstrapped UCB: Core Formulation

In Bootstrapped UCB, uncertainty quantification is achieved via empirical resampling of data (multi-armed/linear bandits) or maintenance of diverse value function estimates (Q-ensembles in RL). A canonical strategy is to construct an ensemble of $K$ estimators (e.g., Q-networks $Q^1, \dots, Q^K$ in deep RL, regression coefficients $\hat\beta^{(b)}$ in linear bandits), each initialized or fitted independently.

For any prediction target (e.g., expected reward or Q-value), the empirical standard deviation across the ensemble serves as a measure of epistemic uncertainty:

$$\bar Q(s,a) = \frac{1}{K} \sum_{k=1}^K Q^k(s,a), \quad b(s,a) = \sqrt{ \frac{1}{K} \sum_{k=1}^K (Q^k(s,a) - \bar Q(s,a))^2 }.$$

The optimistic upper confidence bound augments the exploitation estimate with a scaled uncertainty bonus:

$Q^+(s,a) = Q(s,a) + \alpha b(s,a),$

where $\alpha > 0$ is a tunable coefficient (Bai et al., 2021, Chen et al., 2017).

In bandit settings, bootstrapped predicted rewards $Y_u^{(b)}$ are computed by resampling and refitting the model:

$$\mathrm{UCB}_t(u) = Q_{\delta}\bigl\{ Y_u^{(1)}, \dots, Y_u^{(B)} \bigr\},$$

where $Q_\delta$ denotes the empirical $\delta$-quantile among $Q^1, \dots, Q^K$0 bootstrap replicates (Sudarsanam et al., 2016).

2. Methodological Variants Across Problem Classes

Bootstrapped UCB encompasses diverse algorithmic instantiations:

• Multiplier Bootstrap for Multi-armed Bandits: For each arm, multiplier bootstrap statistics are constructed using resampled weights (e.g., Rademacher variables) applied to de-meaned reward sequences, then upper quantiles are used to form UCB indices. Second-order corrections are included to guarantee finite-sample validity (Hao et al., 2019).
• X-Random and X-Fixed Bootstrap for Linear Bandits: The X-Random method resamples entire data pairs $Q^1, \dots, Q^K$1, while X-Fixed resamples residuals after fitting, constructing bootstrap replicates of the regression parameter. UCBs are set by upper quantiles of the ensemble's predicted rewards for each arm. These estimators avoid explicit distributional assumptions, in contrast to analytic ellipsoidal confidence sets as in LinUCB or OFUL (Sudarsanam et al., 2016).
• Deep RL via Q-Ensembles: An ensemble of $Q^1, \dots, Q^K$2 value function estimators with shared trunk and separate heads is maintained. Bootstrapped Q-values are used to compute state-action bonuses. During exploration, actions are selected by maximizing the sum of ensemble mean and scaled standard deviation (Bai et al., 2021, Chen et al., 2017).

3. Theoretical Guarantees and Connection to Classical UCB

In linear Markov decision processes (MDPs), Bootstrapped UCB recovers the exploration bonus of LSVI-UCB, which is known to yield near-optimal regret bounds. The bootstrap variance converges to the Bayesian posterior variance of the value prediction:

$Q^1, \dots, Q^K$3

as the ensemble size $Q^1, \dots, Q^K$4 and initialization is uninformative. LSVI-UCB achieves regret of order $Q^1, \dots, Q^K$5 in linear MDPs, which is inherited by Bootstrapped UCB (Bai et al., 2021).

In multi-armed bandits under sub-Weibull noise, Bootstrapped UCB satisfies both problem-dependent and problem-independent regret bounds:

• Problem-dependent: $Q^1, \dots, Q^K$6.
• Problem-independent: $Q^1, \dots, Q^K$7 (Hao et al., 2019).

4. Practical Implementation in Deep RL and Bandits

Bootstrapped UCB methods are parameterized by the ensemble size ($Q^1, \dots, Q^K$8 is standard), the bootstrap repetition count ($Q^1, \dots, Q^K$9), and hyperparameters governing the exploration bonus scale ($\hat\beta^{(b)}$0, $\hat\beta^{(b)}$1) and learning schedule.

Table: Implementation Characteristics Across Domains

Setting	Ensemble Construction	UCB Bonus Formula
Multi-armed bandit	Multiplier bootstrap	$\hat\beta^{(b)}$2
Linear bandit	Bootstrapped regressors	$\hat\beta^{(b)}$3
Deep RL (Q-ensemble)	$\hat\beta^{(b)}$4 Q-network heads	$\hat\beta^{(b)}$5

In deep RL, key architectural details vary by domain. For instance, in the MNIST-maze the backbone comprises convolutional layers and a 512-unit FC before linear heads; for Atari benchmarks, a three-layer convolutional trunk is used, followed by a shared FC layer and ensemble heads (Bai et al., 2021, Chen et al., 2017). Training involves experience replay, periodic target network updates, and annealed $\hat\beta^{(b)}$6-greedy exploration to maintain diversity.

5. Comparative Empirical Performance

Bootstrapped UCB methods consistently achieve improved sample efficiency and lower regret relative to classical UCB, Thompson sampling, and other state-of-the-art baselines.

• Bandits: In simulation studies with heavy-tailed noise and misspecified models, bootstrapped UCB (especially X-Random) attains significantly lower cumulative regret—sometimes less than half that of OFUL, LinUCB, or Thompson sampling (Sudarsanam et al., 2016, Hao et al., 2019).
• Deep RL: On MNIST-maze (10×10) and Atari (49 games), OB2I achieves near-optimal path length ratios and superior mean/median human-normalized scores (e.g., 765% mean for OB2I versus 610% BEBU-UCB and 241% DQN on Atari with 1/10th the frames), demonstrating strong exploration and sample efficiency (Bai et al., 2021).
• Uncertainty Localization: Spikes in the bootstrapped uncertainty bonus $\hat\beta^{(b)}$7 align with bottleneck states or significant reward events, providing interpretable attribution of epistemic uncertainty to semantically meaningful regions of the state space (Bai et al., 2021).

6. Advantages, Limitations, and Considerations

Bootstrapped UCB offers several significant advantages:

• Distribution-free Construction: Confidence bounds are constructed empirically, requiring neither sub-Gaussian assumptions nor parametric noise models.
• Robustness to Heavy-tailed Noise: Regret bounds and empirical performance persist under sub-Weibull and heavy-tailed settings where vanilla UCB over-explores (Hao et al., 2019).
• Flexibility for Deep RL: Bootstrapped ensembles naturally extend upper confidence approaches to nonlinear value function classes—unlike classical UCB that is restricted to tabular/linear cases (Bai et al., 2021, Chen et al., 2017).
• Practical Effectiveness: Outperforms or matches state-of-the-art competitors across several benchmarks and settings.

Potential limitations include increased computational cost from ensemble maintenance or repeated resampling and, in some implementations, the need for careful calibration of the exploration scale ($\hat\beta^{(b)}$8, $\hat\beta^{(b)}$9). No explicit finite-sample regret bounds for all bootstrap variants (e.g., X-Random/X-Fixed) have been established (Sudarsanam et al., 2016).

7. Summary and Research Directions

Bootstrapped UCB unifies principled exploration in online learning via non-parametric, data-dependent uncertainty quantification. It recovers worst-case optimal regret bounds in canonical settings, provides robust exploration in nonlinear and deep architectures, and empirically yields strong sample efficiency and reward. Ongoing directions include extension to structured RL, formalization of finite-sample guarantees for all bootstrap approaches, and improved computational schemes for large-scale ensemble management (Bai et al., 2021, Hao et al., 2019, Sudarsanam et al., 2016, Chen et al., 2017).