Upper Confidence Bound Algorithm

• Upper Confidence Bound (UCB) is a decision-making strategy that balances exploration and exploitation by selecting actions based on empirical means and uncertainty estimates.
• It guarantees logarithmic regret under broad assumptions, ensuring each arm is sampled adequately to support robust statistical inference.
• Modern UCB variants extend its application to heavy-tailed, contextual, and nonstationary settings, enhancing algorithmic adaptability and fairness.

The Upper Confidence Bound (UCB) algorithm is a foundational principle in sequential decision-making under uncertainty, particularly within the stochastic multi-armed bandit (MAB) framework. The UCB family of algorithms selects actions according to an optimism-in-the-face-of-uncertainty criterion, quantifying exploration via the empirical uncertainty of each arm's mean reward. UCB lies at the heart of randomized and deterministic exploration algorithms, achieving logarithmic regret under broad assumptions and enabling adaptive data collection suitable for subsequent statistical inference (Khamaru et al., 8 Aug 2024).

1. Formal Definition and Principles

In the stochastic $K$-armed bandit setting, each arm $a \in \{1, \ldots, K\}$ generates i.i.d. rewards from an unknown distribution $P_a$ with mean $\mu_a$ and variance $\sigma_a^2$. At round $t$, the agent selects an arm $A_t$, observes reward $X_t \sim P_{A_t}$, and aims to maximize cumulative reward or, equivalently, minimize (pseudo-)regret: $$\mathrm{Regret}_T = T\mu^* - \mathbb{E}\left[\sum_{t=1}^T X_t\right], \qquad \mu^* = \max_{1\leq a \leq K} \mu_a.$$ The basic UCB1 rule [Auer et al., 2002] is given by: $$UCB(a, t) = \bar{X}_{a, n_{a,t}} + \sqrt{\frac{2 \ln t}{n_{a,t}}},$$ where $\bar{X}_{a, n_{a,t}}$ is the empirical mean and $n_{a,t}$ the number of pulls of arm $a$ up to $t$. At each round, the algorithm selects $A_t = \arg\max_a UCB(a, t)$. Bonus terms may be tuned via constants or replaced by $\ln T$ instead of $\ln t$ for analysis convenience (Khamaru et al., 8 Aug 2024).

2. Statistical Stability and Inference

A critical property underpinning UCB's inferential validity is "stability," defined analogously to the notion in Lai & Wei (1982). An arm $a$ is stable if there exists a deterministic sequence $n^*_a(T)$ such that $n_{a,T}/n^*_a(T) \to_p 1$ as $T \to \infty$. Stability is necessary to ensure that the adaptively collected empirical means are amenable to martingale central limit theorems, resulting in

$$\sqrt{n_{a,T}}(\bar{X}_{a,T} - \mu_a) \xrightarrow{d} \mathcal{N}(0, \sigma_a^2).$$

In classical fixed-$K$ bandits under sub-Gaussian assumptions and sufficient arm gaps (e.g., $\Delta_a = o(\sqrt{\frac{\ln T}{T}})$), UCB ensures that each arm is stable (Khamaru et al., 8 Aug 2024). The deterministic sequence $n^*_a(T)$ is characterized as the unique solution to an implicit fixed-point equation: $$\sum_{a=1}^K \left( \sqrt{T/n^*} + \sqrt{\frac{T\Delta_a^2}{2\ln T}} \right)^{-2} = 1.$$ This stability guarantee is crucial for enabling rigorous downstream inferential tasks such as confidence interval construction for arbitrary linear contrasts of arm means, even when samples are adaptively collected.

3. Theoretical Performance and Extensions

Under sub-Gaussian rewards and fixed $K$, UCB achieves the minimax optimal pseudo-regret of order $$O\left(\sum_{a:\Delta_a > 0} \Delta_a^{-1} \ln T\right)$$. In terms of distributional properties of arm pulls, nontrivial techniques involving uniform law-of-iterated logarithm (LIL) bounds ensure high-probability control of empirical means uniformly across arms and time. Sandwiching arguments then yield sharp bounds on $n_{a,T}$ and characterize optimal sampling rates via the aforementioned fixed-point equation.

When the number of arms grows with the horizon, i.e., $K = K(T)$, stability is preserved if $\ln K / \ln T \to 0$ and a nontrivial fraction of arms remain near-optimal. In this regime, uniform LIL-type bounds can be union-bounded over arms, and the same implicit equation for pull allocations applies (Khamaru et al., 8 Aug 2024).

Regime	Stability Condition	Sampling Law (Implicit)
Fixed $K$	Bounded gaps, sub-Gaussian	$$\sum_a (\sqrt{T/n^*} + \sqrt{T\Delta_a^2/(2\ln T)})^{-2} = 1$$
Growing $K$	$\ln K/\ln T \to 0$, near-opt. arms	Same as fixed $K$, solution $n^*$ exists if $|S|/K \geq \alpha > 0$

4. UCB in Non-Standard and Heavy-Tailed Bandits

Modern variants of UCB expand coverage to models with unbounded or heavy-tailed rewards, as well as complex parametric or nonparametric settings:

• Heavy-tailed Bandits (RMM-UCB): The Resampled Median-of-Means UCB (Tamás et al., 9 Jun 2024) delivers near-optimal regret in the presence of only a finite $(1+a)$-moment (for unknown $a \in (0,1]$). It leverages robust, symmetry-based tests to obtain nonasymptotic, parameter-free confidence bounds, requiring no explicit moment information or variance tuning. The regret can grow as $O(\log^2 n)$ if $a_i$ is small but matches optimal rates for mild tails.
• Distribution-Specific UCB: The Multiplicative UCB (MUCB) is tailored to exponential reward distributions, using multiplicative inflation factors motivated by Cramér-Chernoff concentration for unbounded support (Jouini et al., 2012). This yields regret that matches the Lai-Robbins lower bound for exponential rewards.
• UCB for Nonstationary Reward Models: Discounted-UCB and Sliding-Window-UCB handle nonstationary environments by weighting past data by recentness or restricting to a moving window. They achieve regret $O(\sqrt{TU_T} \log T)$ where $U_T$ is the number of breakpoints (Garivier et al., 2008).

5. Advanced UCB Variants: Bayesian Optimization and Structured Actions

• Gaussian Process UCB (GP-UCB): For continuous or structured domains, GP-UCB leverages Gaussian process posteriors, using

$$\mathrm{UCB}(x,t) = \mu_{t-1}(x) + \sqrt{\beta_t} \sigma_{t-1}(x),$$

where $\beta_t$ grows logarithmically. The expected cumulative regret is shown to be $O(\sqrt{T\gamma_T \log T})$ with $\gamma_T$ the maximal information gain (Takeno et al., 2 Sep 2024, Contal et al., 2016). The improved randomized GP-UCB (IRGP-UCB) replaces the deterministic exploration parameter with a randomized shift-exponential variable, attaining $O(\sqrt{T\gamma_T})$ regret and removing superfluous over-exploration due to conservative bounds (Takeno et al., 2 Sep 2024).

• Contextual and Function Approximation UCB: NeuralUCB and Deep UCB (Zhou et al., 2019, Rawson et al., 2021) extend the UCB principle to contextual and nonlinear reward functions by constructing random feature or neural network-based empirical means and uncertainties, maintaining regret guarantees of order $\tilde{O}(\sqrt{T})$ or $O(\log^2 T)$ under suitable regularity conditions.
• Structured Bandits and MDPs: Accelerated UCB indices for Markov decision processes (MDP-UCB) (Cowan et al., 2019) solve an optimistic value optimization inside a relative-entropy ball by a system of two equations (for KL-UCB), providing asymptotically optimal regret and practical feasibility for moderate state-action spaces.

6. Downstream Inference and Fairness

A major advancement is the formalization of inferential guarantees for UCB-collected data. Thanks to the stability of pull counts, empirical means under UCB satisfy CLT-type asymptotics. This directly enables standard statistical inference (confidence intervals, hypothesis tests) on $u^T \mu$ for contrasts $u$, with coverage guarantees analogous to the i.i.d. setting. The canonical confidence interval is: $$C_{u, \alpha} = \left[ u^T X_T \pm z_{1-\alpha/2} \sqrt{ \sum_a u_a^2 \hat{\sigma}_a^2 / n_{a,T} } \right],$$ where $\hat{\sigma}_a^2$ is the empirical variance for arm $a$ (Khamaru et al., 8 Aug 2024).

Additionally, UCB is "fair" among arms with similar means: if gaps $\Delta_a$ are small, sample allocation is nearly uniform, as formalized by the unique solution to the fixed-point equation, with arms pulled approximately equally often.

7. Practical Considerations and Algorithmic Enhancements

Modern UCB algorithms can be tuned and adapted for applications with specific distributional assumptions, structural properties, or operational requirements:

• Parameter-free Tuning: RMM-UCB and bootstrapped UCB eliminate the need for explicit tail parameter estimation, using robust statistical techniques such as median-of-means or multiplier bootstrap quantiles (Tamás et al., 9 Jun 2024, Hao et al., 2019).
• Computational Efficiency: Closed-form solutions for divergence-based UCBs (e.g., Hellinger-UCB) yield substantial savings in large-scale real-time deployments, such as recommender systems operating under latency constraints (Yang et al., 16 Apr 2024).
• Distributed and Decentralized Variants: In networked decision-making environments, decentralized UCB (and KL-UCB) algorithms disseminate local statistics over communication graphs, with regret constants inversely related to neighborhood size, improving over single-agent analogues (Zhu et al., 2021).
• Regret Refinements: Variance-aware UCB (UCB-V) adapts exploration to empirical arm variances, potentially offering improved or even refined regret rates under favorable variance structure, but can exhibit instability in arm-pulling proportions (Fan et al., 12 Dec 2024).
• Generalization: The unified UCB theory (Kikkawa et al., 1 Nov 2024) demonstrates that, for any oracle quantity with sharp confidence intervals, UCB achieves order-optimal failure counts, extending far beyond classical mean reward objectives.

---

In summary, the UCB algorithm is a robust, theoretically principled backbone for exploration in online sequential decision problems. It maintains a delicate balance between exploration and exploitation, guarantees high statistical stability, admits strong regret and inferential guarantees, and can be specialized efficiently across an extensive array of problem classes and practical constraints (Khamaru et al., 8 Aug 2024, Tamás et al., 9 Jun 2024, Kikkawa et al., 1 Nov 2024, Rawson et al., 2021, Takeno et al., 2 Sep 2024, Fan et al., 12 Dec 2024, Zhu et al., 2021).