Optimism-Based Stochastic Bandit Algorithms

• Optimism-based stochastic bandit algorithms are sequential decision-making methods that use high-probability confidence bounds to balance exploration and exploitation under uncertainty.
• They encompass key instantiations such as UCB, LinUCB, and randomized variants, each providing rigorous regret guarantees through tailored exploration bonuses.
• Recent enhancements involve adaptive and tighter confidence sets to improve performance, though challenges like statistical instability and instance-dependent limitations persist.

Optimism-based stochastic bandit algorithms are a foundational class of sequential decision-making methods in which action selection is driven by the principle of "optimism in the face of uncertainty" (OFU). In this paradigm, algorithms construct upper (or optimistic) estimates of unknown rewards using statistically valid confidence sets or stochastic perturbations, and select actions that maximize these optimistic indices. This systematic bias towards exploration enables favorable regret guarantees in both finite-armed and structured (e.g., linear or contextual) bandit settings. The optimism principle underpins most state-of-the-art stochastic bandit and linear bandit algorithms, informs much of the theoretical literature, and continues to give rise to novel algorithmic and analytical frameworks.

1. Fundamental Principles and Algorithmic Template

Optimism-based algorithms follow a unified structure: at each round $t$, construct, for each arm $i$ (or action $a$), a high-probability upper confidence bound $I_i(t)$ for the unknown mean reward, and select an arm maximizing this optimism-driven index. The prototypical formula is: $$I_i(t) = \widehat\mu_i(t) + \text{exploration-bonus}_i(t),$$ where $\widehat\mu_i(t)$ is an estimator (typically empirical mean or ridge estimate) and the exploration bonus quantifies the statistical uncertainty, decreasing with the number of pulls. Representative choices include the classic UCB1 algorithm for $K$-armed bandits ($\sqrt{\frac{2\ln t}{N_i(t)}}$), LinUCB for linear bandits, and confidence radii derived from concentration inequalities in structured settings (Krishnamurthy, 20 Dec 2025).

A core aspect is the provision of a uniform, high-probability guarantee: $$\mathbb{P}\left(\forall\,t\leq T: |\widehat\mu_i(t) - \mu_i| \leq r_i(t) \right) \geq 1-\delta$$ for carefully tuned radii $r_i(t)$. Two deterministic lemmas—radius collapse and optimism-forced deviations—drive the regret analysis, bounding the number of times a suboptimal arm can be pulled before the confidence radius collapses and any further pull necessitates a confidence interval violation (Krishnamurthy, 20 Dec 2025).

2. Core Instantiations: UCB, LinUCB, and Generalizations

Canonical optimism-based algorithms span a diverse range of settings:

• UCB (Upper Confidence Bound): For $K$-armed bandits, selects $$\arg\max_i \widehat\mu_i(t) + \sqrt{2\ln t / N_i(t)}$$. Guarantees $$\mathbb{E}[R_T] = O\left(\sum_{i\ne i^*} \frac{\log T}{\Delta_i}\right)$$, matching known lower bounds up to constants (Krishnamurthy, 20 Dec 2025, Lattimore, 2015).
• LinUCB (Linear Bandits): Maintains a confidence ellipsoid for $\theta^*\in\mathbb{R}^d$ using observed feature-reward pairs. At round $t$, selects

$$a_t = \arg\max_{a\in\mathcal{A}_t} \langle \hat\theta_t, a\rangle + \beta_t\|a\|_{V_t^{-1}},$$

where $V_t$ is the empirical Gram matrix and $\beta_t$ is a confidence width. This yields regret $\widetilde{O}(d\sqrt{T})$ (Hamidi et al., 2020).

• Minimax- and Problem-Dependent Variants: Optimally Confident UCB (OCUCB) tunes the index bonus via parameters $\alpha,\psi$ for best tradeoff, ensuring both $\sqrt{nK}$ minimax and $\sum_i \Delta_i^{-1}\log n$ problem-dependent regret (Lattimore, 2015). KL-UCB and MOSS variants use information-theoretic or problem-aware radii, adjusting the decay of the bonus for worst-case regret (Praharaj et al., 24 Nov 2025).
• Martingale-Mixture and Adaptive Confidence Sets: Recent approaches leverage tighter confidence sequences, e.g., via adaptive martingale mixtures, yielding strictly smaller confidence regions and matching or improving upon OFUL in both theory and practice (Flynn et al., 2023).

3. Extensions: Randomized Optimism, Bootstrap, and Variational Formulations

Optimism can be induced through random perturbations (randomized optimism) or Bayesian analysis:

• Bootstrap-based Exploration: The LinReBoot algorithm perturbs arm reward estimates by resampling residuals from past observations, effectively injecting stochastic optimism into each arm index. The residual bootstrap bonus, combined with the usual sample concentration, constitutes a “collaborated optimism” principle and achieves $\widetilde{O}(d\sqrt{n})$ regret while bypassing explicit confidence calculations (Wu et al., 2022).
• Follow-the-Perturbed-Leader (FTPL) with Ambiguity: Rather than fixing the perturbation law, OFA-FTPL (Optimism in the Face of Ambiguity) optimizes over an ambiguity set of perturbation distributions, selecting the law that yields the most optimistic potential. The resulting arm-sampling probabilities are computed via regret-minimizing convex programs or efficient bisection, achieving optimal $\sum_k \log T/\Delta_k$ stochastic regret while unifying FTRL, FTPL, and UCB in a single framework (Li et al., 30 Sep 2024).
• Variational Bayesian Optimistic Sampling (VBOS): In the Bayesian setting, policies in the “optimistic set” maximize expected value under a convex risk mapping derived from posterior cumulant-generating functions. VBOS, which generalizes Thompson Sampling, ensures Bayesian regret $\tilde{O}(\sqrt{AT})$, and provably outperforms TS in certain zero-sum games and constrained settings (O'Donoghue et al., 2021).

4. Regret Guarantees and Analytical Frameworks

The regret analysis of optimism-based algorithms typically follows a unifying template: derive a high-probability concentration bound, ensure that the confidence radii shrink rapidly enough, and show via radius collapse and optimism-forced deviation arguments that suboptimal arms are pulled only $O(\log T/\Delta_i^2)$ times (Krishnamurthy, 20 Dec 2025, Hamidi et al., 2020).

Worst-case minimax optimal algorithms (e.g., MOSS and OCUCB) achieve $O(\sqrt{KT})$ regret; problem-dependent UCB variants reach $O(\sum_i \log T/\Delta_i)$. In structured problems—such as stochastic linear bandits—the regret rates scale as $\widetilde{O}(d\sqrt{T})$ (Hamidi et al., 2020, Flynn et al., 2023).

Under additional margin conditions (e.g., Goldenshluger–Zeevi), polylogarithmic regret bounds are achievable, with OFUL and Thompson Sampling attaining $\mathrm{poly}(\log T)$ regret independently of the instance gaps (Hamidi et al., 2020).

5. Limitations, Stability, and Open Problems

Despite their strong minimax and instance-dependent guarantees, optimism-based bandit algorithms display several fundamental limitations:

• Statistical Instability: Minimax-optimal UCB variants (e.g., MOSS, ADA-UCB, KL-MOSS) typically violate the Lai–Wei stability condition, necessary for sample means to exhibit central-limit-theorem behavior. In these algorithms, the exploration bonuses decay too quickly ($O(1/\sqrt{n})$), leading to empirical phenomena such as “lock-in” (one arm monopolizes pulls) and dramatic under-coverage of conventional confidence intervals. UCB1, with slightly slower-decaying bonuses, is stable but not minimax-optimal (Praharaj et al., 24 Nov 2025).

| Algorithm | Regret | Stability | |------------------------|---------------------|------------------| | UCB1 | $O(\sqrt{KT\ln T})$ | Stable | | MOSS, ADA-UCB, etc. | $O(\sqrt{KT})$ | Unstable |

• Instance-Dependent Optimality: In linear bandits with finite arms, no optimism- or Thompson-sampling–based algorithm achieves the information-theoretic lower bound for instance-dependent regret. Optimism eliminates arms as soon as their upper bounds fall below the empirical leader, failing to adequately explore “informative” (but low-mean) directions; as a result, they may accrue regret arbitrarily worse than the optimal convex allocation (Lattimore et al., 2016).

6. Extensions to Structured and Constrained Settings

Optimism-based methods generalize effectively to complex settings:

• Safe Linear Bandits and Directional Optimism: In safety-critical linear bandits where actions must satisfy unknown constraints, “directional optimism” separates the task of exploratory direction selection from constraint-satisfying scaling. Concrete algorithms such as ROFUL guarantee both strong regret bounds (e.g., $O(d\sqrt{T})$) and high-probability constraint satisfaction, outperforming global pessimism-based schemes especially when the constraint set has favorable geometry (Hutchinson et al., 2023).
• Pessimistic–Optimistic Algorithms for Constrained Bandits: Primal–dual approaches combine optimistic reward selection (e.g., LinUCB indices) with pessimistically-updated dual multipliers to maintain adherence to general nonlinear or stochastic constraints, yielding both sublinear regret and (eventual) zero cumulative constraint violation (Liu et al., 2021).
• Randomized Optimism in Multi-Agent and Game-Theoretic Bandits: In bandit feedback zero-sum games, algorithms such as COEBL leverage random evolutionary mutation operators to induce optimistic payoffs. This randomised optimism achieves sublinear regret matching deterministic OFU, and empirically outperforms classic UCB and EXP3 in large and adversarial games (Lin, 19 May 2025).

7. Unifying Theoretical Framework and Contemporary Developments

A minimal two-lemma framework now subsumes the analysis of UCB, UCB-V, LinUCB, GP-UCB, and many modern variants: prove a uniform-in-time concentration bound for the estimators, establish a deterministic “radius collapse” threshold, and use optimism-forced deviation to cap pulls to suboptimal arms at $O(\log T/\Delta_i^2)$. This structure extends to randomized optimism (e.g., FTPL, bootstrap), heteroskedastic settings, and ML-assisted estimators, providing a universal route to logarithmic regret (Krishnamurthy, 20 Dec 2025).

Recent advances—martingale mixture confidence sets, ambiguity-optimizing perturbation laws, and variational Bayesian optimism—offer strictly tighter confidence sets, unified regret proofs, or computationally scalable algorithms across stochastic, adversarial, and structured bandit regimes (Li et al., 30 Sep 2024, Flynn et al., 2023, O'Donoghue et al., 2021).

---

Optimism-based stochastic bandit algorithms continue to serve as the backbone of contemporary bandit theory, offering a systematic and extensible template for the design of efficient exploration strategies. The unifying analysis and emerging variants, together with the understanding of their statistical and computational limitations, represent a mature but still rapidly developing area of online learning research.