Bayesian Upper Confidence Bound (BUCB)

• BUCB is a family of Bayesian decision rules that employs posterior optimism by combining the mean and uncertainty estimates to guide action selection.
• It is applied in various settings such as Gaussian process optimization, multi-armed bandits, and adaptive sampling, demonstrating improved regret and efficient parallelization.
• Randomized variants of BUCB correct for over-exploration by replacing fixed confidence schedules with stochastic exploration weights, achieving near-optimal performance.

Searching arXiv for recent and foundational papers on Bayesian Upper Confidence Bound (BUCB). Bayesian Upper Confidence Bound (BUCB) denotes a family of Bayesian decision rules that select actions by maximizing or minimizing an upper credible bound derived from a posterior distribution. Across the literature, BUCB appears in several distinct but related settings: Gaussian-process Bayesian optimization, stochastic multi-armed bandits, fixed-budget best-arm identification, adaptive sampling for distribution estimation, and approximate-inference bandits. The common construction is an index of the form posterior mean plus a posterior uncertainty term, with the uncertainty scaled either by a deterministic confidence schedule, a posterior quantile, or a randomized exploration weight. In Gaussian-process optimization, this rule is typically written as $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)$$ and underlies GP-UCB and its variants [(Contal et al., 2013); (Berk et al., 2020); (Takeno et al., 2024); (Iwazaki, 2 Jun 2025)]. In bandit models with conjugate posteriors, BUCB is often implemented directly through posterior quantiles, such as the $(1-1/t)$-quantile of a Beta posterior (Huang et al., 2022) or the $\gamma$-quantile of a Gaussian posterior (Russo, 2019).

1. Conceptual definition and core index

The defining feature of BUCB is optimism under posterior uncertainty. At each decision time, one constructs an optimistic estimate of the latent utility or reward of each candidate action and then acts greedily with respect to that estimate. In the Gaussian-process setting, after observing $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$ with $y_i=f(x_i)+\epsilon_i$, the posterior mean and variance are

$$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$

$$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$

and the standard BUCB acquisition is

$$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$

The point selected is

$$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$

This form is explicit in GP-UCB and its randomized extensions (Takeno et al., 2024, Berk et al., 2020, Takeno et al., 2023).

The same principle appears in finite-armed Bayesian bandits. In Bernoulli bandits, classical Bayesian UCB uses the posterior quantile

$qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$

where $(1-1/t)$0 is the exact posterior of arm $(1-1/t)$1 (Huang et al., 2022). In Gaussian bandits with discount factor $(1-1/t)$2, the relevant index is the upper $(1-1/t)$3-quantile of a Gaussian posterior,

$(1-1/t)$4

which approximates the Gittins index as $(1-1/t)$5 (Russo, 2019). In fixed-budget Bayesian best-arm identification, the index takes the explicit form

$(1-1/t)$6

with Gaussian posterior mean $(1-1/t)$7 and variance $(1-1/t)$8 (Zhu et al., 2024).

A recurring misconception is that BUCB refers to a single algorithm. The literature instead uses the term for a broader design pattern: posterior optimism with a confidence width determined by theory, posterior quantiles, or randomization. This suggests that “BUCB” is best understood as a methodological class rather than a unique procedure.

2. Gaussian-process BUCB in Bayesian optimization

In Bayesian optimization, BUCB is most closely associated with GP-UCB. The unknown objective $(1-1/t)$9 is modeled as a Gaussian process $\gamma$0, observations are noisy, and the learner sequentially queries points in order to control cumulative regret. Classical regret analysis chooses $\gamma$1 large enough so that with probability at least $\gamma$2, the confidence interval

$\gamma$3

holds uniformly over $\gamma$4 and $\gamma$5 (Takeno et al., 2024). For finite $\gamma$6, a standard schedule is

$\gamma$7

(Takeno et al., 2024). This leads to the familiar regret rate

$\gamma$8

where $\gamma$9 is the maximum information gain after $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$0 points (Takeno et al., 2024).

The conservative nature of $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$1 is a major practical issue. Multiple papers explicitly note that the theoretical confidence parameter grows too large and induces over-exploration (Takeno et al., 2024, Takeno et al., 2023, Berk et al., 2020). Randomized GP-UCB addresses this by replacing deterministic $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$2 with a random exploration weight. In RGP-UCB, one draws $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$3 and uses

$D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$4

with $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$5 chosen so the Bayesian regret remains controlled (Berk et al., 2020). Takeno et al. generalized this perspective and proposed IRGP-UCB, in which

$D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$6

and

$D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$7

For finite $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$8, setting $D_{t-1}=\{(x_i,y_i)\}_{i=1}^{t-1}$9 and $y_i=f(x_i)+\epsilon_i$0 yields Bayesian cumulative regret

$y_i=f(x_i)+\epsilon_i$1

with $y_i=f(x_i)+\epsilon_i$2 and $y_i=f(x_i)+\epsilon_i$3, giving $y_i=f(x_i)+\epsilon_i$4 and thereby removing the extra $y_i=f(x_i)+\epsilon_i$5 factor (Takeno et al., 2024, Takeno et al., 2023).

More recent GP-UCB analysis sharpens high-probability regret even without randomization. Under a Matérn kernel satisfying the stated smoothness regime, GP-UCB achieves $y_i=f(x_i)+\epsilon_i$6 cumulative regret with high probability; under the squared exponential kernel, the bound becomes $y_i=f(x_i)+\epsilon_i$7 (Iwazaki, 2 Jun 2025). The argument uses concentration of the realized query sequence near the maximizer and replaces global maximum information gain by local information gain on shrinking balls around $y_i=f(x_i)+\epsilon_i$8 (Iwazaki, 2 Jun 2025). This indicates that BUCB performance depends not only on generic information-gain bounds but also on the trajectory induced by optimism.

3. Parallel and batch BUCB

A major extension of GP-UCB is the batch setting, where multiple evaluations are launched in parallel. GP-UCB-PE combines one UCB point with $y_i=f(x_i)+\epsilon_i$9 pure-exploration points within each batch (Contal et al., 2013). At iteration $$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$0, the posterior is

$$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$1

and the optimistic and pessimistic bounds are

$$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$2

The UCB point is

$$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$3

A lower-bound maximizer $$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$4 is computed, and a relevant region is defined as

$$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$5

where $$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$6 (Contal et al., 2013). The remaining batch points are then chosen greedily by maximizing posterior variance within $$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$7.

The regret guarantees explicitly quantify the value of parallelism. Let $$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$8 be the maximum information gain in $$\mu_{t-1}(x)=k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}y_{t-1},$$9 Gaussian-process queries. Then, with probability at least $$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$0, the batch simple regret

$$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$1

satisfies

$$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$2

with $$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$3 and $$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$4 (Contal et al., 2013). The corresponding full regret satisfies

$$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$5

where $$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$6 (Contal et al., 2013). In big-$$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$7 form, the simple-regret bound is

$$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$8

which yields a factor $$\sigma_{t-1}^2(x)=k(x,x)-k_{t-1}(x)^\top (K+\sigma^2 I)^{-1}k_{t-1}(x),$$9 improvement over sequential GP-UCB for fixed iteration count (Contal et al., 2013).

The constants in these bounds are dimension-free in the sense that $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$0 and $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$1 depend only on $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$2 and $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$3, not on the input dimension $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$4; dimension enters only through $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$5 and $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$6 (Contal et al., 2013). The paper also states that GP-UCB-PE, unlike GP-BUCB, requires no separate initialization phase and avoids exponential-in-$$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$7 constants (Contal et al., 2013).

4. BUCB in multi-armed bandits and best-arm identification

In finite-armed Bayesian bandits, BUCB often appears as a quantile rule rather than a mean-plus-standard-deviation heuristic. For Bernoulli rewards with exact Beta posteriors, the classical Bayesian UCB index is the $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$8-quantile of the posterior distribution of each arm mean (Huang et al., 2022). This is the exact Bayesian analogue of optimism and connects closely to KL-style concentration through Beta tail behavior (Huang et al., 2022).

A distinct line of work studies approximate inference. The Enhanced Bayesian Upper Confidence Bound (EBUCB) framework assumes only approximate posteriors $$a_t(x)=\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x).$$9 rather than exact posteriors $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$0 and measures approximation quality through two $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$1-divergences:

$$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$2

with $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$3 and $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$4 (Huang et al., 2022). The algorithm uses a modified quantile level

$$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$5

where $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$6, and selects the arm with largest $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$7 (Huang et al., 2022). Under the bounded-error assumption and mild support conditions, EBUCB achieves the optimal regret order

$$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$8

even when the inference error bound is constant (Huang et al., 2022). The same work shows that one bounded $$x_t=\arg\max_{x\in\mathcal X}\bigl[\mu_{t-1}(x)+\sqrt{\beta_t}\,\sigma_{t-1}(x)\bigr].$$9-divergence alone is insufficient to guarantee sub-linear regret (Huang et al., 2022). This is an important corrective to the view that approximate Bayesian inference merely perturbs BUCB mildly; the theory instead distinguishes sharply between one-sided and two-sided control.

In fixed-budget Bayesian best-arm identification, BUCB is used to allocate pulls rather than minimize cumulative regret. The algorithm in (Zhu et al., 2024) assumes a Gaussian random-effects prior

$qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$0

and constructs posterior means $qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$1 and posterior variances $qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$2 using pooled information across arms. The index

$qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$3

is then used to choose

$qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$4

After budget $qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$5, the algorithm returns $qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$6 (Zhu et al., 2024). The failure probability $qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$7 satisfies a problem-independent upper bound of order

$qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$8

up to logarithmic factors, and the simple Bayes regret obeys the same order (Zhu et al., 2024). The paper contrasts this with UCBE, whose tuning depends on the instance-complexity $qu_i(t)=Qu\bigl(1-1/t,\Pi_{t-1,i}\bigr),$9 and whose guarantees are instance-dependent (Zhu et al., 2024). Here, BUCB’s Bayesian prior learning tightens confidence intervals by estimating shared structure across arms.

5. BUCB beyond optimization: adaptive sampling and local search

BUCB has also been adapted to problems where the objective is not direct reward maximization. In adaptive sampling for estimating probability mass functions, each arm corresponds to an unknown pmf

$(1-1/t)$00

with independent Dirichlet priors (Kartik et al., 2020). The key quantity for arm $(1-1/t)$01 is

$(1-1/t)$02

since an oracle allocating $(1-1/t)$03 samples attains mean-squared error approximately $(1-1/t)$04 (Kartik et al., 2020). BUCB computes coordinatewise credible intervals, defines the feasible set

$(1-1/t)$05

and then sets

$(1-1/t)$06

The index is

$(1-1/t)$07

and the algorithm samples the arm maximizing $(1-1/t)$08 (Kartik et al., 2020). The resulting worst-case MSE regret satisfies

$(1-1/t)$09

matching the known classical frequentist UCB rate up to logarithmic factors (Kartik et al., 2020). In application to SARS-CoV-2 seroprevalence, the method allocates more tests to subpopulations with larger uncertainty and higher variance-type parameter $(1-1/t)$10 (Kartik et al., 2020).

A separate development appears in local Bayesian optimization. In the minimization setting, the UCB acquisition is written

$(1-1/t)$11

or equivalently one defines

$(1-1/t)$12

and chooses

$(1-1/t)$13

(Fan et al., 2024). MinUCB replaces a gradient-descent step by minimizing the UCB objective of a Gaussian-process surrogate, while LA-MinUCB adds a one-step look-ahead criterion that minimizes the expected post-update UCB minimum (Fan et al., 2024). Under the stated assumptions and with $(1-1/t)$14, Theorem 4.1 gives

$(1-1/t)$15

with probability at least $(1-1/t)$16 (Fan et al., 2024). This use of BUCB is structurally different from global BO: the upper bound guides local improvement steps rather than direct global exploration.

6. Theoretical themes, variants, and interpretation

Several theoretical motifs recur across BUCB variants. The first is the exploration–exploitation coefficient. In GP-UCB, $(1-1/t)$17 is chosen to guarantee simultaneous confidence bounds, but this often causes over-exploration in practice (Takeno et al., 2024, Takeno et al., 2023). Randomized variants respond by replacing a deterministic schedule with a distribution over confidence widths. RGP-UCB uses a Gamma law for $(1-1/t)$18 (Berk et al., 2020), whereas IRGP-UCB uses a shifted exponential law with constant mean in finite domains (Takeno et al., 2024, Takeno et al., 2023). This suggests that the critical object is not a single prescribed confidence sequence but a distribution whose moments and tails support the regret analysis.

The second theme is information gain. In GP bandits, regret bounds are typically controlled by $(1-1/t)$19, the maximum information gain after $(1-1/t)$20 observations (Takeno et al., 2024, Iwazaki, 2 Jun 2025). In batch GP-UCB-PE, the same mechanism yields regret bounds featuring $(1-1/t)$21 and a $(1-1/t)$22 simple-regret improvement (Contal et al., 2013). In the refined 2025 analysis, the realized trajectory’s concentration around the maximizer allows local, rather than global, information-gain bounds to govern regret (Iwazaki, 2 Jun 2025). A plausible implication is that BUCB analysis is increasingly shifting from worst-case domain geometry toward path-dependent geometry.

The third theme is the role of posterior exactness. Exact posteriors lead naturally to quantile indices or standard posterior standard deviations. Once inference is approximate, the optimism mechanism itself may become distorted. EBUCB shows that two bounded $(1-1/t)$23-divergences suffice for logarithmic regret, whereas one bounded divergence does not (Huang et al., 2022). This directly addresses a common misconception that posterior approximation error can be summarized by a single scalar discrepancy without affecting regret order.

A fourth theme is the relation between BUCB and alternative indices. In Gaussian bandits with discounting, the Gittins index satisfies

$(1-1/t)$24

with $(1-1/t)$25 as $(1-1/t)$26 (Russo, 2019). Thus, for sufficiently patient agents, the Gittins index is asymptotically equivalent to a Bayesian upper confidence bound (Russo, 2019). This equivalence places BUCB within a broader index-policy tradition rather than isolating it as uniquely “Bayesian optimization” machinery.

7. Empirical behavior and comparative position

The empirical literature consistently frames BUCB against alternatives such as Expected Improvement, Thompson Sampling, MES, JES, PIMS, UCBE, Successive Rejects, Sequential Halving, and gradient-based local BO methods. In GP optimization, randomized BUCB variants are reported to outperform deterministic GP-UCB on synthetic functions and real-world emulators or materials tasks, with the stated interpretation that randomization bridges the gap between conservative theory and practical calibration (Takeno et al., 2024, Berk et al., 2020, Takeno et al., 2023). In particular, IRGP-UCB is reported to outperform GP-UCB, RGP-UCB, Thompson sampling, EI, MES, JES, and PIMS on synthetic GP-drawn functions with $(1-1/t)$27 and $(1-1/t)$28, and to converge fastest on Holder table, Cross-in-tray, and Ackley benchmarks (Takeno et al., 2024). Similar empirical advantages are reported on perovskite, P3HT/CNT, and AgNP materials optimization (Takeno et al., 2024).

In batch optimization, GP-UCB-PE is reported to be efficient on real and synthetic problems compared to state-of-the-art competitors (Contal et al., 2013). In fixed-budget best-arm identification, BUCB reduces error by 20–30% versus SR/SH and by up to 10% versus infeasible UCBE on random-means instances, and on fixed-means setups outperforms SR/SH by 10–60% in error rate while matching or beating infeasible UCBE in four of six cases (Zhu et al., 2024). In adaptive sampling for pmf estimation, BUCB is described as achieving nearly the oracle allocation $(1-1/t)$29 and reducing error in small or high-variance subgroups in SARS-CoV-2 seroprevalence estimation (Kartik et al., 2020). In local BO, MinUCB and especially LA-MinUCB are reported to improve over GIBO/MPD in synthetic and MuJoCo experiments (Fan et al., 2024).

These results should not be conflated across problem classes. “Best performance” in fixed-budget best-arm identification, “faster convergence” in local minimization, and improved simple regret in Bayesian optimization are not interchangeable claims. The literature instead indicates that BUCB is a portable optimism principle whose concrete benefits depend strongly on the posterior model, the meaning of uncertainty, and the target performance criterion—cumulative regret, simple regret, failure probability, or worst-case estimation error.