P-UCB Sampling: Probabilistic UCB Methods

• P-UCB Sampling is a class of algorithms that enhances the classical UCB framework by integrating probabilistic and data-driven exploration bonuses for improved risk control and adaptivity.
• It encompasses variants like Randomized GP-UCB for Bayesian optimization, UCB^τ methods with parametric decay, and Bayesian UCB strategies for uniform estimation, each tailored to specific performance needs.
• Widely applied in multi-armed bandits, molecular simulations, and heavy-tailed scenarios, P-UCB Sampling demonstrates robust regret bounds and efficient sampling in complex decision-making tasks.

P-UCB Sampling

P-UCB ("Probabilistic" or "Posterior" UCB) sampling refers generically to a class of algorithms that modify the canonical Upper Confidence Bound (UCB) principle to enhance adaptivity, risk control, or domain-specific performance by incorporating probabilistic, randomized, or data- and prior-driven mechanisms into the exploration bonus, acquisition function, or sampling index. Originally rooted in the bandit literature, "P-UCB" now encompasses several distinct (and sometimes unrelated) strands, including randomized GP-UCB for Bayesian optimization, UCB algorithms with parameterized decay for exploration, parallel or adaptive UCB frameworks for molecular simulation, risk-calibrated and Bayesian UCBs for uniform estimation of means or distributions, and pure-exploration UCB strategies under minimal distributional assumptions.

1. General UCB Framework and Common Extensions

The classical UCB framework maintains, for each arm $i$ at time $t$, an empirical mean $\hat\mu_i(t)$ and augments it with an exploration bonus $b_i(t)$:

$\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$

where $b_i(t)$ is typically chosen as a (possibly data-driven) upper confidence bound for the estimation error of $\mu_i$. The arm selected is the one maximizing $\text{UCB}_i(t)$.

Standard choices include $b_i(t) = \sqrt{2\log t/n_i(t)}$ as in UCB1, where $n_i(t)$ is the number of times arm $t$0 has been pulled. However, limitations of this fixed-bonus approach (such as over-exploration, suboptimal finite-sample regret, or lack of robustness to heavy tails and structure) motivate P-UCB variants.

2. Randomized UCB in Bayesian Optimization

RGP-UCB ("Randomized GP-UCB"), as developed in "Randomised Gaussian Process Upper Confidence Bound for Bayesian Optimisation" (Berk et al., 2020), replaces the deterministic acquisition function

$t$1

with a randomized one:

$t$2

where $t$3 is determined via a regret-optimal formula and $t$4 tunes the mean. This induces stochasticity in the exploration-exploitation tradeoff at each iteration. Empirical results show that RGP-UCB outperforms fixed $t$5 GP-UCB, and can be tuned (via $t$6) to favor exploration or exploitation as desired. The algorithm admits a provable sub-linear Bayesian regret bound; crucially, by randomizing $t$7, it avoids the "large fixed bonus" conservatism of standard GP-UCB (Berk et al., 2020).

Attribute	RGP-UCB (P-UCB) Feature	Classical GP-UCB Feature
$t$8	Sampled Gamma; controlled by $t$9	Deterministic; fixed rule
Exploration ctrl	Variable, tunable (via $\hat\mu_i(t)$0)	Fixed, often too large
Regret	Sub-linear (theor. & empirically lower)	Sub-linear but looser

3. Parametric UCB (Decay-Parameterized Exploration)

UCB$\hat\mu_i(t)$1 methods, as in "Little Exploration is All You Need" (Chen et al., 2023), generalize UCB by raising the exploration decay rate to an exponent $\hat\mu_i(t)$2:

$\hat\mu_i(t)$3

allowing direct control of the exploration decay. With proper tuning of $\hat\mu_i(t)$4 (depending on the gap $\hat\mu_i(t)$5 and noise), this formulation yields both logarithmic (distribution-dependent) and $\hat\mu_i(t)$6 (distribution-free) regret. Empirical evidence demonstrates that $\hat\mu_i(t)$7 consistently yields lower mean and risk of regret than $\hat\mu_i(t)$8 (UCB1) and rivals or outperforms Thompson Sampling across a broad class of bandit regimes (Chen et al., 2023).

Parameterization	Exploration Bonus	Regret Scaling
$\hat\mu_i(t)$9	$b_i(t)$0	$b_i(t)$1
$b_i(t)$2	$b_i(t)$3	$b_i(t)$4

4. Adaptive and Bayesian UCB for Distribution Estimation

Bayesian P-UCB, as studied in "Adaptive Sampling for Estimating Distributions" (Kartik et al., 2020), and prior UCB-on-variance schemes (Carpentier et al., 2015), target uniform estimation or allocation under a finite sample budget. Here, the index for each item/distribution is a credible Bayesian upper bound (e.g., for per-arm variance or mean-squared error), computed from Dirichlet-Beta posteriors or empirical Bernstein bounds. The sampling rule selects the distribution with largest upper confidence index for further sampling. These approaches achieve $b_i(t)$5 regret in uniform MSE loss, often outperforming classical UCB allocation methods under both synthetic and real-world data (e.g., adaptive group testing for SARS-CoV-2 seroprevalence) (Kartik et al., 2020, Carpentier et al., 2015).

5. P-UCB for Large-Scale Pure Exploration and Heavy Tails

Meta-UCB strategies, as analyzed in "UCB for Large-Scale Pure Exploration: Beyond Sub-Gaussianity" (Li et al., 27 Nov 2025), encompass a broad class of P-UCB-type policies for best-arm identification under minimal distributional assumptions (bounded variance or finite $b_i(t)$6th moment, with $b_i(t)$7). The essential form is

$b_i(t)$8

where $b_i(t)$9 is any decaying, arm-wise, validity-guaranteed bonus (e.g., $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$0 for bounded variance, or a union-bound-adapted $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$1 for finite $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$2th moment). The recommended selection rule is to choose, at stopping time, the arm with the most samples, or the one with the largest $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$3. This framework yields the optimal $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$4 sample complexity for $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$5 arms, extends to heavy-tailed settings, and is parallelizable (Li et al., 27 Nov 2025).

6. P-UCB in Molecular Simulation and Adaptive Bandits

In molecular simulation (e.g., "AdaptiveBandit: A multi-armed bandit framework..." (Pérez et al., 2020)), P-UCB denotes the use of UCB1-style rules for state-adaptive spawning: each possible MSM microstate is treated as an arm; empirical rewards are free energy proxies (e.g., $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$6), and the P-UCB selection is

$\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$7

with $\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$8 controlling the exploitation-exploration balance. Empirical evidence shows that this approach accelerates equilibrium sampling and is robust to misleading or goal-oriented priors, outperforming heuristic baselines on both toy potentials and realistic protein folding (Pérez et al., 2020).

7. Posterior UCB and Thompson Sampling

Posterior sampling (Thompson Sampling), sometimes referred to as "Posterior-UCB" (Russo et al., 2013), can be interpreted as an implicit randomized UCB: rather than maximizing an analytic confidence bound, one samples a random reward function (from the posterior), then selects the action maximizing that sample. Bayesian regret decomposes into sums of standard UCB-style confidence gap and optimism terms. This equivalence enables transfer of regret guarantees between UCB and Thompson sampling, with advantages in empirical regret and avoiding conservative over-exploration (Russo et al., 2013).

Summary Table: Main P-UCB Variants

Context/Problem	P-UCB Modality	Defining Formula/Pseudocode	Key Theoretical Property	Reference
Bayesian Optimization	Randomized GP-UCB, β~Gamma	$\text{UCB}_i(t) = \hat\mu_i(t) + b_i(t),$9	Sublinear Bayes regret; tunable	(Berk et al., 2020)
Multi-armed Bandit	UCB$b_i(t)$0, parametric decay	$b_i(t)$1	$b_i(t)$2 or $b_i(t)$3 regret	(Chen et al., 2023)
Uniform Estimation	UCB-on-variance / Bayesian UCB	$b_i(t)$4 or Bayesian credible bound	$b_i(t)$5 MSE regret	(Kartik et al., 2020, Carpentier et al., 2015)
Pure Exploration (large $b_i(t)$6)	Meta-UCB, decoupled, robust	$b_i(t)$7 ($b_i(t)$8 per moment control)	$b_i(t)$9 sample complexity	(Li et al., 27 Nov 2025)
Molecular Simulation	P-UCB (bandit) adaptive sampling	$\mu_i$0	Logarithmic regret, fast equil.	(Pérez et al., 2020)
Bayesian Bandit/General	Thompson Sampling (Posterior-UCB)	Sample $\mu_i$1 from posterior, play $\mu_i$2	Bayesian regret $\mu_i$3 etc.	(Russo et al., 2013)

• "Randomised Gaussian Process Upper Confidence Bound for Bayesian Optimisation" (Berk et al., 2020)
• "Little Exploration is All You Need" (Chen et al., 2023)
• "Adaptive Sampling for Estimating Distributions: A Bayesian Upper Confidence Bound Approach" (Kartik et al., 2020)
• "Upper-Confidence-Bound Algorithms for Active Learning in Multi-Armed Bandits" (Carpentier et al., 2015)
• "UCB for Large-Scale Pure Exploration: Beyond Sub-Gaussianity" (Li et al., 27 Nov 2025)
• "Learning to Optimize Via Posterior Sampling" (Russo et al., 2013)
• "AdaptiveBandit: A multi-armed bandit framework for adaptive sampling in molecular simulations" (Pérez et al., 2020)