---
title: Finite-Arm GP-UCB
url: https://www.emergentmind.com/topics/finite-arm-gp-ucb
type: topic
---

# Finite-Arm GP-UCB

Finite-Arm GP-UCB refers to Gaussian Process Upper Confidence Bound algorithms for Bayesian optimization and bandit problems where the arm set $\mathcal{X}$ is finite. In this regime, the agent sequentially selects arms from a fixed finite set, and observations are modeled as $y_t = f(x_t) + \epsilon_t$, with $f$ following a Gaussian process (GP) prior and $\epsilon_t$ additive noise. Finite-armed GP-UCB provides strong regret guarantees with efficient computation, and, in recent research, variants with randomized confidence parameters have further improved regret bounds and reduced over-exploration.

## 1. Problem Setting: Bayesian Optimization with Finite Arms

The setting consists of a finite domain $\mathcal{X} = \{x^1, \dots, x^{|\mathcal{X}|}\} \subset \mathbb{R}^d$. The unknown reward function $f$ is drawn from a GP prior with mean zero and covariance $k(\cdot,\cdot)$ satisfying $k(x,x) \leq 1$ for all $x$, and the learner observes $y_t = f(x_t) + \epsilon_t$ at each round $t$, with i.i.d. noise $\epsilon_t \sim \mathcal{N}(0, \sigma^2)$. The aim is to minimize cumulative regret $R_T = \sum_{t=1}^T [f(x^*) - f(x_t)]$, where $x^*$ is an arm maximizing $f$ over $\mathcal{X}$ [2409.00979, 2512.18409].

## 2. The Standard Finite-Arm GP-UCB Algorithm

GP-UCB is established for maximizing an unknown function under GP priors. At each round $t$, posterior mean $\mu_{t-1}(x)$ and variance $\sigma_{t-1}^2(x)$ are computed using past data. The classic GP-UCB rule selects:

\[
x_t = \arg\max_{x \in \mathcal{X}} \left[ \mu_{t-1}(x) + \sqrt{\beta_t} \, \sigma_{t-1}(x) \right]
\]

where $\beta_t$ is a deterministic confidence parameter, typically chosen to ensure a concentration inequality over all arms and all rounds, e.g. $\beta_t = O(\log(|\mathcal{X}| t /\delta))$ [2512.18409, 2506.10091]. The resulting algorithm is computationally tractable on finite domains, as all required GP quantities can be efficiently updated [2506.10091].

## 3. Regret Analysis and Information Gain

A central result is that the concentration inequality

\[
\Pr\left(\forall x, t: |f(x) - \mu_{t-1}(x)| \leq \sqrt{\beta_t}\, \sigma_{t-1}(x)\right) \geq 1 - \delta
\]

holds with a judicious choice of $\beta_t$. Regret can be decomposed into two analytical components:

- **Instance-dependent bound:** Each suboptimal arm $x$ is chosen at most $O(\beta_T \sigma^2 / \Delta_x^2)$ times, where $\Delta_x = f(x^*) - f(x)$.
- **Distribution-free bound:** The cumulative regret satisfies

\[
R_T = O(\sqrt{T \beta_T \gamma_T})
\]

with $\gamma_T$ the maximum mutual information (information gain) between $f$ and $T$ observations, i.e.

\[
\gamma_T = \max_{A \subset \mathcal{X}, |A|=T} \frac{1}{2}\log \det(I + \sigma^{-2} K_{A,A})
\]

where $K_{A,A}$ is the Gram matrix restricted to the set $A$ [2512.18409, 2506.10091]. For finite $\mathcal{X}$, $\gamma_T = O(\log |\mathcal{X}|)$.

Table 1: Key analytical ingredients in finite-arm GP-UCB

| Ingredient             | Formula/Description                                                  | Reference      |
|------------------------|---------------------------------------------------------------------|---------------|
| Posterior mean         | $\mu_{t-1}(x) = k_{t-1}(x)^\top V_{t-1}^{-1} y_{1:t-1}$             | [2512.18409]  |
| Posterior variance     | $\sigma_{t-1}^2(x) = k(x,x) - k_{t-1}(x)^\top V_{t-1}^{-1} k_{t-1}(x)$ | [2512.18409]  |
| Regret bound           | $R_T = O(\sqrt{T \beta_T \gamma_T})$                                | [2512.18409]  |
| Max info. gain         | $\gamma_T = \max_{A \subset \mathcal{X}} \frac{1}{2} \log \det(I + \sigma^{-2}K_{A,A})$ | [2506.10091]  |

## 4. Improved Randomized GP-UCB (IRGP-UCB)

A recognized limitation of classic GP-UCB is the need for a confidence parameter $\beta_t$ that grows with $t$ to maintain high-probability bounds. This growth is conservative and leads to excessive long-term exploration [2409.00979].

**IRGP-UCB** replaces $\beta_t$ by a random variable $\zeta_t$ sampled from a shifted exponential distribution $\zeta_t \sim s_t + \mathrm{Exp}(\lambda)$, with $s_t = 2 \log(|\mathcal{X}|/2)$ and $\lambda = 1/2$. This results in:

- The expected confidence parameter is constant in $t$, so the width of the exploration bonus does not grow.
- The regret bound (Bayesian cumulative regret and high-probability regret) becomes

\[
\mathrm{BCR}_T \leq O\left(\sqrt{T \gamma_T \log |\mathcal{X}|}\right)
\]

This is asymptotically sharper in $T$ than standard GP-UCB, where $R_T = O(\sqrt{T \gamma_T \log T})$ [2409.00979]. Numerical studies on synthetic and real datasets confirm IRGP-UCB achieves faster convergence and consistently lower regret.

## 5. Deterministic Proof Ingredients: Concentration, Radius Collapse, and Forced Deviations

The regret proof for finite-arm GP-UCB is structurally identical to that for finite-armed UCB and linear UCB algorithms [2512.18409]. The analysis consists of:

- **A single uniform concentration bound** (for posterior deviation),
- **Radius collapse:** After sufficient pulls, a suboptimal arm's posterior variance (and hence its bonus) falls below a threshold proportional to its suboptimality gap,
- **Optimism-forced deviation:** The selection rule cannot pick suboptimal arms after enough information is acquired, unless the confidence bound fails,
- Combining these, one derives both instance-dependent (logarithmic) and distribution-free ("oracle") regret bounds.

This structural unity extends across most optimism-based finite bandit algorithms [2512.18409].

## 6. Practical Considerations and Computational Aspects

With moderate $|\mathcal{X}|$, GP-UCB and IRGP-UCB are computationally efficient. The $K \times K$ Gram matrix can be cached, and Cholesky or inverse updates are $O(K^2)$ per round [2506.10091]. No discretization or approximate maximization is required. The regularization via the GP kernel smooths out estimates, exploiting correlations between arms for more sample-efficient exploration.

Typical applications include finite-design experimental settings, algorithm configuration, and resource allocation with categorical options [2409.00979].

## 7. Comparison of IRGP-UCB and Classic GP-UCB

IRGP-UCB eliminates the need for time-growing confidence bonuses on finite domains. This removes late-iteration over-exploration and provides tighter bounds. In particular:

- **Exploration bonus:** Fixed in IRGP-UCB; grows with time in GP-UCB.
- **Regret rate:** $O(\sqrt{T \gamma_T \log |\mathcal{X}|})$ for IRGP-UCB, vs. $O(\sqrt{T \gamma_T \log T})$ for GP-UCB.
- **Empirical performance:** In all reported experiments with finite $\mathcal{X}$, IRGP-UCB outperforms GP-UCB, Thompson Sampling, and other state-of-the-art BO algorithms in sample complexity and regret [2409.00979].

A plausible implication is that, for the finite-arm setting, randomization in the confidence parameter eliminates the necessity for conservative high-probability guarantees, yielding both theoretical and practical benefits.

---

References:  
[2409.00979], [2506.10091], [2512.18409]

Source: https://www.emergentmind.com/topics/finite-arm-gp-ucb