---
title: Threshold-Based Bandit Problem
url: https://www.emergentmind.com/topics/threshold-based-bandit-problem
type: topic
---

# Threshold-Based Bandit Problem

Threshold-based bandit problem denotes a family of bandit formulations in which performance is defined relative to a prescribed threshold rather than solely by maximization of expected reward. In the canonical stochastic Thresholding Bandit Problem (TBP), a learner sequentially samples arms and, after a finite budget, must identify the set of arms whose means exceed a threshold; related formulations replace final-set identification by fixed-confidence certification, cumulative satisficing relative to a threshold level, threshold-activated cooperative rewards, or posterior cut-off policies in continuous time [1605.08671][2402.09467][2406.06802][2506.15856][2104.00102]. This suggests that threshold-based bandits are best viewed as a unifying decision principle—classification, continuation, or coordination relative to a threshold—rather than as a single objective.

## 1. Canonical formulations

In the fixed-budget stochastic TBP, there are \(K\) arms with means \(\mu_i\), a threshold \(\tau\), and optionally a precision parameter \(\epsilon \ge 0\). The target set is
\[
G_\tau := \{ i \in [K] : \mu_i \ge \tau \},
\]
and the learner outputs \(\widehat{S}_\tau\) after \(T\) rounds. With tolerance \(\epsilon\), the loss is
\[
L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),
\]
so \(\mathbb{E}[L(T)]\) is the probability of at least one consequential misclassification [1605.08671].

A distinct fixed-budget formulation evaluates the *expected worst misclassification margin* rather than the event of any error. Let \(\Delta_k := |\mu_k-\tau|\) and \(M_T:=\{k:\widehat Q_k\neq Q_k\}\). Then the simple-regret criterion is
\[
R_T^{\nu,\pi}=\mathbb{E}_\nu\!\left[\max_{k\in M_T}\Delta_k\right].
\]
Under this view, TBP becomes a level-set classification problem whose minimax difficulty depends strongly on structural assumptions on the mean sequence [2006.10006].

Fixed-confidence thresholding in linear bandits replaces armwise means by a shared linear model. Each arm \(a\) has feature vector \(x_a\), rewards satisfy \(r_t=x_{a_t}^\top \theta+\eta_t\), and the good-arm set is
\[
\Pi_\rho(\theta)=\{a: x_a^\top \theta > \rho\}.
\]
An \((\epsilon,\delta)\)-correct algorithm must output \(\phi\) such that
\[
\mathbb{P}\!\left[\Pi_{\rho+\epsilon}(\theta)\subseteq \phi \ \land\ (\Pi_{\rho-\epsilon}(\theta))^c \subseteq \phi^c\right]\ge 1-\delta
\]
with almost-sure finite stopping time [2402.09467].

A different threshold-based objective appears in satisficing bandits. Here the learner seeks arms with mean reward at least \(S\), and performance is measured by satisficing regret,
\[
\operatorname{Regret}_S=\mathbb{E}\!\left[\sum_{t=1}^T \max\{S-r(X_t),0\}\right].
\]
This changes the role of the threshold from terminal classification to repeated online acceptability [2406.06802].

| Formulation | Threshold acts on | Criterion |
|---|---|---|
| Fixed-budget TBP | Final arm classification | \(\mathbb{E}[L(T)]\) |
| Structured TBP | Final label vector | \(\mathbb{E}[\max_{k\in M_T}\Delta_k]\) or error probability |
| Fixed-confidence linear TBP | Certified good-arm set | \((\epsilon,\delta)\)-correctness |
| Satisficing bandit | Online action quality | \(\operatorname{Regret}_S\) |

## 2. Complexity measures and optimal rates

For the classical fixed-budget TBP, the central per-arm gap is
\[
\Delta_i^{\tau,\epsilon}=|\mu_i-\tau|+\epsilon,
\qquad
H=\sum_{i=1}^K (\Delta_i^{\tau,\epsilon})^{-2}.
\]
The instance complexity \(H\) governs the optimal error exponent. For \(R\)-sub-Gaussian arms, APT attains
\[
\mathbb{E}[L(T)] \le \exp\!\left(-\frac{1}{64R^2}\frac{T}{H}+2\log((\log T+1)K)\right),
\]
while the minimax lower bound has the form
\[
\exp\!\left(-3\frac{T}{H}-4\log(12(\log T+1)K)\right),
\]
establishing the fixed-budget TBP rate \(\exp(-\Theta(T/H))\) up to constants and logarithmic factors [1605.08671].

Aggregate-regret thresholding leads to a different complexity description. If \(\mathcal{R}^{\mathbb A}(I;T)\) is the expected number of misclassified arms, the relevant offline program is
\[
\mathscr{P}_c(\{\Delta_i\},T)=\min_{\substack{x_1+\cdots+x_K=T\\x_i\ge 0}}
\sum_{i=1}^K \exp(-c x_i \Delta_i^2).
\]
The lower bound
\[
\mathcal{R}^{\mathbb A}(I;T)\ge \tfrac14\,\mathscr{P}_{16}(\{\Delta_i\},T)
\]
shows that aggregate regret is controlled by an optimal error-allocation program rather than only by the event of at least one mistake; the LSA algorithm is instance-wise asymptotically optimal with respect to this benchmark [1905.11046].

Shape constraints fundamentally modify thresholding hardness. In the problem-independent fixed-budget regime, the minimax expected simple regret is \(\sqrt{\log(K)K/T}\) for unstructured TBP, \(\sqrt{\log(K)/T}\) for monotone TBP, \(\sqrt{K/T}\) for unimodal TBP, and \(\sqrt{\log\log K/T}\) for concave TBP [2006.10006]. In the problem-dependent regime, both monotone and concave TBP collapse to hardness \(1/\Delta_{\min}^2\), where \(\Delta_{\min}=\min_k |\mu_k-\tau|\), with upper and lower bounds matching in the exponential rate up to constants and additive \(O(\log K)\) terms in the exponent [2106.10166]. A plausible implication is that structural information changes thresholding more radically than it changes standard regret minimization, because the decision boundary is localized.

## 3. Algorithmic paradigms

The canonical fixed-budget algorithm is APT, “Anytime Parameter-free Thresholding.” After pulling each arm once, it computes
\[
\widehat{\Delta}_i(t)=|\widehat{\mu}_i(t)-\tau|+\epsilon,
\qquad
B_i(t)=\sqrt{T_i(t)}\,\widehat{\Delta}_i(t),
\]
and samples
\[
I_t \in \arg\min_{i\in[K]} B_i(t).
\]
The heuristic equalizes \(T_i(t)\widehat{\Delta}_i(t)^2\), approximating the static optimal allocation \(T_i\propto 1/\Delta_i^2\) without knowing \(H\), \(R\), or even the horizon in advance [1605.08671].

A second paradigm targets *aggregate* misclassification rather than simple error. LSA selects
\[
i_t \in \arg\min_{i\in S}\left\{\alpha\,T_i(t-1)\bigl(\widehat{\Delta}_i(t-1)\bigr)^2+\tfrac12\ln T_i(t-1)\right\},
\]
combining a work term with a logarithmic incentive term. The logarithmic component is not a generic exploration bonus; it is tied to the KKT structure of the offline aggregate-regret program and yields instance-wise asymptotic optimality [1905.11046].

Variance-aware elimination leads to AugUCB. It maintains empirical means and variances and uses
\[
s_i(t)=\sqrt{\frac{\rho\,\psi_m\,(\widehat{\sigma}_i^2(t)+1)\,\log(T\epsilon_m)}{4\,n_i(t)}},
\]
then samples
\[
j\in \arg\min_{i\in B_m}\Big\{|\widehat{\mu}_i(t)-\tau|-2s_i(t)\Big\}.
\]
Arms are eliminated when \(\widehat{\mu}_i(t)\pm 2s_i(t)\) lie confidently on one side of the threshold. Its upper bound is expressed through the variance-aware complexity \(H_{\sigma,2}\), and its principal practical feature is that it does not require oracle knowledge of that complexity, unlike UCBEV [1704.02281].

A more abstract line of work treats thresholding as a special case of active exploration. In the Gaussian fixed-confidence setting,
\[
F(w,\mu)=\min_{a\in[K]} w_a\, d(\mu_a,T),
\qquad
T^*(\mu)=\sum_{a=1}^K \frac{1}{d(\mu_a,T)},
\]
with \(d(\mu_1,\mu_2)=(\mu_1-\mu_2)^2/2\). An online lazy mirror-ascent algorithm tracks the optimal allocation on the simplex, uses forced exploration and a Chernoff-type stopping rule, and is asymptotically optimal:
\[
\limsup_{\delta\to 0}\frac{\mathbb{E}_\mu[\tau_\delta]}{\log(1/\delta)}\le T^*(\mu).
\]
In thresholding, this recovers the intuition that arms near the boundary dominate the fixed-confidence sample complexity [1905.08165].

## 4. Structure and linear generalization

Structured thresholding algorithms exploit order, unimodality, or concavity to localize the threshold crossing. In monotone TBP, STB implements a noisy binary search with corrections on a binary tree over indices; in unimodal TBP, UTB first localizes the peak and then applies monotone threshold searches on both sides; in concave TBP, CTB uses phased refinement on dyadic “log-sets” near interval endpoints. These constructions are designed so that coarse probes already reveal large regions that must lie above or below the threshold under the assumed geometry [2006.10006].

Linear thresholding replaces armwise means by shared estimation of \(\theta\). In the fixed-confidence setting, the fundamental instance-dependent lower bound is
\[
\mathbb{E}[\tau]\ge \sigma^2 \log(1/(2\delta))\, T_\theta^\star,
\]
where
\[
T_\theta^\star
=
\min_{\lambda\in\Lambda}\max_{a\in A_{\rho,\epsilon}(\theta)}
\frac{2\|x_a\|_{A_\lambda^{-1}}^2}{(|x_a^\top \theta-\rho|+\epsilon)^2}.
\]
Lazy Track-Threshold-and-Stop extends linear Track-and-Stop to thresholding, combining forced exploration, optimal design tracking, and a GLRT-style stopping rule, and is asymptotically optimal in both almost-sure and expected senses [2402.09467].

The fixed-budget linear counterpart, LinearAPT, adapts the APT ambiguity rule to a global ridge-regression estimate:
\[
\widehat{\theta}_t = V_t^{-1} b_t,
\qquad
\widehat{\mu}_i(t)=x_i^\top \widehat{\theta}_t,
\qquad
B_i(t)=\sqrt{T_i(t-1)}\,\widehat{\Delta}_i(t-1).
\]
Under \(1\)-sub-Gaussian noise and bounded features, its expected loss is bounded by
\[
\mathbb{E}[\mathcal{L}(T)]
\le
\exp\!\left\{\log(1+TL^2)-\frac1d\left(\sqrt{\frac{T}{\gamma^2H}}-\|\theta\|\right)^2\right\},
\]
with \(\gamma=4\). The dependence on \(d\) rather than on \(K\) reflects the shift from independent-arm estimation to shared linear generalization [2403.06230].

## 5. Cumulative threshold objectives

Thresholds can define *online performance deficits* rather than only terminal classification. In satisficing bandits, an arm is acceptable when \(r(X)\ge S\), the realizable case is \(r(X^*)\ge S\), and the key gap is the exceeding gap
\[
\Delta_S^\star=r(X^*)-S.
\]
SELECT builds on any oracle with sub-linear standard regret, uses roundwise candidate identification, forced sampling, and lower-confidence-bound testing against \(S\), and achieves
\[
\operatorname{Regret}_S
\le
\min\!\left\{
C_1^{1/(1-\alpha)}
\left(\frac1{\Delta_S^\star}\right)^{\alpha/(1-\alpha)}
\operatorname{polylog}(C_1/\Delta_S^\star),
\;
C_1 T^\alpha \operatorname{polylog}(T)
\right\}
\]
in the realizable case, while matching the oracle’s standard-regret rate in the non-realizable case [2406.06802]. The dependence on \(\Delta_S^\star\) rather than on the classical satisficing gap \(\Delta_S\) is essential in infinite or continuous arm spaces.

In finite-horizon restless bandits, thresholding can be used as a surrogate objective. The reduction defines an incremental action benefit
\[
I^m(s)=\phi^m(s,1)-\phi^m(s,0),
\]
labels agent-state pairs with \(I^m(s)\ge \gamma\) as “good,” and measures regret through threshold violations under a budget constraint. The LCB-guided randomized thresholding algorithm has regret
\[
\mathcal{R}(\pi)=
\min\!\left\{
\mathcal{O}\!\left(\frac{2M-|\mathcal G|}{\Delta_{s_{\min}}^{m_{\min}}}\right),
\;
\mathcal{O}\!\left(\sqrt{B|\mathcal G|(2M-|\mathcal G|)T}\right)
\right\},
\]
yielding a constant-regret regime when sufficiently many good agent-state pairs are always available [2502.05145].

A more radical operational threshold is the survival constraint. In the Survival Bandit Problem, the budget process
\[
B_t=B+\sum_{s=1}^t X_s^{\pi_s}
\]
induces ruin time
\[
\tau(B,\pi)=\inf\left\{t\ge 1: B+\sum_{s=1}^t X_s^{\pi_s}\le 0\right\}.
\]
No policy can achieve uniformly sublinear survival regret, so the relevant optimality notion is Pareto-optimality across instances. The key quantity is the ruin index
\[
\gamma(F_k)=\inf_{Q:\, \mathbb E_Q[X]<0}\frac{KL(Q\|F_k)}{\mathbb E_Q[-X]},
\]
and EXPLOIT-UCB-DOUBLE with \(n=\log T\) is regret-wise Pareto-optimal [2206.03019]. This is a thresholded bandit only in an operational sense—the threshold acts on cumulative budget rather than on arm means—but it demonstrates how threshold constraints can fundamentally alter the admissible regret notion.

## 6. Specialized extensions and continuous-time interpretations

Thresholding can be enriched by alternative feedback modalities. In TBP with Dueling Choices, the learner may both pull arms and duel pairs. Rank-Search alternates between approximate ranking by Borda scores and binary search by pulls, achieving
\[
n_{\text{duel}}
\le
32\,\mathcal{C}_{d,2}\,
\log\!\left(\frac{4K\log(1/\bar{\Delta}_{\min})}{\delta}\right),
\qquad
n_{\text{pull}}
\le
16R^2 H_\ell(n_\ell)\,
\log\!\left(\frac{n_\ell\log(1/\delta)}{\delta}\right),
\]
with \(n_\ell=O(\log K\cdot \log(1/\bar{\Delta}_{\min}))\). The use of duels reduces the pull dependence from \(O(K)\) to \(O(\log K)\) when duel separation is favorable [1910.06368].

Multi-agent threshold activation introduces a different threshold semantics: an arm pays only when at least \(h_i\) agents pull it simultaneously. In the decentralized T-Coop-UCB framework, agents jointly estimate thresholds \(\widehat h_i\) and rewards \(\widehat\mu_i\), form coalitions via synchronized UCB indices, and empirically approach near-Oracle performance, but the paper does not provide formal regret bounds [2506.15856]. The more explicit TAC-MAB model treats censored threshold activation, where sub-threshold executions always return zero, and proves that centralized C-TAC achieves
\[
R(T)=\mathcal{O}\!\left(
\sum_{k=1}^K \frac{\min(\tau_k,M)\log T}{p_{\min}\mu^\star}
+
\sum_{k:\tau_k\le M} \frac{v_k^2\log T}{\Delta_k}
+
K\Delta_{\max}
\right),
\]
while decentralized D-TAC requires only \(O(KM)\) structural synchronization events and yields a reported \(23\times\) communication reduction relative to the centralized baseline [2605.27076].

Continuous-time bandit theory uses thresholds as posterior cut-offs. In the robust two-armed model with ambiguity aversion, the optimal policy is a belief-threshold rule: there exists \(\bar p\in[0,1]\) such that the safe arm is selected if and only if the posterior belief falls below \(\bar p\), and \(\bar p\) increases with ambiguity aversion and with the addition of an unambiguous expert-information source [2104.00102]. In the two-armed Lévy bandit, the risky arm is also governed by an explicit posterior cut-off \(p^\star\); the optimal policy experiments if and only if \(p>p^\star\), with \(p^\star\) determined by the Lévy triplets and the safe payoff [1407.7241]. These models are not thresholding bandits in the pure-exploration sense, but they show that threshold rules are equally fundamental in continuous-time experimentation.

A further specialized line studies monotonic bandits where the target is not the whole above-threshold set but the *first arm above \(\tau\)*, the \(k\)-th arm above or below \(\tau\), or the arm *closest* to \(\tau\). The asymptotic regret lower bounds depend only on arms adjacent to the threshold crossing, and KL-UCB-style algorithms focus sampling on those critical neighbors [2509.02119]. This sharp localization is consistent with the broader thresholding literature: once the threshold boundary is the central object, arms far from it matter primarily through the structure they induce.

Source: https://www.emergentmind.com/topics/threshold-based-bandit-problem