Threshold-Based Bandit Problem

• Threshold-based bandit problems are formulations where arms are classified as acceptable if their mean rewards exceed a given threshold rather than maximizing cumulative reward.
• They employ fixed-budget and fixed-confidence setups with algorithms like APT, LSA, and variance-aware elimination to efficiently allocate samples and reduce misclassification.
• The research informs decision boundaries, instance complexity, and algorithmic paradigms that adapt to structured environments and practical operational constraints.

Threshold-based bandit problem denotes a family of bandit formulations in which performance is defined relative to a prescribed threshold rather than solely by maximization of expected reward. In the canonical stochastic Thresholding Bandit Problem (TBP), a learner sequentially samples arms and, after a finite budget, must identify the set of arms whose means exceed a threshold; related formulations replace final-set identification by fixed-confidence certification, cumulative satisficing relative to a threshold level, threshold-activated cooperative rewards, or posterior cut-off policies in continuous time (Locatelli et al., 2016, Rivera et al., 2024, Feng et al., 2024, Ledford et al., 18 Jun 2025, Pourbabaee, 2021). This suggests that threshold-based bandits are best viewed as a unifying decision principle—classification, continuation, or coordination relative to a threshold—rather than as a single objective.

1. Canonical formulations

In the fixed-budget stochastic TBP, there are $K$ arms with means $\mu_i$, a threshold $\tau$, and optionally a precision parameter $\epsilon \ge 0$. The target set is

$G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$

and the learner outputs $\widehat{S}_\tau$ after $T$ rounds. With tolerance $\epsilon$, the loss is

$$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$

so $\mathbb{E}[L(T)]$ is the probability of at least one consequential misclassification (Locatelli et al., 2016).

A distinct fixed-budget formulation evaluates the expected worst misclassification margin rather than the event of any error. Let $\mu_i$0 and $\mu_i$1. Then the simple-regret criterion is

$\mu_i$2

Under this view, TBP becomes a level-set classification problem whose minimax difficulty depends strongly on structural assumptions on the mean sequence (Cheshire et al., 2020).

Fixed-confidence thresholding in linear bandits replaces armwise means by a shared linear model. Each arm $\mu_i$3 has feature vector $\mu_i$4, rewards satisfy $\mu_i$5, and the good-arm set is

$\mu_i$6

An $\mu_i$7-correct algorithm must output $\mu_i$8 such that

$\mu_i$9

with almost-sure finite stopping time (Rivera et al., 2024).

A different threshold-based objective appears in satisficing bandits. Here the learner seeks arms with mean reward at least $\tau$0, and performance is measured by satisficing regret,

$\tau$1

This changes the role of the threshold from terminal classification to repeated online acceptability (Feng et al., 2024).

Formulation	Threshold acts on	Criterion
Fixed-budget TBP	Final arm classification	$\tau$2
Structured TBP	Final label vector	$\tau$3 or error probability
Fixed-confidence linear TBP	Certified good-arm set	$\tau$4-correctness
Satisficing bandit	Online action quality	$\tau$5

2. Complexity measures and optimal rates

For the classical fixed-budget TBP, the central per-arm gap is

$\tau$6

The instance complexity $\tau$7 governs the optimal error exponent. For $\tau$8-sub-Gaussian arms, APT attains

$\tau$9

while the minimax lower bound has the form

$\epsilon \ge 0$0

establishing the fixed-budget TBP rate $\epsilon \ge 0$1 up to constants and logarithmic factors (Locatelli et al., 2016).

Aggregate-regret thresholding leads to a different complexity description. If $\epsilon \ge 0$2 is the expected number of misclassified arms, the relevant offline program is

$\epsilon \ge 0$3

The lower bound

$\epsilon \ge 0$4

shows that aggregate regret is controlled by an optimal error-allocation program rather than only by the event of at least one mistake; the LSA algorithm is instance-wise asymptotically optimal with respect to this benchmark (Tao et al., 2019).

Shape constraints fundamentally modify thresholding hardness. In the problem-independent fixed-budget regime, the minimax expected simple regret is $\epsilon \ge 0$5 for unstructured TBP, $\epsilon \ge 0$6 for monotone TBP, $\epsilon \ge 0$7 for unimodal TBP, and $\epsilon \ge 0$8 for concave TBP (Cheshire et al., 2020). In the problem-dependent regime, both monotone and concave TBP collapse to hardness $\epsilon \ge 0$9, where $G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$0, with upper and lower bounds matching in the exponential rate up to constants and additive $G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$1 terms in the exponent (Cheshire et al., 2021). A plausible implication is that structural information changes thresholding more radically than it changes standard regret minimization, because the decision boundary is localized.

3. Algorithmic paradigms

The canonical fixed-budget algorithm is APT, “Anytime Parameter-free Thresholding.” After pulling each arm once, it computes

$G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$2

and samples

$G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$3

The heuristic equalizes $G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$4, approximating the static optimal allocation $G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$5 without knowing $G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$6, $G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$7, or even the horizon in advance (Locatelli et al., 2016).

A second paradigm targets aggregate misclassification rather than simple error. LSA selects

$G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$8

combining a work term with a logarithmic incentive term. The logarithmic component is not a generic exploration bonus; it is tied to the KKT structure of the offline aggregate-regret program and yields instance-wise asymptotic optimality (Tao et al., 2019).

Variance-aware elimination leads to AugUCB. It maintains empirical means and variances and uses

$G_\tau := \{ i \in [K] : \mu_i \ge \tau \},$9

then samples

$\widehat{S}_\tau$0

Arms are eliminated when $\widehat{S}_\tau$1 lie confidently on one side of the threshold. Its upper bound is expressed through the variance-aware complexity $\widehat{S}_\tau$2, and its principal practical feature is that it does not require oracle knowledge of that complexity, unlike UCBEV (Mukherjee et al., 2017).

A more abstract line of work treats thresholding as a special case of active exploration. In the Gaussian fixed-confidence setting,

$\widehat{S}_\tau$3

with $\widehat{S}_\tau$4. An online lazy mirror-ascent algorithm tracks the optimal allocation on the simplex, uses forced exploration and a Chernoff-type stopping rule, and is asymptotically optimal: $\widehat{S}_\tau$5 In thresholding, this recovers the intuition that arms near the boundary dominate the fixed-confidence sample complexity (Ménard, 2019).

4. Structure and linear generalization

Structured thresholding algorithms exploit order, unimodality, or concavity to localize the threshold crossing. In monotone TBP, STB implements a noisy binary search with corrections on a binary tree over indices; in unimodal TBP, UTB first localizes the peak and then applies monotone threshold searches on both sides; in concave TBP, CTB uses phased refinement on dyadic “log-sets” near interval endpoints. These constructions are designed so that coarse probes already reveal large regions that must lie above or below the threshold under the assumed geometry (Cheshire et al., 2020).

Linear thresholding replaces armwise means by shared estimation of $\widehat{S}_\tau$6. In the fixed-confidence setting, the fundamental instance-dependent lower bound is

$\widehat{S}_\tau$7

where

$\widehat{S}_\tau$8

Lazy Track-Threshold-and-Stop extends linear Track-and-Stop to thresholding, combining forced exploration, optimal design tracking, and a GLRT-style stopping rule, and is asymptotically optimal in both almost-sure and expected senses (Rivera et al., 2024).

The fixed-budget linear counterpart, LinearAPT, adapts the APT ambiguity rule to a global ridge-regression estimate: $\widehat{S}_\tau$9 Under $T$0-sub-Gaussian noise and bounded features, its expected loss is bounded by

$T$1

with $T$2. The dependence on $T$3 rather than on $T$4 reflects the shift from independent-arm estimation to shared linear generalization (Wu et al., 2024).

5. Cumulative threshold objectives

Thresholds can define online performance deficits rather than only terminal classification. In satisficing bandits, an arm is acceptable when $T$5, the realizable case is $T$6, and the key gap is the exceeding gap

$T$7

SELECT builds on any oracle with sub-linear standard regret, uses roundwise candidate identification, forced sampling, and lower-confidence-bound testing against $T$8, and achieves

$T$9

in the realizable case, while matching the oracle’s standard-regret rate in the non-realizable case (Feng et al., 2024). The dependence on $\epsilon$0 rather than on the classical satisficing gap $\epsilon$1 is essential in infinite or continuous arm spaces.

In finite-horizon restless bandits, thresholding can be used as a surrogate objective. The reduction defines an incremental action benefit

$\epsilon$2

labels agent-state pairs with $\epsilon$3 as “good,” and measures regret through threshold violations under a budget constraint. The LCB-guided randomized thresholding algorithm has regret

$\epsilon$4

yielding a constant-regret regime when sufficiently many good agent-state pairs are always available (Xu et al., 7 Feb 2025).

A more radical operational threshold is the survival constraint. In the Survival Bandit Problem, the budget process

$\epsilon$5

induces ruin time

$\epsilon$6

No policy can achieve uniformly sublinear survival regret, so the relevant optimality notion is Pareto-optimality across instances. The key quantity is the ruin index

$\epsilon$7

and EXPLOIT-UCB-DOUBLE with $\epsilon$8 is regret-wise Pareto-optimal (Riou et al., 2022). This is a thresholded bandit only in an operational sense—the threshold acts on cumulative budget rather than on arm means—but it demonstrates how threshold constraints can fundamentally alter the admissible regret notion.

6. Specialized extensions and continuous-time interpretations

Thresholding can be enriched by alternative feedback modalities. In TBP with Dueling Choices, the learner may both pull arms and duel pairs. Rank-Search alternates between approximate ranking by Borda scores and binary search by pulls, achieving

$\epsilon$9

with $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$0. The use of duels reduces the pull dependence from $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$1 to $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$2 when duel separation is favorable (Xu et al., 2019).

Multi-agent threshold activation introduces a different threshold semantics: an arm pays only when at least $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$3 agents pull it simultaneously. In the decentralized T-Coop-UCB framework, agents jointly estimate thresholds $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$4 and rewards $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$5, form coalitions via synchronized UCB indices, and empirically approach near-Oracle performance, but the paper does not provide formal regret bounds (Ledford et al., 18 Jun 2025). The more explicit TAC-MAB model treats censored threshold activation, where sub-threshold executions always return zero, and proves that centralized C-TAC achieves

$$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$6

while decentralized D-TAC requires only $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$7 structural synchronization events and yields a reported $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$8 communication reduction relative to the centralized baseline (Ledford et al., 26 May 2026).

Continuous-time bandit theory uses thresholds as posterior cut-offs. In the robust two-armed model with ambiguity aversion, the optimal policy is a belief-threshold rule: there exists $$L(T)=\mathbf{1}\!\left((S_{\tau+\epsilon}\cap \widehat{S}_\tau^C \neq \emptyset)\ \vee\ (S_{\tau-\epsilon}^C \cap \widehat{S}_\tau \neq \emptyset)\right),$$9 such that the safe arm is selected if and only if the posterior belief falls below $\mathbb{E}[L(T)]$0, and $\mathbb{E}[L(T)]$1 increases with ambiguity aversion and with the addition of an unambiguous expert-information source (Pourbabaee, 2021). In the two-armed Lévy bandit, the risky arm is also governed by an explicit posterior cut-off $\mathbb{E}[L(T)]$2; the optimal policy experiments if and only if $\mathbb{E}[L(T)]$3, with $\mathbb{E}[L(T)]$4 determined by the Lévy triplets and the safe payoff (Cohen et al., 2014). These models are not thresholding bandits in the pure-exploration sense, but they show that threshold rules are equally fundamental in continuous-time experimentation.

A further specialized line studies monotonic bandits where the target is not the whole above-threshold set but the first arm above $\mathbb{E}[L(T)]$5, the $\mathbb{E}[L(T)]$6-th arm above or below $\mathbb{E}[L(T)]$7, or the arm closest to $\mathbb{E}[L(T)]$8. The asymptotic regret lower bounds depend only on arms adjacent to the threshold crossing, and KL-UCB-style algorithms focus sampling on those critical neighbors (Varude et al., 2 Sep 2025). This sharp localization is consistent with the broader thresholding literature: once the threshold boundary is the central object, arms far from it matter primarily through the structure they induce.