---
title: Controlled Stochastic Bandit Problems
url: https://www.emergentmind.com/topics/controlled-stochastic-bandit-setting
type: topic
---

# Controlled Stochastic Bandit Problems

Searching arXiv for recent and relevant papers on controlled stochastic bandit settings to ground the article in published work.
The controlled stochastic bandit setting is a family of sequential decision problems in which a learner chooses actions, queries, or controls that directly determine what stochastic information is observed, and then uses those observations to optimize a performance criterion such as cumulative reward, cumulative cost, simple regret, or a safety-constrained objective. In the simplest stochastic \(K\)-armed model, the control is the arm-selection sequence \(i(t)\) and rewards are independent random variables with arm-dependent means [1903.07839]. In stochastic convex bandit optimization, the control is the query point \(x_t\), the observation is a noisy function value \(y_t=f(x_t)+\varepsilon_t\), and the learner controls where it samples [1107.1744]. In stochastic contextual linear bandits, the action depends on a context or even on a distribution over contexts, while in bandit linear control the control \(u_t\) affects both the next state and the scalar cost feedback [2107.09912] [1906.02685] [2007.00759]. Across these formulations, the common structure is that the learner does not passively receive data: it actively shapes the data-generating process through sequential decisions under stochastic uncertainty.

## 1. Canonical formulation and scope

A controlled stochastic bandit problem typically specifies an action space, a stochastic feedback model conditional on the chosen action, and a benchmark against which performance is measured. In the standard stochastic \(K\)-armed bandit, there are \(K\) arms, arm \(i\) has unknown mean reward \(\mu_i\), and the learner chooses \(i(t)\in\{1,\dots,K\}\) at each round and observes a reward, often modeled as Bernoulli in the KL-UCB analysis [1903.07839]. In the empirical-moment formulation, each arm has reward distribution \(F_i\) supported on \([0,1]\), rewards are independent across time conditional on the chosen arm, and regret is measured as \(\sum_{i:\mu_i<\mu^*}(\mu^*-\mu_i)T_i(n)\) [1105.2879].

The same controlled structure persists in richer models. In stochastic convex bandit optimization, the learner sequentially chooses \(x_1,\dots,x_T\in X\) over a convex, compact set \(X\subset\mathbb{R}^d\), observes only noisy function values, and minimizes
\[
R_T=\sum_{t=1}^T \bigl(f(x_t)-f(x^*)\bigr),
\qquad
x^*\in\arg\min_{x\in X} f(x),
\]
with \(f\) assumed convex and \(1\)-Lipschitz [1107.1744]. In stochastic contextual linear bandits, each round begins with a context \(s\) drawn i.i.d. from an unknown distribution \(\mu\), the learner sees a context-dependent action set \(\mathcal A_s\), and rewards follow
\[
r(s,a)=\phi(s,a)^\top\theta^\star+\eta
\]
with mean-zero, \(1\)-subGaussian noise [2107.09912].

A useful unifying description is that the learner’s action determines the conditional law of the observation. In stochastic convex bandits this is explicit in
\[
y_t=f(x_t)+\varepsilon_t,
\]
so the learner controls where it samples and the sampling changes the information obtained [1107.1744]. In contextual models with distributional contexts, the learner observes only a context distribution \(\mu_t\), not the realization \(c_t\), chooses \(x_t\), and then observes
\[
y_t=f(x_t,c_t)+\epsilon_t,\qquad c_t\sim \mu_t,
\]
which again makes the chosen action the mechanism through which uncertainty is probed [1906.02685].

The scope of the setting is correspondingly broad. It includes semi-bandit models with multiple simultaneous plays and budget depletion [1711.05928], continuous-time bandits with controlled restarts [2007.00081], conservative and safety-constrained bandits [1602.04282] [1908.05814] [2203.15629], graph-feedback contextual bandits with side observations [2308.15107], and stochastic bandit formulations for discrete concave optimization and linear dynamical control [2405.12439] [2007.00759]. This suggests that “controlled stochastic bandit” is best understood as a structural viewpoint rather than a single formal model.

## 2. Feedback models and observability

The defining informational feature of the setting is partial observability: the learner generally does not observe full reward functions, gradients, latent contexts, or all arm outcomes. What is observed depends on the chosen action and on the specific feedback model.

In zeroth-order stochastic convex optimization, feedback is noisy function evaluation. The learner may query any \(x\in X\), but only sees a noisy value
\[
y=f(x)+\varepsilon,
\]
where \(\varepsilon\) is independent, mean-zero, and \(\sigma\)-subgaussian, so the same point may be queried multiple times to build confidence intervals [1107.1744]. In standard stochastic multi-armed bandits, the learner observes only the reward of the selected arm, with bounded or Bernoulli rewards depending on the formulation [1903.07839] [1105.2879].

Contextual variants refine the same principle. In the distributional-context model, the learner observes \(\mu_t\in\mathcal P(\mathcal C)\), not the realized context \(c_t\), and chooses based on expected features
\[
\bar\psi_{x,\mu_t}=\mathbb E_{c\sim\mu_t}[\phi_{x,c}\mid \mathcal F_{t-1},\mu_t].
\]
The effective noise then contains both observation noise and context-realization uncertainty, with
\[
\xi_t:=(\phi_{x_t,c_t}-\psi_t)^\top\theta
\]
acting as an additional mean-zero term in the linear case [1906.02685]. A conservative extension keeps the same informational asymmetry while also requiring cumulative performance to remain within a \((1-\alpha)\) fraction of a baseline policy at every time [2203.15629].

Other feedback structures explicitly exploit side observations. In stochastic graph bandits, the learner observes the context \(x_t\) and a directed graph \(G_t\), chooses one action \(a_t\), and then observes the rewards of all actions in its out-neighborhood
\[
\{(x_t,a,y_{t,a})\}_{a\in \mathcal N^{out}_{a_t}(G_t)}.
\]
The observation model is therefore neither pure bandit nor full information; it is shaped by the graph revealed at the round [2308.15107]. In budget-constrained multiple-play semi-bandits, the learner plays exactly \(K\) arms and observes individual rewards and costs for each selected arm, but not for unplayed arms [1711.05928].

A particularly distinctive feedback pattern appears in continuous-time bandits with controlled restarts. The learner selects both an arm \(I_n\) and a restart time \(\nu_n\). If the run is interrupted at cutoff \(t\), the realized duration and reward become right-censored:
\[
U_{k,n}(t)=\min\{X_{k,n},t\}+C_k(t)\,\mathbb I\{X_{k,n}>t\},
\qquad
V_{k,n}(t)=R_{k,n}\,\mathbb I\{X_{k,n}\le t\}.
\]
The paper emphasizes that this feedback is right-censored in the sense that a larger cutoff reveals at least as much information as a smaller cutoff [2007.00081].

These models support a broad taxonomy of observability regimes.

| Model | Controlled decision | Observed feedback |
|---|---|---|
| Stochastic convex bandit | Query point \(x_t\) | Noisy value \(f(x_t)+\varepsilon_t\) |
| \(K\)-armed stochastic bandit | Arm \(i(t)\) | Reward of chosen arm |
| Distributional contextual bandit | Action given \(\mu_t\) | Reward with hidden realized context |
| Graph contextual bandit | Action \(a_t\) on graph \(G_t\) | Rewards on out-neighborhood |
| Budgeted semi-bandit | Subset \(a_t\), \(|a_t|=K\) | Individual rewards and costs of played arms |
| Controlled restarts | Pair \((I_n,\nu_n)\) | Right-censored duration and reward |

A common misconception is that the benchmark should always be a policy that sees the finest latent state available in the environment. The distributional-context model shows otherwise: the relevant comparator is the best action given the observed distribution \(\mu_t\), not the realized hidden context \(c_t\), and the paper gives an example showing that competing against a policy that sees \(c_t\) before acting can incur \(\Omega(T)\) regret [1906.02685].

## 3. Objectives and regret criteria

The controlled stochastic bandit setting does not have a single universal performance functional. Instead, the objective is determined by what the learner is meant to optimize under the available control and feedback constraints.

The most common criterion is cumulative regret or pseudo-regret. In stochastic convex bandit optimization this takes the form
\[
R_T=\sum_{t=1}^T \bigl(f(x_t)-f(x^*)\bigr),
\]
which is a cumulative optimization loss relative to the best fixed point in the feasible set [1107.1744]. In stochastic \(K\)-armed bandits the pseudo-regret is
\[
\mathrm{Regret}(T)=\sum_{t=1}^T (\mu^*-\mu_{i(t)}),
\]
or equivalently the arm-count form \(\sum_{i:\mu_i<\mu^*}(\mu^*-\mu_i)T_i(n)\) [1903.07839] [1105.2879]. In contextual linear bandits with distributional contexts, the benchmark is the best action chosen from the observed distribution:
\[
\mathcal R_T=\sum_{t=1}^T \bigl(f(x_t^*,c_t)-f(x_t,c_t)\bigr),
\qquad
x_t^*=\arg\max_{x\in\mathcal X}\mathbb E_{c\sim\mu_t}[f(x,c)].
\]
This definition makes the benchmark information-compatible with the learner [1906.02685].

Other formulations depart substantially from reward maximization. In controlled experiment design for stochastic contextual linear bandits, the goal is to collect a dataset under a fixed exploration policy \(\pi_e\) so that the recovered greedy policy \(\hat\pi\) has small expected suboptimality
\[
\mathbb E_{s\sim\mu}\Big[\max_{a\in\mathcal A_s}\phi(s,a)^\top\theta^\star-\phi(s,\hat\pi(s))^\top\theta^\star\Big].
\]
This is not immediate online regret minimization; it is policy-recovery quality after deliberate data collection [2107.09912].

Pure-exploration and offline-optimization variants use simple regret. For stochastic bandits over \(M^\natural\)-concave functions, after \(T\) noisy value-oracle queries the learner outputs \(x^{T+1}\), and the expected simple regret is
\[
\mathrm{sReg}_T=f^*(x^*)-\mathbb E[f^*(x^{T+1})].
\]
The same work also considers cumulative pseudo-regret
\[
\mathrm{Reg}_T=T\cdot f^*(x^*)-\mathbb E\!\left[\sum_{t=1}^T f^*(x^t)\right],
\]
showing that controlled stochastic bandit models can simultaneously support exploration-only and reward-accumulation viewpoints [2405.12439].

Several papers replace standard regret with constrained or risk-adjusted criteria. In conservative bandits, the learner must maintain
\[
\sum_{s=1}^t \mu_{I_s}\ge (1-\alpha)\mu_0 t \qquad \forall t\le n,
\]
or the analogous realized-reward condition, while minimizing pseudo-regret [1602.04282]. In linear stochastic bandits under safety constraints, the learner minimizes
\[
R_T=\sum_{t=1}^T \mu^\dagger x_t-\mu^\dagger x^*,
\qquad
x^*\in\arg\min_{x\in\mathcal D(\mu)}\mu^\dagger x,
\]
subject to the unknown-parameter safety constraint \(\mu^\dagger Bx_t\le c\) at every round with high probability [1908.05814]. In Sharpe-ratio optimization, the object of interest is not cumulative reward but the empirical Sharpe ratio of the action sequence, with stabilized arm index
\[
\xi_i=\frac{\mu_i}{L_0+\rho \sigma_i^2}
\]
and regret
\[
\mathbb E[\mathcal R_n(\pi)]=n\left(\xi_1-\mathbb E[\xi_n(\pi)]\right).
\]
This makes mean and variance jointly relevant to exploration [2508.13749].

Time-constrained models induce yet another criterion. In continuous-time bandits with controlled restarts, the natural quantity is the renewal reward rate
\[
r_k(t)=\frac{\mathbb E[V_{k,1}(t)]}{\mathbb E[U_{k,1}(t)]},
\]
and regret is measured relative to the optimal expected cumulative reward achievable before a time budget \(\tau\) [2007.00081]. A plausible implication is that the controlled stochastic bandit setting is unified less by a single loss function than by a shared informational architecture: sequential control under stochastic, action-dependent feedback.

## 4. Algorithmic principles

Algorithm design in controlled stochastic bandits is driven by the interaction between estimation, confidence, and the geometry or structure of the action space. The classical paradigm is optimism under uncertainty. In KL-UCB and KL-UCB\((\alpha)\), the index of arm \(i\) is
\[
U_i(t)=\sup\left\{\mu\in[0,1]: d(\hat\mu_i(t),\mu)\le \frac{\log\!\left(t/(N_i(t))^\alpha\right)}{N_i(t)}\right\},
\]
with \(\alpha=0\) giving KL-UCB and \(\alpha=1\) giving KL-UCB+ [1903.07839]. Conservative UCB modifies this by adding a budget-feasibility check before allowing the optimistic arm to be played, reverting to the default arm when the lower confidence bound on the budget would become negative [1602.04282]. Safe-LUCB similarly restricts optimism to actions certified safe under every parameter in a confidence ellipsoid [1908.05814].

Confidence-driven elimination and repeated sampling appear in nonparametric and geometric models. In stochastic convex optimization with bandit feedback, the one-dimensional algorithm queries interval quartiles and a center point, constructs confidence intervals of width \(\gamma_i\), and discards a quartile only when the evidence is strong enough. The center point acts as a sentinel that detects when the function dips in the middle, which the paper identifies as the key innovation over simpler interval-shrinking schemes [1107.1744]. The high-dimensional version generalizes the ellipsoid method through a regular simplex, a sequence of pyramids, cone-cutting, and hat-raising, using repeated sampling to ensure confidence intervals valid with probability at least \(1-1/T\) [1107.1744].

Moment-based and distribution-aware indices provide another principle. DMED-M replaces the full empirical distribution in DMED by the first \(d\) empirical moments and schedules arm \(i\) again when
\[
T_i(n)\,{}_d({}_d\hat F_i(n),\hat\mu^*(n)) \le \log n - \log T_i(n).
\]
The resulting policy depends only on the first \(d\) empirical moments of each arm and realizes a computational-complexity versus regret tradeoff [1105.2879]. A different form of estimator adaptation appears in multi-armed bandits with limited control variates, where the learner combines a reward-only estimator and a control-variate estimator through
\[
\hat\mu_{S_i(t),i}=\lambda_{t,i}\hat\mu_{N_i(t),i}^{(nc)}+(1-\lambda_{t,i})\hat\mu_{M_i(t),i}^{(c)}
\]
and then builds a \(t\)-based UCB index from the combined mean and variance estimate [2603.02100].

Structured exploration may also be deliberately non-reactive. In design of experiments for stochastic contextual linear bandits, the planner uses offline contexts and a reward-free LinUCB-style uncertainty criterion
\[
\pi_m(s)=\arg\max_{a\in\mathcal A_s}\|\phi(s,a)\|_{\Sigma_{\underline m}^{-1}}
\]
to produce a mixture of policies, and the online sampler then keeps this exploration rule fixed while collecting rewards on fresh contexts [2107.09912]. The paper explicitly contrasts this with standard reactive algorithms such as UCB and LinUCB.

In graph-feedback contextual bandits, the exploration policy is shaped simultaneously by candidate-optimal actions, graph structure, and empirical gaps. AdaCB.G constructs an exploration set from an induced subgraph over actions that could still be optimal, then solves a linear program to assign action probabilities while ensuring enough side observations through the graph [2308.15107]. In stochastic \(M^\natural\)-concave maximization, the key principle is to reduce each greedy update to a small stochastic bandit problem over feasible one-step increments and then compose local decisions via a robustness theorem:
\[
f(x_K)\ge f(x^*)-\sum_{k=1}^K \mathrm{err}(i_k\,|\,x_{k-1}).
\]
This converts local estimation accuracy into global optimization quality [2405.12439].

Control can also be embedded in dynamics rather than static action choice. In bandit linear control, the disturbance-action policy
\[
u_t(M)=-K_0x_t+\sum_{i=1}^H M^{(i)}w_{t-i}
\]
reparameterizes control so that the induced surrogate losses have bounded memory, after which a new reduction from bandit convex optimization with memory to standard bandit convex optimization becomes possible [2007.00759]. This suggests that controlled stochastic bandit algorithms often hinge on an intermediate representation that makes action-dependent information accumulation tractable.

## 5. Constraints, structure, and specialized regimes

A major line of work studies controlled stochastic bandits under explicit safety, budget, or feasibility constraints. Conservative bandits introduce a default arm and require cumulative reward never to fall below a fixed fraction of the default strategy uniformly over time; the budget process is
\[
\widetilde Z_t=\sum_{s=1}^t \mu_{I_s}-(1-\alpha)\mu_0 t,
\]
and safety means \(\widetilde Z_t\ge 0\) for all \(t\) [1602.04282]. Linear stochastic bandits under safety constraints define the true safe set as
\[
\mathcal D(\mu)=\{x\in D_0:\mu^\dagger Bx\le c\},
\]
replace it with an inner approximation
\[
\mathcal D_t^s=\{x\in D_0: v^\dagger Bx\le c,\ \forall v\in C_t\},
\]
and choose only actions certified safe under the current confidence ellipsoid [1908.05814]. A contextual analogue requires cumulative reward to remain above a \((1-\alpha)\) fraction of a baseline policy even though the realized context is hidden and only the context distribution is observed [2203.15629].

Resource constraints lead to further specializations. In budget-constrained multiple-play semi-bandits, the learner must choose exactly \(K\) out of \(N\) arms per round, pay the realized costs of all selected arms, and stop when the chosen set would exceed the remaining budget. The stochastic oracle benchmark repeatedly plays the fixed best subset maximizing the sum of bang-per-buck ratios \(\mu_r^i/\mu_c^i\) [1711.05928]. In continuous-time bandits with controlled restarts, restart times become decision variables, and the optimal static action is characterized by the reward-rate maximizer \((k^*,t^*)\) [2007.00081].

Another line exploits latent structure in the reward model. Distributional-context bandits replace exact pre-decision context information by distributions over contexts [1906.02685]. High-dimensional contextual bandits with missing covariates assume a sparse parameter \(\beta^*\), masked observations
\[
Z_{t,i}=X_{t,i}\odot U_{t,i},
\]
and missing completely at random covariates with coordinate-wise observation probabilities \(\zeta_j\), leading to a missingness-adjusted lasso plug-in policy [2207.11165]. Graph-feedback bandits exploit side observations encoded by a directed graph revealed at each round [2308.15107].

The setting also accommodates structured non-stationarity. Generalized non-stationary bandits assume time-varying arm means \(\mu_k(t)=f_k(t/T)\), controlled short-term variation of the best mean, and a geometric condition on the gap process. The same framework covers switching bandits, locally polynomial means, locally smooth means, and bounded-inflexion gap models, and the algorithm PrudentBandits performs confidence-based gap estimation, active-set maintenance, and significant change-point detection on gaps rather than raw means [2102.00725].

Risk-sensitive and economic-control variants extend the notion of “control” beyond arm choice alone. Sharpe-ratio bandits optimize a stabilized ratio depending jointly on mean and variance [2508.13749]. Continuous-time two-armed restless bandits with imperfect information model a risky arm that raises future expected returns through human capital \(H\), a safe arm with deterministic payoff, and a hidden type \(X\); the optimal policy is a stopping rule characterized by an index that formally coincides with Gittins’ index [1506.07291]. A plausible implication is that the controlled stochastic bandit setting reaches into stochastic control whenever present choices alter not only information but also future opportunity sets.

## 6. Guarantees, lower bounds, and conceptual significance

Regret guarantees in controlled stochastic bandits reflect both classical exploration–exploitation limits and additional costs induced by geometry, constraints, or richer information structures. In stochastic convex optimization with bandit feedback, the generalized ellipsoid algorithm achieves \(\tilde O(\mathrm{poly}(d)\sqrt T)\) regret with high probability, and the paper states that any algorithm must incur at least \(\Omega(\sqrt T)\) regret, so the dependence on \(T\) is optimal up to logarithmic factors [1107.1744]. In the Bernoulli stochastic bandit, KL-UCB\((\alpha)\) satisfies
\[
\limsup_{T\to\infty}\frac{\mathbb E[\mathrm{Regret}(T)]}{\log T}
\le
\sum_{i\neq i^*}\frac{\mu^*-\mu_i}{d(\mu_i,\mu^*)},
\]
matching the Lai–Robbins lower bound and establishing asymptotic optimality of KL-UCB+ [1903.07839].

For contextual and structured models, rates depend on dimension or on the structure that mediates observation quality. The design-of-experiments approach to stochastic contextual linear bandits achieves near-minimax online sample complexity
\[
N\approx \widetilde O\!\left(\frac{d\beta}{\epsilon^2}\right)
\]
for recovery of a near-optimal policy from non-reactive exploration data [2107.09912]. Distributional-context linear bandits obtain order-optimal high-probability regret \(\tilde O(d\sqrt T)\), matching the classical linear contextual rate up to logarithmic factors [1906.02685]. Graph-feedback contextual bandits obtain a minimax-style bound driven by the expected independence number \(\mathbb E_G[\alpha(G)]\) rather than the full action set size, and also admit a gap-dependent upper bound under a uniform gap condition [2308.15107]. High-dimensional sparse contextual bandits with missing covariates incur regret
\[
\mathcal O\!\left(\frac{1}{\zeta_{\min}^2}\sqrt{s_0T\log(dT)}\right)
\]
with high probability, so missingness worsens regret by at most the factor \(1/\zeta_{\min}^2\) [2207.11165].

Safety and conservativeness add explicit prices. Conservative UCB achieves the usual stochastic-bandit \(\sqrt{nK}\)-type term plus an additive penalty of order \(K/(\alpha\mu_0)\) up to logarithmic factors, and the paper proves a lower bound showing that any algorithm satisfying the conservative constraint must pay at least \(\Omega(K/(\alpha\mu_0))\) additional regret in the worst case [1602.04282]. In linear stochastic bandits under safety constraints, if the safety gap \(\Delta=c-\mu^\dagger Bx^*\) is positive and known, Safe-LUCB yields \(O(\sqrt T\log T)\) regret up to problem-dependent constants; when \(\Delta=0\) or unknown, the worst-case bound degrades to \(O(T^{2/3}\log T)\) [1908.05814]. The conservative contextual extension decomposes regret into a standard linear-UCB term plus two time-independent constants, one for conservativeness and one for unknown contexts [2203.15629].

Stochastic structure can also separate sharply from adversarial structure. The \(M^\natural\)-concave maximization work gives \(O(T^{-1/2})\) simple regret and \(O(T^{2/3})\) cumulative regret under an unbiased noisy value oracle, but also proves that in the adversarial full-information setting no polynomial-time per-round algorithm can achieve \(O(T^{1-c})\) regret for any constant \(c>0\) unless \(\mathsf P=\mathsf{NP}\) [2405.12439]. In bandit linear control, strongly convex and smooth costs allow \(\tilde O(\sqrt T)\) pseudo-regret under bandit feedback, while full-information online control can achieve much faster rates, including logarithmic regret in some regimes [2007.00759]. This clarifies that partial feedback is not merely a technical nuisance; it changes the attainable rate.

Several specialized models retain the canonical logarithmic or square-root patterns after suitable reformulation. Continuous-time bandits with controlled restarts achieve \(O(\log \tau)\) regret for finite restart sets and \(O(\sqrt{\tau\log\tau})\) regret for continuous restart times [2007.00081]. Corralling stochastic bandit algorithms yields regret close to that of the best base learner, with gap-dependent terms involving the gap between the best base learner and the others [2006.09255]. Sharpe-ratio Thompson sampling achieves logarithmic regret with a matching lower bound in the Gaussian model, establishing order-optimality for this risk-sensitive objective [2508.13749]. Generalized non-stationary bandits obtain
\[
\mathbb E R_T \le C\log(T)\sqrt{KTM}+CTB^*,
\]
recovering the classical switching-bandit rate up to logarithmic factors when \(B^*=0\) [2102.00725].

Taken together, these results show that the controlled stochastic bandit setting is not defined by a single reward model or a single algorithmic template. Its central property is that decisions simultaneously affect immediate performance and the information available for future decisions. Convexity, linear structure, graph side-observations, censoring, safety constraints, control variates, and restart choices all alter how this dual role is expressed, but the core research problem remains the same: to exploit stochastic regularity without losing control of the information process itself.

Source: https://www.emergentmind.com/topics/controlled-stochastic-bandit-setting