Contextual Budget Bandit Overview

• Contextual Budget Bandits are sequential decision-making models that integrate observed context, uncertain rewards, and explicit budget constraints to optimize cumulative outcomes.
• They encompass various formulations—including stochastic, adversarial, episodic, and multi-resource models—using methods like optimism under uncertainty, primal–dual control, and regression oracles.
• Theoretical guarantees are derived from regret bounds and constraint violation measures, with applications spanning epidemic control, recommendation systems, and resource allocation.

Searching arXiv for recent and foundational papers on contextual budget bandits and related contextual bandits with knapsacks. In the literature surveyed here, “Contextual Budget Bandit” is used as an umbrella description for contextual bandit problems with explicit resource constraints, most commonly under the names contextual bandits with knapsacks (CBwK), budget-constrained contextual bandits, or contextual combinatorial bandits with budget. The common structure is sequential decision making with observed context, uncertain reward, and a budget or cost process that couples decisions across rounds; the objective is to maximize cumulative reward while respecting either per-round, cumulative, or average budget constraints. This family includes stochastic i.i.d. contexts, adversarial contexts, hard-stopping and continuing formulations, single- and multi-resource models, delayed-feedback variants, episodic replenishment, and stateful extensions built on restless bandits (Han et al., 2022).

1. Formal model and problem variants

A standard stochastic CBwK model has a context space $X$, a finite action set $A$ with $|A|=K$, $d$ resources, horizon $T$, and budgets $B\in\mathbb{R}_+^d$. At each round $t=1,\dots,T$, a context $x_t$ is observed, an action $a_t\in A$ is chosen, and the learner observes a reward $r_t\in[0,1]$ together with a consumption vector $A$0. Under realizability, there exist function classes $A$1 and $A$2, with $A$3 and $A$4, such that $A$5 and $A$6. A hard-stopping version defines a stopping time $A$7, so that $A$8 coordinatewise almost surely (Han et al., 2022).

Another formulation studies budget-constrained contextual bandits with adversarial contexts, where rewards and costs satisfy $A$9 and the learner continues operating even after the cumulative budget is exhausted. In this continuing setting, performance is measured by both regret and cumulative constraint violation (CCV), rather than by hard stopping. The paper defines $|A|=K$0 as the class of stationary randomized policies $|A|=K$1, and studies benchmarks ranging from round-wise feasibility in expectation to long-term feasibility with budget $|A|=K$2 (Sarkar et al., 7 May 2026).

A third major formulation is the classical constrained contextual bandit with discrete contexts $|A|=K$3, action set $|A|=K$4, horizon $|A|=K$5, and hard budget $|A|=K$6. In the unit-cost case, the oracle LP uses probabilities $|A|=K$7 of taking the best action in context $|A|=K$8, subject to $|A|=K$9, where $d$0. This leads to an Adaptive-Linear-Programming (ALP) rule that replaces the fixed average budget ratio $d$1 by the instantaneous ratio $d$2, where $d$3 is remaining budget and $d$4 remaining time (Wu et al., 2015).

The family also includes several specialized extensions. One line models vector-valued costs and continuous contexts, allowing signed costs $d$5 and hard componentwise constraints $d$6 (Chzhen et al., 2023). Another treats episodic contextual bandits with knapsacks in repeated episodes with replenished budgets and non-stationary context distributions inside an episode, but a shared latent conversion model across episodes (Li et al., 9 Jul 2025). A recent stateful extension defines a Contextual Budget Bandit over restless multi-armed bandits with a global context $d$7, context-specific transition kernels $d$8, and context-specific budgets $d$9, together with an average budget constraint $T$0 (Tang et al., 13 Sep 2025).

2. Reward, cost, and benchmark structure

The central modeling device is the simultaneous treatment of reward and cost as context-dependent random variables. In one widely used benchmark, the optimal static policy is the solution of a constrained optimization problem,

$T$1

with total oracle value $T$2. The regret is then $T$3, where $T$4 is the hard-stopping time ensuring budget feasibility (Han et al., 2022).

In the continuing adversarial-context formulation, the paper distinguishes regret from budget violation. For long-term budget $T$5, cumulative constraint violation is

$T$6

while a traditional positive-part violation measure is

$T$7

The objective is to control both regret and CCV, rather than only reward (Sarkar et al., 7 May 2026).

Several papers use a Lagrangian or LP relaxation to convert the constrained problem into a sequence of unconstrained or approximately unconstrained decisions. In the linear contextual knapsack setting, one introduces a scalar tradeoff parameter $T$8 satisfying

$T$9

and selects arms according to an adjusted optimistic reward

$B\in\mathbb{R}_+^d$0

where $B\in\mathbb{R}_+^d$1 is an online-learned dual price for resource consumption (Agrawal et al., 2015).

The same reward–cost coupling appears in conversion models. In “Contextual Bandits with Knapsacks for a Conversion Model” (Li et al., 2022), the realized reward and cost are

$B\in\mathbb{R}_+^d$2

with expectations $B\in\mathbb{R}_+^d$3 and $B\in\mathbb{R}_+^d$4. This couples reward and cost through a single Bernoulli conversion variable, rather than through unrelated mean functions.

A related but distinct benchmark arises in the finite-context constrained contextual bandit literature. The static LP upper bound is

$B\in\mathbb{R}_+^d$5

which has a threshold solution indexed by $B\in\mathbb{R}_+^d$6 with $B\in\mathbb{R}_+^d$7. This yields a single-round oracle value $B\in\mathbb{R}_+^d$8 and an upper bound $B\in\mathbb{R}_+^d$9 (Wu et al., 2015).

3. Algorithmic frameworks

Three algorithmic paradigms dominate the literature: optimism under uncertainty, primal–dual control, and regression-oracle reductions.

The optimism-based line includes linear UCB, GLM-UCB, and LP/UCB hybrids. In the linear contextual knapsack setting, the algorithm maintains confidence ellipsoids for reward parameters $t=1,\dots,T$0 and cost parameters $t=1,\dots,T$1, forms optimistic reward estimates $t=1,\dots,T$2 and optimistic lower-confidence cost estimates $t=1,\dots,T$3, and then chooses the arm maximizing the adjusted optimistic score above (Agrawal et al., 2015). In the generalized-linear decision-deferral model, optimistic reward and cost estimates are

$t=1,\dots,T$4

with action rule

$t=1,\dots,T$5

where $t=1,\dots,T$6 is a budget-control variable (Reid et al., 2024).

The primal–dual line is explicit in several papers. “Optimal Contextual Bandits with Knapsacks under Realizability via Regression Oracles” (Han et al., 2022) proposes SquareCBwK, combining reward and cost regression oracles with a dual vector $t=1,\dots,T$7. It defines the predicted Lagrangian score

$t=1,\dots,T$8

chooses a randomized action via an Abe–Long scheme, and updates $t=1,\dots,T$9 using online mirror descent with

$x_t$0

The same dual-control idea appears in the projected-gradient strategy for small total-cost constraints, where

$x_t$1

and actions are chosen by maximizing

$x_t$2

(Chzhen et al., 2023).

The regression-oracle line is especially prominent under realizability. In the adversarial-context continuing setting, the algorithm builds on the $x_t$3 framework of Foster et al. [2018] and uses online regression oracles for rewards and costs to construct a surrogate reward

$x_t$4

where the virtual queue evolves as $x_t$5. Actions are chosen by inverse-gap weighting on surrogate losses, turning the constrained problem into an unconstrained contextual bandit with adaptively defined rewards (Sarkar et al., 7 May 2026).

LP-based and oracle-efficient reductions remain important for large policy classes. “An efficient algorithm for contextual bandits with knapsacks, and an extension to concave objectives” (Agrawal et al., 2015) generalizes ILOVETOCONBANDITS to the knapsack setting, maintaining a smoothed distribution $x_t$6 over actions and solving an optimization problem that simultaneously controls empirical regret and inverse-propensity variance. “Linear Contextual Bandits with Knapsacks” (Agrawal et al., 2015) removes dependence on a policy-optimization oracle by exploiting linear structure directly.

Several papers adapt the same ideas to specialized regimes. The conversion-model CBwK algorithm learns logistic conversion probabilities and solves a per-round optimistic primal LP over static policies (Li et al., 2022). The episodic conversion-model algorithm uses a backward dynamic program with current UCB/LCB estimates of the latent conversion probability $x_t$7 (Li et al., 9 Jul 2025). In delayed-feedback settings, DORAL combines robust delay estimation with a delay-aware LinUCB index and an LP based on $x_t$8 (Liu et al., 2024).

4. Theoretical guarantees

Theoretical results vary sharply by context model, feedback model, and feasibility notion.

For stochastic CBwK under realizability and regression oracles, SquareCBwK achieves

$x_t$9

and, in the budget-linear regime $a_t\in A$0 with $a_t\in A$1,

$a_t\in A$2

The hard-stopping construction ensures $a_t\in A$3 almost surely (Han et al., 2022).

For adversarial contexts in the continuing setting, the Lyapunov-based reduction yields multiple regimes. Under round-wise feasibility in expectation with quadratic $a_t\in A$4, the paper states

$a_t\in A$5

Under almost-sure round-wise feasibility with exponential $a_t\in A$6, it obtains

$a_t\in A$7

and for long-term feasibility with nonnegative costs,

$a_t\in A$8

The same work also gives a lower bound showing that any multiplicative budget factor $a_t\in A$9 must satisfy $r_t\in[0,1]$0 in both hard-stopping and continuing settings (Sarkar et al., 7 May 2026).

For continuous contexts and signed costs, the adaptive projected-gradient method breaks the previously required $r_t\in[0,1]$1 budget scale. With $r_t\in[0,1]$2, the paper proves that, with probability at least $r_t\in[0,1]$3,

$r_t\in[0,1]$4

and

$r_t\in[0,1]$5

while explicitly allowing signed costs $r_t\in[0,1]$6 (Chzhen et al., 2023).

Classical finite-context constrained contextual bandits achieve sharper, regime-dependent rates. In the unit-cost setting with known context distribution, ALP has $r_t\in[0,1]$7 regret in non-boundary cases and $r_t\in[0,1]$8 regret in boundary cases. When rewards are unknown and learned by UCB-ALP, the same paper proves $r_t\in[0,1]$9 regret in non-boundary cases and $A$00 in boundary cases (Wu et al., 2015).

For linear contextual bandits with knapsacks, the main theorem states that if $A$01, then with probability at least $A$02,

$A$03

The paper emphasizes that linear contextual bandits already have an $A$04 lower bound, so this is near-optimal up to logarithmic factors (Agrawal et al., 2015).

A useful comparative summary is given below.

Regime	Representative guarantee	Source
Stochastic CBwK under realizability	$A$05	(Han et al., 2022)
Adversarial contexts, continuing	$A$06 or $A$07, depending on feasibility regime	(Sarkar et al., 7 May 2026)
Continuous contexts, signed costs	Hard feasibility $A$08 and $A$09	(Chzhen et al., 2023)

These guarantees also clarify a recurring misconception: there is no single canonical regret rate for contextual budget bandits. The correct rate depends on whether contexts are stochastic or adversarial, whether stopping is hard or continuing, whether costs are signed or nonnegative, and whether the analysis is benchmarked against a static relaxation, a feasible stationary policy, or a finite-horizon oracle.

5. Modeling extensions and application domains

The application literature uses the same budgeted contextual formalism to express markedly different operational constraints.

In epidemic control, the 2021 paper “Optimal Epidemic Control as a Contextual Combinatorial Bandit with Budget” formulates dynamic intervention design as a contextual combinatorial bandit problem that jointly optimizes a multi-criteria reward function balancing “the number of daily COVID-19 cases” and “the stringency of the recommended interventions.” The abstract states that, given historical daily cases and past intervention plans, the agent generates intervention plans in real time and that simulations demonstrate “a clear advantage in providing a pareto optimal solution in the epidemic intervention problem” (Lin et al., 2021).

In delayed-feedback resource allocation, DORAL studies a finite arm set with total budget $A$10, finite context space $A$11, and arm-dependent delay distributions. It first identifies “top responsive arms” via robust median-of-means delay estimates and then solves a delay-aware LP under an average budget constraint $A$12, where the context-level value uses $A$13 to discount rewards by the probability that feedback arrives within a cut-off window (Liu et al., 2024).

In episodic conversion models, the repeated-episode formulation has budgets $A$14 that reset at the start of each episode, a non-stationary context distribution inside each episode $A$15, and a shared latent conversion probability $A$16. The Bellman recursion is

$A$17

which embeds the resource consumption into the state variable $A$18 and yields regret guarantees sublinear in the number of episodes $A$19, independent of $A$20 (Li et al., 9 Jul 2025).

In recommendation systems and advertising, HATCH uses a hierarchical structure: a global resource allocator chooses a retention probability $A$21 for each user class $A$22, where $A$23, and a local LinUCB-style personalized recommender selects the arm. Its class-level LP is

$A$24

with a threshold solution analogous to ALP (Yang et al., 2020).

In machine learning decision deferral, the two-action model $A$25 formalizes “automate/model” versus “defer to human expert,” with $A$26 and $A$27. The optimal static policy $A$28 is defined by

$A$29

subject to

$A$30

so the budget becomes a cap on expert attention (Reid et al., 2024).

Finally, volunteer-based food rescue introduces a stateful and explicitly fairness-aware extension. Contexts correspond to regions, arms correspond to volunteers or communities, and the paper allows context-specific budgets $A$31 subject to $A$32. Its fairness index is defined as the minimum, over contexts, of the normalized reward share, and the paper states that higher budgets can be allocated to communities with lower match rates in order to alleviate geographical disparities (Tang et al., 13 Sep 2025).

6. Fairness, structure, and open technical directions

Fairness appears in two distinct ways in the surveyed literature. The first is as a small total-cost constraint represented directly in CBwK. In the continuous-context signed-cost paper, “equalized average costs between groups” is encoded through $A$33 cost components, where each group $A$34 contributes paired signed constraints of the form $A$35 and its negative. The budgets are then chosen as $A$36, allowing fairness tolerance $A$37 on the order of $A$38 up to polylogarithmic factors (Chzhen et al., 2023).

The second is as context-dependent budget allocation. In the food-rescue restless setting, fairness is not imposed only through additional constraints on outcomes; it is built into the budget vector $A$39 itself. The paper defines a fairness-aware LP by adding, for each context $A$40,

$A$41

and then optimizes over $A$42 using either Branch-and-Bound or the Mitosis algorithm (Tang et al., 13 Sep 2025).

A broader structural trend is the move from stateless CBwK to richer dynamics. The thresholding reformulation of online restless bandits converts a finite-horizon stateful control problem into a budgeted thresholding contextual bandit by defining

$A$43

and then intervening on agents whose action benefit exceeds a threshold $A$44. In a homogeneous two-state setting under stochastic dominance, the paper proves that greedily selecting the $A$45 agents with largest $A$46 is exactly optimal for the offline finite-horizon problem, and its online LGRT algorithm achieves constant-in-$A$47 regret under the “enough good arms” assumption (Xu et al., 7 Feb 2025). This suggests a technically important direction: some budgeted contextual problems become much easier once long-term dynamics are embedded into a learnable one-step proxy.

Across the literature, several open problems recur. Realizability dependence remains central: adversarial-context guarantees in the continuing setting rely on known function classes and squared-loss oracle bounds, and handling misspecification robustly is identified as open (Sarkar et al., 7 May 2026). Multiple resources are conceptually straightforward but can be computationally intractable in episodic dynamic programs because the budget state becomes high-dimensional (Li et al., 9 Jul 2025). Delayed and aggregate feedback complicate credit assignment, a point emphasized both in delayed-feedback allocation and in epidemic-control formulations (Liu et al., 2024). A plausible implication is that future work will continue to hybridize primal–dual pacing, confidence-based contextual learning, and state aggregation or dynamic programming, rather than relying on a single universal template.

Taken together, the literature presents Contextual Budget Bandit not as one fixed model but as a research program: contextual learning under explicit resource coupling, where the mathematically decisive objects are the reward model, the cost model, the comparator class, and the precise notion of feasibility.