Bandits with Knapsacks (BwK)

• Bandits with Knapsacks (BwK) is a framework for online learning where each action provides a reward while consuming one or more limited resources under knapsack constraints.
• The approach leverages LP relaxations and primal-dual methods to balance resource consumption with reward collection, outperforming fixed-arm strategies.
• Advanced variants extend BwK to combinatorial, contextual, adversarial, and quantum settings, each addressing unique challenges in dynamic resource allocation.

Searching arXiv for recent and canonical Bandits with Knapsacks papers to ground the article. Bandits with Knapsacks (BwK) is a stochastic or adversarial online learning framework in which each action yields both reward and consumption of one or more limited resources, and interaction stops when some budget is exhausted (Badanidiyuru et al., 2013). Relative to classical multi-armed bandits, BwK replaces the sole time constraint with a collection of knapsack constraints and thereby changes both the benchmark and the difficulty of learning: the optimal policy for a given latent distribution may significantly outperform the policy that plays the optimal fixed arm, so regret analysis must account for budget allocation across time and resources rather than only arm identification (Badanidiyuru et al., 2013). Subsequent work has developed LP-based, primal-dual, combinatorial, contextual, non-stationary, adversarial, and quantum formulations of BwK, and has clarified when no-regret learning is possible, when only competitive-ratio guarantees are attainable, and how structural properties of the resource-consumption process determine learnability (Immorlica et al., 2018, Braverman et al., 19 Mar 2025, Su et al., 6 Jul 2025).

1. Foundational formulation and benchmark structure

The canonical BwK model has a finite set of arms, a finite horizon, and one or more resource budgets. In each round, the learner picks an arm, observes a reward and a vector of consumptions, and the process stops when a budget is violated; total reward is the sum collected before stopping (Badanidiyuru et al., 2013). In the stochastic formulation introduced by Badanidiyuru, Kleinberg, and Slivkins, each arm has a fixed latent distribution over reward-consumption outcome vectors, outcomes are drawn independently over time conditional on the chosen arm, and time itself can be encoded as an additional resource so that all constraints take a common knapsack form (Badanidiyuru et al., 2013). A later stochastic formulation by Li, Sun, and Ye writes the model with $m$ arms, $d$ resources, horizon $T$, common budget $B$, and i.i.d. reward and resource vectors with unknown means $\boldsymbol{\mu}$ and $\boldsymbol{C}$, together with a stopping time $\tau$ defined by budget violation or horizon exhaustion (Li et al., 2021).

A central structural fact is that the relevant benchmark is not the best arm. In standard stochastic MAB, the optimal dynamic policy is always to pull a fixed best arm, whereas in BwK the optimal dynamic policy can exploit mixtures of arms that deplete different resources in balanced proportions; consequently, the policy that plays the optimal fixed arm can be arbitrarily worse than the optimal dynamic policy (Badanidiyuru et al., 2013). This observation motivates LP relaxations as the dominant benchmarking device. The primal LP allocates fractional pull counts $\xi_x$ or $\boldsymbol{x}$ to arms subject to expected resource constraints, while the dual LP assigns shadow prices to resources (Badanidiyuru et al., 2013, Li et al., 2021). In the stochastic setting, the LP value upper-bounds the expected reward of the optimal dynamic policy (Badanidiyuru et al., 2013, Kumar et al., 2022). In Li, Sun, and Ye’s formulation, the offline LP is

$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$

and its dual interprets $d$0 as resource shadow prices (Li et al., 2021).

This LP perspective leads to a precise decomposition of regret. Li, Sun, and Ye define reduced costs

$d$1

which vanish on optimal arms and are positive on sub-optimal arms, and prove a generic upper bound

$d$2

The first term is arm-side regret from pulling sub-optimal arms, while the second is knapsack-side regret from leaving valuable resources unused (Li et al., 2021). This decomposition formalizes a recurring theme across the BwK literature: learning quality depends not only on identifying reward-efficient arms but also on discovering and exhausting the binding constraints.

2. Primal-dual methods and stochastic regret guarantees

Primal-dual algorithms are a canonical methodological core of BwK. The original 2013 framework introduced two algorithms—BalancedExploration and a multiplicative-update primal-dual method—and proved regret bounds of order

$d$3

optimal up to polylogarithmic factors (Badanidiyuru et al., 2013). BalancedExploration reasons directly over distributions that are LP-perfect for plausible latent structures and explores mixtures of arms subject to confidence constraints, whereas the primal-dual algorithm maintains shadow prices for resources and selects arms by optimistic bang-per-buck ratios (Badanidiyuru et al., 2013).

The primal-dual viewpoint was sharpened by Li, Sun, and Ye through an explicit symmetry between arms and knapsacks (Li et al., 2021). They distinguish optimal versus sub-optimal arms and binding versus non-binding constraints via auxiliary LPs $d$4 and $d$5, and define the joint gap

$d$6

They further introduce structural parameters $d$7 and $d$8, measuring LP geometry and the smallest positive optimal allocation (Li et al., 2021).

Their algorithm is explicitly two-phase. Phase I uses UCB/LCB LPs to identify $d$9 and $T$0, with stopping certified by the counting identity $T$1 (Li et al., 2021). Phase II solves an adaptive LP on the remaining budget vector $T$2, restricted to the estimated optimal arms, so that binding resources are exhausted in a balanced manner (Li et al., 2021). The resulting regret bound is logarithmic in $T$3: $T$4 which the authors state is the first problem-dependent logarithmic regret bound for the general BwK problem (Li et al., 2021). This establishes a gap-dependent analogue of classical stochastic-bandit logarithmic regret, but now controlled by LP basis structure rather than only reward gaps.

Earlier logarithmic BwK results already existed in restricted settings. Flajolet and Jaillet developed “UCB-Simplex,” an LP-basis algorithm with logarithmic distribution-dependent bounds in several cases, including one-resource stochastic consumptions, multiple deterministic resources, and horizon-plus-budget settings (Flajolet et al., 2015). There, regret scales like $T$5 or $T$6 divided by generalized basis gaps, again reflecting that the right primitive objects are LP bases rather than arms (Flajolet et al., 2015). A plausible implication is that the later symmetry-based formulation can be read as a systematic reorganization of this earlier basis-centric viewpoint into arm-side and constraint-side hardness parameters.

A different stochastic generalization allows non-monotonic resource utilization. In that model, budgets evolve via drifts $T$7, may increase or decrease, and the process always runs for $T$8 rounds, with a null arm forced whenever some budget falls below 1 (Kumar et al., 2022). The LP benchmark becomes

$T$9

With known distributions, the ControlBudget policy achieves constant regret against this LP relaxation, and with unknown distributions, ExploreThenControlBudget achieves logarithmic regret (Kumar et al., 2022). This suggests that the LP-and-dual methodology of stochastic BwK extends beyond pure consumption models to renewable-budget regimes, provided the null-arm and positive-drift assumptions restore sufficient controllability.

3. Variants: combinatorial, contextual, and structured action spaces

BwK has been repeatedly generalized by enlarging the action space while preserving an LP backbone. Combinatorial Semi-Bandits with Knapsacks (SemiBwK) unifies stochastic BwK and combinatorial semi-bandits by letting actions be feasible subsets of a ground set of atoms, with additive rewards and consumptions over atoms and semi-bandit feedback on chosen atoms (Sankararaman et al., 2017). The action set is represented through the polytope $B$0, and for matroid constraints the algorithm solves an optimistic LP over $B$1 using per-atom UCB/LCB estimates, then applies a negatively correlated randomized rounding scheme to obtain a feasible combinatorial action (Sankararaman et al., 2017). The main regret bound is

$B$2

which recovers the BwK lower bounds in the singleton-action special case and the $B$3 rate of combinatorial semi-bandits when budgets are large enough not to bind (Sankararaman et al., 2017).

A related but distinct combinatorial model is Budgeted Combinatorial MAB reduced to BwK with a single pull (Das et al., 2022). There, a round may pull a super-arm $B$4, with additive reward $B$5 and total cost $B$6, under a global budget and semi-bandit feedback (Das et al., 2022). The reduction refines each round into $B$7 primitive BwK plays and introduces $B$8 artificial per-arm resources plus one real budget resource, yielding a standard BwK LP with $B$9 resources (Das et al., 2022). This makes PrimalDualBwK applicable and produces the algorithm CBwK-LP-UCB, with regret against the LP optimum controlled by a theorem the paper states explicitly (Das et al., 2022). The conceptual contribution is the use of artificial resources to encode combinatorial feasibility within BwK.

Linear Contextual Bandits with Knapsacks generalize both classical linear contextual bandits and standard BwK (Agrawal et al., 2015). In each round, the learner observes arm-specific contexts $\boldsymbol{\mu}$0, while expected reward and consumption are linear in the context: $\boldsymbol{\mu}$1 Standard BwK is recovered by setting $\boldsymbol{\mu}$2 and $\boldsymbol{\mu}$3 for all $\boldsymbol{\mu}$4 (Agrawal et al., 2015). The algorithm combines OFUL-style confidence ellipsoids for $\boldsymbol{\mu}$5 and $\boldsymbol{\mu}$6 with an OMD-based dual update over resource prices, choosing the arm that maximizes optimistic adjusted reward $\boldsymbol{\mu}$7 (Agrawal et al., 2015). The main theorem yields regret

$\boldsymbol{\mu}$8

under the budget condition $\boldsymbol{\mu}$9 (Agrawal et al., 2015). Relative to oracle-based unstructured contextual BwK, the linear model replaces dependence on $\boldsymbol{C}$0 or $\boldsymbol{C}$1 by dependence on the feature dimension $\boldsymbol{C}$2 (Agrawal et al., 2015).

Across these variants, the common pattern is that BwK serves as a generic resource-constrained online optimization template. Once a structured action model admits an LP or convex relaxation with resource-consumption coordinates, primal-dual selection, UCB-style confidence sets, and resource-price updates typically remain available, albeit with new rounding, exploration, or estimation subroutines.

4. Adversarial, non-stationary, and benchmark design

The adversarial version of BwK changes the problem qualitatively. Immorlica, Sankararaman, Schapire, and Slivkins showed that regret minimization is no longer feasible, and instead established an $\boldsymbol{C}$3 competitive ratio relative to the best fixed distribution over actions, together with a matching lower bound (Immorlica et al., 2018). Their analysis is built on a repeated Lagrangian zero-sum game whose minimax value matches the LP value in the stochastic setting and remains useful as a control device in adversarial sequences (Immorlica et al., 2018). The same paper also provides a simpler stochastic analysis via regret minimization in repeated games, offering a unifying primal-dual game-theoretic perspective (Immorlica et al., 2018).

A different adversarial line studies regret rather than competitive ratio but under a fixed-arm benchmark. “Unifying the stochastic and the adversarial Bandits with Knapsack” formulates adversarial BwK with rewards $\boldsymbol{C}$4, costs $\boldsymbol{C}$5, stopping time determined by budget exhaustion, and efficiency $\boldsymbol{C}$6 (Rangi et al., 2018). It introduces EXP3.BwK and EXP3++.BwK, described as order-optimal in the adversarial BwK setup and near-optimal in the stochastic setup up to an additional $\boldsymbol{C}$7 factor (Rangi et al., 2018). The paper also shows that when costs are large and comparable to the total budget, achievable regret bounds can be significantly worse than in the bounded-cost regime (Rangi et al., 2018). This suggests that cost scale relative to budget is itself a structural difficulty parameter in adversarial resource-constrained learning.

Recent work has shown that the benchmark itself is the central issue in adversarial BwK. “A New Benchmark for Online Learning with Budget-Balancing Constraints” studies adversarial BwK with bandit feedback, one resource, and hard stopping, and proves that the classical benchmark—best fixed distribution respecting the budget in expectation—admits no-regret learning because of the “spend-or-save” dilemma (Braverman et al., 19 Mar 2025). The paper introduces an Earth Mover’s Distance benchmark $\boldsymbol{C}$8 based on closeness of a strategy’s spending pattern to a sub-pacing sequence $\boldsymbol{C}$9 with the same total spend: $\tau$0 For any strategy class $\tau$1, the benchmark is

$\tau$2

They show that sublinear regret is achievable whenever the benchmark spending pattern is within EMD $\tau$3 of some sub-pacing pattern (Braverman et al., 19 Mar 2025).

The corresponding algorithm, LagrangianEMD, combines EXP4-IX over a strategy class $\tau$4 with OGD on a scalar dual variable $\tau$5, using adjusted payoffs $\tau$6 (Braverman et al., 19 Mar 2025). The main theorem yields

$\tau$7

under the bounded reward-to-cost ratio $\tau$8 (Braverman et al., 19 Mar 2025). For the “pacing over windows” benchmark, where each window of length $\tau$9 has its own fixed distribution subject to a window pacing constraint, the abstract states a regret bound

$\xi_x$0

and the paper proves a matching lower bound (Braverman et al., 19 Mar 2025). It further shows that EMD scaling $\xi_x$1 is essentially necessary: if two benchmark spending patterns are at EMD $\xi_x$2 apart, some adversarial environment forces $\xi_x$3 regret against at least one of them (Braverman et al., 19 Mar 2025).

This adversarial-benchmark development clarifies a widespread misconception: impossibility in adversarial BwK is not merely a failure of existing algorithms, but a failure of the classical global-budget comparator. A plausible implication is that the right notion of learnability in adversarial BwK is not absolute no-regret versus all budget-feasible policies, but no-regret versus structurally paced comparators.

Non-stationary BwK occupies an intermediate regime between stochastic and adversarial environments. In this model, $\xi_x$4 independently across time but with time-varying distributions $\xi_x$5, and budgets again scale as $\xi_x$6 (Liu et al., 2022). The paper shows that local variation budgets

$\xi_x$7

are insufficient by themselves because constraints couple decisions globally across time (Liu et al., 2022). It introduces global non-stationarity measures

$\xi_x$8

and shows that the dynamic LP benchmark differs from the per-period LP benchmark by at most $\xi_x$9, where $\boldsymbol{x}$0 bounds dual prices (Liu et al., 2022). A sliding-window UCB algorithm for per-step LPs then achieves regret

$\boldsymbol{x}$1

with matching lower bounds for all four terms up to logarithmic factors and the linear dependence on $\boldsymbol{x}$2 in the $\boldsymbol{x}$3 term (Liu et al., 2022). This identifies a second boundary of learnability: constrained non-stationary problems require both local and global measures of temporal variation.

5. Competitive-ratio refinements and vector-cost reductions

A parallel line of work approaches BwK through online learning with vector costs. In OLVC$\boldsymbol{x}$4, each action yields a $\boldsymbol{x}$5-dimensional cost vector and the objective is to minimize the $\boldsymbol{x}$6-norm of the cumulative cost (Kesselheim et al., 2020). The key reduction uses a smooth approximation

$\boldsymbol{x}$7

and surrogate scalar costs

$\boldsymbol{x}$8

This converts $\boldsymbol{x}$9-dimensional online learning into a 1-dimensional surrogate problem handled by Hedge or Exp3.P (Kesselheim et al., 2020). The paper proves that stochastic OLVC$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$0 admits $$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$1, whereas adversarial OLVC$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$2 admits $$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$3 (Kesselheim et al., 2020).

The same surrogate enters adversarial and stochastic BwK$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$4, where the Lagrangian reward becomes reward minus $$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$5 times the surrogate cost (Kesselheim et al., 2020). For adversarial BwK$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$6, the main result gives

$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$7

when $$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$8 is known, and

$$\max_{\boldsymbol{x}\ge 0}\ \boldsymbol{\mu}^\top \boldsymbol{x} \quad\text{s.t.}\quad \boldsymbol{C}\boldsymbol{x}\le \boldsymbol{B},$$9

otherwise (Kesselheim et al., 2020). In the classical $d$00 case this yields an $d$01 competitive ratio, improving over the prior $d$02 bound of Immorlica et al. (Kesselheim et al., 2020). The paper also establishes matching lower bounds for the $d$03 dependence, showing that logarithmic rather than linear dependence on $d$04 is optimal in adversarial BwK$d$05 (Kesselheim et al., 2020).

This vector-cost reduction is significant for BwK because it isolates the dimensional difficulty of multi-resource control from the bandit component. Rather than treating each resource as a separate adversarial constraint directly, the reduction compresses all resources into a scalar surrogate driven by the gradient of a smooth potential. A plausible implication is that similar reductions may be useful wherever BwK-style budget constraints appear inside larger online optimization systems with many coupled resources.

6. Emerging directions and current frontier

Several recent directions expand BwK beyond its classical algorithmic envelope. One is quantum computation. “Quantum Algorithms for Bandits with Knapsacks with Improved Regret and Time Complexities” initiates quantum BwK with quantum reward and resource oracles, as well as quantum LP access through QRAM (Su et al., 6 Jul 2025). In the problem-independent regime, the quantum algorithm improves the classical regret bound by a factor of

$d$06

yielding

$d$07

in the paper’s stated form (Su et al., 6 Jul 2025). In the problem-dependent regime, a quantum algorithm using an inexact quantum LP solver achieves leading regret

$d$08

improving the classical $d$09 dependence to $d$10 and the leading $d$11 factor to $d$12 (Su et al., 6 Jul 2025). The same work also reports polynomial speedups in LP-solver time complexity relative to classical counterparts (Su et al., 6 Jul 2025). This suggests that some hardness in BwK is computational rather than information-theoretic, particularly in gap-dependent LP-identification phases.

Another active theme is the search for “best-of-both-worlds” guarantees across stochastic and adversarial settings. The repeated-game perspective of Immorlica et al. already unified algorithmic primitives between stochastic and adversarial BwK (Immorlica et al., 2018), while EXP3++.BwK was explicitly designed to be order-optimal adversarially and almost optimal stochastically up to a $d$13 factor (Rangi et al., 2018). More recent adversarial benchmark work indicates that such unification must be benchmark-sensitive: stochastic regret against LP relaxations and adversarial competitiveness against fixed-distribution or paced comparators can coexist, but only because the comparators themselves differ (Braverman et al., 19 Mar 2025).

A recurring limitation across the literature is dependence on structural assumptions. Logarithmic stochastic regret requires nondegenerate LP solutions, positive basis gaps, and favorable geometric parameters such as $d$14 and $d$15 (Flajolet et al., 2015, Li et al., 2021). Constant or logarithmic regret in non-monotonic BwK requires a positive-drift null arm and separation conditions on drifts (Kumar et al., 2022). Sublinear regret in adversarial BwK requires paced benchmark spending patterns at EMD $d$16 from sub-pacing (Braverman et al., 19 Mar 2025). These assumptions are not merely technical conveniences: lower bounds in the corresponding papers show that when they fail, $d$17, linear, or competitive-ratio-only guarantees reappear.

Taken together, these results position BwK as a general theory of online learning under dynamic budget balance. The classical stochastic regime is now well understood through LP relaxations and primal-dual algorithms; structured action spaces can often be absorbed into the same framework; adversarial learnability depends critically on benchmark design; and current frontier work studies refined comparators, renewable resources, and computational acceleration (Badanidiyuru et al., 2013, Immorlica et al., 2018, Li et al., 2021, Kumar et al., 2022, Braverman et al., 19 Mar 2025, Su et al., 6 Jul 2025).