Adversarial Bandits with Knapsacks

• Adversarial Bandits with Knapsacks are models for sequential decision-making that manage budgeted resource consumption when rewards and costs are chosen in an adversarial manner.
• The framework reveals a critical spend-or-save dilemma, where classical no-regret methods fail and competitive ratio benchmarks alongside dual pricing become essential.
• Recent advances redesign benchmarks using pacing and Earth Mover’s Distance, and address non-stationary, approximate, and replenishable environments to broaden practical applications.

Adversarial Bandits with Knapsacks (BwK) study sequential decision-making under resource constraints when rewards and resource consumptions are not generated by a fixed stochastic law but may vary adversarially over time. In the canonical formulation, a learner repeatedly chooses an arm, observes only the reward and consumptions of the chosen arm, and must stop or switch to a null action once a budget is exhausted. What makes the adversarial setting distinctive is that resource timing becomes part of the difficulty: unlike stochastic BwK, where sublinear regret against a natural LP benchmark is attainable, adversarial BwK exposes a “spend-or-save” tension that can make classical no-regret objectives unattainable, forcing the literature to reconsider the benchmark itself, the performance criterion, or the structure imposed on the environment (Braverman et al., 19 Mar 2025, Immorlica et al., 2018).

1. Stochastic origins and the resource-constrained bandit model

BwK generalizes stochastic multi-armed bandits by attaching a vector of resource consumptions to each arm pull and ending the process when some resource budget is exceeded. The foundational stochastic model has $m$ arms, $d$ resources, horizon $T$, and outcome vectors in $[0,1]^{d+1}$, with time often encoded as an additional resource. The central benchmark is not the best fixed arm but an LP relaxation, $\LPOPT$, that upper-bounds the optimal dynamic policy and already reflects the fact that mixtures of arms can strictly outperform any single arm under budgets (Badanidiyuru et al., 2013).

That distinction is fundamental for all later adversarial work. In BwK, exploration consumes scarce resources, and the optimal policy is governed by the joint geometry of rewards and consumptions rather than by reward means alone. The early stochastic literature therefore developed LP-based primal-dual methods and basis-sensitive analyses instead of arm-wise regret decompositions. This line later yielded logarithmic, problem-dependent regret bounds for stochastic BwK and emphasized the symmetry between arms and knapsacks: optimality depends both on which arms are basic variables and on which constraints are binding (Flajolet et al., 2015, Li et al., 2021).

From the adversarial perspective, the stochastic foundations matter for two reasons. First, they identify the LP benchmark and dual resource prices as the natural language of the problem. Second, they clarify why the adversarial setting is not merely “stochastic BwK with worse concentration”: the stochastic theory relies on a stationary latent structure, whereas adversarial BwK must cope with the possibility that the best spending schedule itself changes with time.

2. Classical adversarial formulation and the impossibility of additive no-regret

An early adversarial formulation already departed from the stochastic regret paradigm. In “Adversarial Bandits with Knapsacks” (Immorlica et al., 2018), outcomes are chosen by an oblivious adversary, regret minimization is declared infeasible, and the objective becomes minimizing competitive ratio relative to the best fixed distribution over actions; the paper gives an $O(\log T)$ competitive ratio and a matching lower bound.

The later impossibility results sharpened why additive no-regret fails. In the classical adversarial benchmark, the learner is compared with the best fixed distribution over actions that respects the budget in expectation. The obstacle is the “spend-or-save” dilemma: the learner must decide when to spend budget without knowing whether later rounds will be better or worse. A canonical construction uses two actions, horizon $T$, and budget $B=T/2$: one costly action gives reward $1/2$ in the first half of the horizon, while in the second half its reward is either always $1$ or always $d$0. Because the first half looks identical in both instances, any spending schedule chosen there is misaligned with the optimal benchmark in one of the two cases, yielding $d$1 regret for every algorithm against the classical benchmark (Braverman et al., 19 Mar 2025).

This impossibility is not a technical artifact of a particular proof method. It reflects a structural feature of adversarial BwK: total expenditure is not enough; temporal alignment of expenditure matters. In stochastic BwK, the best fixed distribution is natural because future opportunities are sampled from the same law. In adversarial BwK, the same benchmark becomes too strong because the adversary can place value in time-sensitive ways that turn budget timing into a hidden decision variable.

3. Competitive-ratio theory and the fully adversarial limit

Once additive no-regret against the classical benchmark is ruled out, the dominant objective becomes competitive ratio. The first-generation adversarial theory established $d$2 competitiveness relative to the best fixed distribution over actions (Immorlica et al., 2018). A subsequent line, built through online learning with vector costs, improved the dependence on the number of resources: for adversarial $d$3, there is an $d$4-competitive algorithm when $d$5 is known and an $d$6-competitive algorithm when $d$7 is unknown; for the classical $d$8 case this yields an $d$9-competitive algorithm, improving the earlier $T$0 dependence, and the bound is tight (Kesselheim et al., 2020).

A different but related perspective comes from “Approximately Stationary Bandits with Knapsacks” (Fikioris et al., 2023), which parameterizes the environment by multiplicative stationarity parameters $T$1 controlling how much expected rewards and consumptions may vary over time. This model interpolates between stochastic BwK $T$2 and fully adversarial BwK $T$3. In the fully adversarial case, the best achievable fraction of $T$4 is $T$5, where $T$6; more precisely, the paper states $T$7, matching known lower bounds up to constants. The same work gives algorithms whose guarantees transition smoothly between the stochastic and adversarial extremes as the environment becomes more stationary (Fikioris et al., 2023).

This competitive-ratio literature establishes an important negative lesson and a positive one. The negative lesson is that fully adversarial BwK generally cannot support additive regret against strong benchmarks. The positive lesson is that dual pricing, multiplicative updates, and constrained online learning still yield meaningful guarantees, but the right guarantee is often multiplicative rather than additive.

4. Intermediate regimes: non-stationarity, approximate stationarity, and replenishment

Between i.i.d. stochastic inputs and fully adversarial inputs lies a broad regime of structured change. “Non-stationary Bandits with Knapsacks” (Liu et al., 2022) studies exactly this regime and shows that the usual variation budget is insufficient in the presence of resource constraints. The paper introduces local variation measures $T$8 and $T$9 for rewards and consumptions, but also global non-stationarity measures $[0,1]^{d+1}$0 and $[0,1]^{d+1}$1. The reason is specifically knapsack-driven: even a single change point can induce linear regret against a dynamic benchmark because resource constraints couple early and late decisions. The resulting upper and lower bounds make this explicit, with regret terms of order $[0,1]^{d+1}$2, $[0,1]^{d+1}$3, and an unavoidable $[0,1]^{d+1}$4 term capturing global temporal mismatch (Liu et al., 2022).

Approximate stationarity yields a different interpolation. In that model, if an arm is ever good, its expected reward cannot later collapse by more than a multiplicative factor $[0,1]^{d+1}$5, and similarly for consumptions via $[0,1]^{d+1}$6. This lets the algorithm compete against scaled versions of the optimal fixed distribution with a performance factor that improves continuously from $[0,1]^{d+1}$7 toward $[0,1]^{d+1}$8 as the environment becomes more stationary (Fikioris et al., 2023). The conceptual point is that adversarial hardness is driven not only by unknown means, but by the ability of the environment to concentrate value in narrow temporal windows.

The literature also extends adversarial BwK beyond monotone resource depletion. “Bandits with Replenishable Knapsacks: the Best of both Worlds” allows consumptions in $[0,1]^{d+1}$9, so actions may replenish resources. In this model there is a void action that replenishes each resource by at least $\LPOPT$0, and a primal-dual template guarantees a constant competitive ratio $\LPOPT$1 in the adversarial setting when $\LPOPT$2 or when the per-round replenishment is a positive constant; in the stochastic regime the same framework yields $\LPOPT$3 regret (Bernasconi et al., 2023). This result is notable because it gives the first positive adversarial guarantees for replenishable BwK and shows that primal-dual resource pricing remains viable even when budgets need not be monotone.

5. Regaining additive regret through benchmark redesign: pacing, spending patterns, and EMD

A major recent development is the shift from changing the environment class to changing the benchmark. “A New Benchmark for Online Learning with Budget-Balancing Constraints” argues that the classical adversarial benchmark is too strong and replaces it with one defined through spending patterns, pacing, and Earth Mover’s Distance (EMD) (Braverman et al., 19 Mar 2025). The key object is the sequence of expected per-round expenditures. A comparator is considered admissible only if its spending pattern is close, in EMD, to some sub-pacing sequence that spends at most $\LPOPT$4 per round.

For one-dimensional time, the paper uses the unnormalized EMD

$\LPOPT$5

This quantity measures how much budget mass must be shifted over time to transform one spending trajectory into another. The main result states that sublinear regret is attainable against any strategy whose spending pattern is within EMD $\LPOPT$6 of a sub-pacing pattern. In other words, adversarial BwK becomes learnable in an additive-regret sense once the benchmark is prevented from front-loading or back-loading expenditure too aggressively (Braverman et al., 19 Mar 2025).

The paper then studies a concrete special case, “pacing over windows,” in which time is partitioned into disjoint windows of size $\LPOPT$7, and the benchmark may choose a different fixed distribution over actions in each window while respecting a per-window pacing budget. Against this benchmark, the algorithm $\LPOPT$8 achieves

$\LPOPT$9

regret, and the paper proves a matching lower bound, establishing optimality up to logarithmic factors (Braverman et al., 19 Mar 2025).

Equally important is the necessity evidence. The same work shows that the classical spend-or-save comparator in the impossibility example is at EMD distance $O(\log T)$0 from any sub-pacing pattern, and more generally that two spending patterns at EMD distance $O(\log T)$1 can force $O(\log T)$2 regret against at least one of them. This makes EMD not merely a convenient proof device but a candidate answer to the benchmark question itself: temporal misalignment of spend, rather than total spend alone, is the adversarial quantity that must be controlled (Braverman et al., 19 Mar 2025).

6. Algorithmic motifs, benchmark diversity, and applications

Across the literature, several algorithmic motifs recur. The first is primal-dual control: one maintains implicit or explicit prices for resources and chooses actions by balancing reward against price-weighted consumption. This architecture originates in stochastic BwK and remains central in adversarial work, whether via multiplicative updates on resource prices, online gradient descent on dual variables, or Lagrangian saddle-point views (Badanidiyuru et al., 2013, Bernasconi et al., 2023).

The second motif is that adversarial constraint handling increasingly relies on richer online-learning primitives than ordinary no-regret. “No-Regret is not enough! Bandits with General Constraints through Adaptive Regret Minimization” shows that standard primal and dual no-regret guarantees can still permit linear constraint violations in general long-term constrained bandits. The remedy is to require weakly adaptive primal and dual regret minimizers; their interaction induces a self-bounding property for dual variables and yields best-of-both-worlds guarantees, including the tight adversarial competitive ratio $O(\log T)$3 and sublinear constraint violations (Bernasconi et al., 2024). In a related direction, “Beyond Primal-Dual Methods in Bandits with Stochastic and Adversarial Constraints” replaces dual-game machinery with optimistic estimates of constraints and a regret minimizer over moving strategy sets; this gives a best-of-both-worlds algorithm with bounds logarithmic in the number of constraints and $O(\log T)$4 regret in stochastic settings without Slater’s condition (Bernasconi et al., 2024).

A third motif is reduction. Combinatorial or structured action spaces are often encoded as single-pull BwK instances with auxiliary resources. In “Budgeted Combinatorial Multi-Armed Bandits,” a super-arm is reduced to a single-pull BwK with one true budget resource plus one auxiliary resource per arm, and the resulting LP-dual interpretation is explicitly presented as a template that is useful for combinatorial adversarial BwK even though the paper itself is stochastic (Das et al., 2022).

A persistent source of confusion in the area is that not all “adversarial BwK” papers optimize the same objective. Some papers analyze competitive ratio against the best fixed distribution over actions (Immorlica et al., 2018, Kesselheim et al., 2020). Some interpolate between stochastic no-regret and adversarial competitive ratio through stationarity parameters (Fikioris et al., 2023). Some recover additive regret only after restricting the comparator through pacing and EMD (Braverman et al., 19 Mar 2025). Others adopt an efficiency-based benchmark against the best fixed arm in hindsight and obtain order-optimal budget-dependent regret in that alternative model (Rangi et al., 2018). These are not contradictory results; they are answers to different benchmark choices.

The application base is correspondingly broad. The foundational stochastic papers position BwK as a model for electronic commerce, routing, and scheduling (Badanidiyuru et al., 2013). Later work targets dynamic pricing, bid optimization in online advertisement auctions, and dynamic procurement (Flajolet et al., 2015). The EMD benchmark is motivated by autobidding and budget pacing in online advertising, where smooth temporal expenditure is operationally meaningful (Braverman et al., 19 Mar 2025). Replenishable variants connect BwK to inventory management, bilateral trade, and other economic settings in which resources can both deplete and replenish over time (Bernasconi et al., 2023). A common thread is that resource timing is endogenous and strategically important; adversarial BwK is the mathematical regime in which that timing can no longer be ignored.

Adversarial BwK has therefore evolved into a benchmark-sensitive theory of constrained online decision-making. The field no longer revolves around a single question of “can one get no-regret?” but around a sharper taxonomy: which benchmark is meaningful, which temporal structure is realistic, and which adversarial freedoms can be tolerated while preserving learnability.