AdvBandit: Extensions in Adversarial Bandit Models

• AdvBandit is a family of adversarial bandit models that extend classical frameworks by incorporating non-i.i.d. losses, delayed feedback, and adaptive comparator designs.
• It introduces diverse formulations such as Bandits with Knapsacks, composite anonymous delayed feedback, and stateful adversarial bandits to tackle constrained and dynamic environments.
• AdvBandit algorithms employ motifs like primal-dual optimization, structured exploration, and hyperparameter tuning to enhance performance under complex adversarial conditions.

“AdvBandit” (Editor’s term) can be read as an umbrella for adversarial-bandit and closely related bandit formulations in which sequential action selection is carried out under partial feedback, but the environment, comparator, or control objective departs from the classical i.i.d. stochastic setting. In the cited literature, this umbrella includes oblivious adversarial losses, global budget constraints, composite anonymous delayed feedback, latent deterministic state evolution, multi-player collision models, adaptive hyperparameter selection for speculative decoding, and test-time routing among red-teaming experts (Ma et al., 2023, Immorlica et al., 2018, Wan et al., 2022, Khosravi et al., 2023, Hou et al., 21 May 2025, Ziakas et al., 8 Oct 2025). A common thread is that “arm” no longer denotes only a static reward source: it may instead denote a sensor, a spectrum sub-band, a meta-arm, a decoding configuration, or an attack-style expert, and performance may be evaluated by regret, policy regret, competitive ratio, stopping time regret, attack cost, or detection delay.

1. Formal scope and representative formulations

At its core, the adversarial multi-armed bandit framework specifies a finite arm set $A=\{1,\dots,K\}$, a horizon $T$, and a sequence of losses or rewards chosen without stochastic stationarity. In the standard loss convention, at each round $t\in[T]$ the environment specifies a loss function $L_t:A\to[0,1]$, the learner chooses an arm $a_t$, and observes only $L_t(a_t)$ (Ma et al., 2023). This partial-information template is then specialized in several directions.

A resource-constrained specialization is Bandits with Knapsacks, where pulling an arm yields reward and simultaneously consumes one or more resources; the process stops when a budget is exhausted (Immorlica et al., 2018). A multi-player specialization models a radar network as several independent nodes selecting frequency sub-bands, with collisions forcing reward $0$ and the environment pre-selecting reward sequences for each sub-band (Howard et al., 2021). A delayed-feedback specialization uses composite anonymous delayed feedback, in which losses are split into $d$ components and the learner observes only the aggregate of delayed components arriving at each round (Wan et al., 2022). A stateful specialization is Bandits with Deterministically Evolving States, where the latent state obeys

$q_{t+1} = (1-\lambda)q_t + \lambda b_{I_t},$

and the expected reward at time $t$ is $T$0 (Khosravi et al., 2023).

This suggests that “AdvBandit” is best understood not as a single algorithm but as a modeling family in which adversarial or non-classical feedback structures are primary, and in which the learner must couple exploration with control of constraints, state, or downstream utility. A further interpolation between stochastic and adversarial extremes appears in Approximately Stationary BwK, where $T$1-stationarity bounds the multiplicative range of expected rewards and consumptions across time for each arm (Fikioris et al., 2023).

2. Comparator design and objective functions

A defining feature of AdvBandit formulations is that the classical regret comparator—best fixed arm in hindsight—is often no longer sufficient. In the standard adversarial setting, regret is

$T$2

which remains meaningful when actions do not alter the future environment (Ma et al., 2023). Once feedback or rewards become history dependent, stronger or different benchmarks are required.

In composite anonymous delayed feedback with non-oblivious losses, the paper distinguishes pseudo-regret from policy regret and proves that pseudo-regret can be $T$3 even for a $T$4-bounded memory loss sequence; the appropriate comparator is instead the loss incurred by always playing the same action, yielding policy regret (Wan et al., 2022). In Bandits with Deterministically Evolving States, the benchmark is the best-fixed sequence of arms pulled, not the best-fixed action in hindsight, because current actions shape future rewards through the latent state (Khosravi et al., 2023). In fidelity bandits, where rewards include loyalty-points or subscription bonuses, single-arm strategies are not always optimal, so weak regret, strong regret, and mean regret are introduced to reflect the structure of the minimal sufficient comparator sets (Lugosi et al., 2021).

Other AdvBandit objectives depart even further from cumulative reward. In BanditSpec, the primary quantity is stopping time regret: $T$5 where $T$6 is the number of speculative decoding calls required until EOS (Hou et al., 21 May 2025). In Bandit Quickest Changepoint Detection, the objective is expected detection delay under a false alarm constraint, rather than reward maximization (Gopalan et al., 2021). In adversarial BwK, regret minimization is no longer feasible, so the relevant guarantee is competitive ratio against the best fixed distribution over actions (Immorlica et al., 2018). In reward-poisoning attacks on adversarial bandits, the attacker’s goal is neither regret nor reward, but to force the learner to play a target arm in $T$7 rounds while keeping cumulative attack cost $T$8 sublinear (Ma et al., 2023).

These benchmark shifts are not cosmetic. They encode which aspects of history are endogenous, whether actions alter future feasibility, and whether latency, safety failure, or coverage is the real control objective.

3. Recurrent algorithmic motifs

A first recurrent motif is primal–dual optimization through repeated games. In adversarial BwK, the main algorithm instantiates a primal adversarial bandit learner over arms and a dual full-information learner over resources, coupled through a Lagrangian payoff. This repeated-game view yields a single framework that recovers near-optimal stochastic BwK regret and an $T$9-competitive adversarial BwK guarantee (Immorlica et al., 2018).

A second motif is structured exploration under interaction constraints. In the radar setting, Coordinate & Play partitions time into blocks and sub-blocks; a coordinator chooses a meta-arm, and collisions are used as an implicit communication channel so that each radar node learns its assigned sub-band without direct messaging (Howard et al., 2021). In Bandit Quickest Changepoint Detection, $t\in[T]$0-GCD separates exploration rounds, used to estimate the post-change parameter, from exploitation rounds, used to query the most informative action and grow a GLR/CUSUM-style statistic (Gopalan et al., 2021).

A third motif is wrapping a base bandit algorithm to handle harder feedback models. In bounded-memory adversarial bandits with composite anonymous delayed feedback, a mini-batch wrapper converts any no-delay bandit algorithm into one that obtains sublinear policy regret under delayed anonymous aggregation (Wan et al., 2022). In speculative decoding, the wrapper interpretation is literal: BanditSpec treats any speculative decoding routine as a black-box subroutine and places a bandit policy on top of the hyperparameter configuration set $t\in[T]$1 (Hou et al., 21 May 2025).

A fourth motif is higher-level adaptation of algorithmic hyperparameters. Meta-learning across adversarial bandit tasks tunes the initialization, step-size, and entropy parameter of the Tsallis-entropy generalization of Exp3 in multi-armed bandits, and tunes the initialization, step-size, and boundary-offset of online mirror descent with self-concordant barrier regularizers in bandit linear optimization (Balcan et al., 2022). This pushes AdvBandit from single-task online control toward task-averaged adaptation.

A fifth motif is expert routing under bandit feedback. Red-Bandit trains one LoRA expert per attack style and then uses a multi-armed bandit policy at inference to choose among those experts based on the safety of the target model’s response, using either $t\in[T]$2-greedy or UCB (Ziakas et al., 8 Oct 2025). The arm abstraction is therefore compatible with modular policy selection, not only primitive actions.

4. Guarantees, impossibility results, and robustness boundaries

Several papers establish that the difficulty of AdvBandit is not merely algorithmic but information-theoretic. In adversarial bandits with reward poisoning, any no-regret adversarial bandit algorithm can be misled into selecting a suboptimal target arm in every but sublinear number of rounds while the attacker incurs only sublinear cumulative attack cost. In the easy case the attack cost is $t\in[T]$3, in the general case it is $t\in[T]$4, and any victim-agnostic attacker must pay at least $t\in[T]$5 on some instance (Ma et al., 2023). This shows that no-regret does not imply robustness to targeted feedback manipulation.

In composite anonymous delayed feedback, the pessimistic result is even sharper: pseudo-regret is $t\in[T]$6 for a $t\in[T]$7-armed problem with a $t\in[T]$8-bounded memory non-oblivious adversary, so pseudo-regret is fundamentally the wrong metric. Yet the mini-batch wrapper achieves $t\in[T]$9 policy regret for $L_t:A\to[0,1]$0-armed bandits, and the paper proves a matching $L_t:A\to[0,1]$1 lower bound when the loss sequence is oblivious but the delay is non-oblivious (Wan et al., 2022).

In the adversarial scaling model, where mean rewards factor as $L_t:A\to[0,1]$2, many standard algorithms are shown to fail under small means and cold-start attacks. By contrast, AAEAS and BROAD adapt to the effective number of informative samples: AAEAS obtains

$L_t:A\to[0,1]$3

while BROAD attains

$L_t:A\to[0,1]$4

with $L_t:A\to[0,1]$5 and $L_t:A\to[0,1]$6 the minimum nonzero gap (Lykouris et al., 2020).

Stopping-time formulations also admit sharp theory. Under stationary mean values for accepted speculative lengths, UCBSpec achieves stopping time regret

$L_t:A\to[0,1]$7

and an information-theoretic impossibility result implies that this rate is optimal up to universal constants for truncated geometric acceptance models (Hou et al., 21 May 2025). Under oblivious adversarial accepted-length sequences, EXP3Spec yields a stopping time regret bound of order

$L_t:A\to[0,1]$8

up to the paper’s more refined minimum form (Hou et al., 21 May 2025).

For quickest detection, the lower bound takes the classical $L_t:A\to[0,1]$9 form, with $a_t$0. The proposed $a_t$1-GCD algorithm matches this lower bound asymptotically at low false alarm rates, establishing optimality in that regime (Gopalan et al., 2021).

5. Application domains

The surveyed literature uses AdvBandit-style abstractions in several distinct applied domains. The mapping between domain object and arm varies, but the partial-feedback control logic is consistent.

Domain	Arm or action	Main objective
Database activity monitoring	sampled transactions or population units	population coverage and alert quality
Cognitive radar networks	frequency sub-bands or meta-arms	high SINR with low cumulative regret
Speculative decoding	decoding hyperparameter configurations	low stopping time and high throughput
LLM red-teaming	attack-style LoRA experts	maximize unsafe responses
Quickest changepoint detection	sensors or sensing actions	low detection delay
BwK applications	budget-consuming actions	reward before exhaustion

In database activity monitoring, the data sampling problem is recast as a special case of the multi-armed bandit problem. The proposed algorithm combines expert knowledge with random exploration, and the reported effect is improved diversity and population coverage without decreasing the quality of issuing alerts about events (Grushka-Cohen et al., 2019). In cognitive radar, the network is modeled as an adversarial multi-player bandit in which rewards are proportional to SINR and collisions force reward $a_t$2; adversarial multi-player bandit algorithms allow continued target tracking without a loss in tracking precision (Howard et al., 2021).

In sequence-to-sequence learning, bandit structured prediction uses only scalar task-loss feedback for sampled output structures, without gold-standard targets. The work lifts linear bandit learning to attention-based recurrent neural machine translation and adds control variates for variance reduction, reporting improvements of up to $a_t$3 BLEU points for domain adaptation from simulated bandit feedback (Kreutzer et al., 2017). In speculative decoding, BanditSpec applies UCBSpec and EXP3Spec to draft-model choice, tree-structure choice, or speculation-length choice, and experiments with LLaMA3 and Qwen2 show throughput close to the oracle best hyperparameter in simulated real-life LLM serving scenarios with diverse input prompts (Hou et al., 21 May 2025). In automated red-teaming, Red-Bandit dynamically routes among attack-style experts, achieving state-of-the-art results on AdvBench under sufficient exploration (ASR@10) while producing lower-perplexity prompts (Ziakas et al., 8 Oct 2025).

The BwK literature emphasizes another family of domains—dynamic pricing, repeated auctions, dynamic ad allocation, and network routing and scheduling—where arm pulls consume scarce resources and the policy must balance reward against budget depletion (Immorlica et al., 2018). Approximately Stationary BwK extends this picture to workloads that are neither exactly stochastic nor worst-case adversarial, and the guarantee improves smoothly as the instance becomes closer to stationary (Fikioris et al., 2023).

6. Limitations, misconceptions, and open directions

A recurring misconception is that standard no-regret guarantees automatically imply robustness. The reward-poisoning results show the opposite: an adversarial bandit algorithm can satisfy its regret guarantee on the attacked loss sequence and still be hijacked into selecting a target arm almost all the time (Ma et al., 2023). A second misconception is that classical regret remains the right metric under delayed or history-dependent feedback. In composite anonymous delayed feedback, pseudo-regret becomes linear even for bounded-memory losses, and only policy regret remains learnable (Wan et al., 2022).

Another boundary concerns adversary models. The radar formulation assumes an oblivious adversary, yet the paper explicitly studies an “intelligent emitter” that reacts to the radar’s previous actions; this violates the core assumption and degrades the meaning of the regret guarantees, even though adversarial multi-player bandit methods still outperform simpler baselines empirically (Howard et al., 2021). Similar caveats apply to speculative decoding and red-teaming: the finite arm set may be too restrictive when hyperparameter or attack spaces are structured, continuous, or prompt-dependent.

Several directions are explicitly identified as open. BanditSpec points to structured bandits, robust and non-stationary bandits, and contextual bandits for speculative decoding (Hou et al., 21 May 2025). Bandit Quickest Changepoint Detection highlights continuous parameter spaces, multiple changepoints, and controlled Markovian extensions (Gopalan et al., 2021). Approximately Stationary BwK leaves open a tighter characterization of the attainable competitive-ratio curve and extensions to contextual or combinatorial BwK (Fikioris et al., 2023). Meta-learning adversarial bandits suggests a broader shift toward learning not just actions but also geometry, initialization, and hyperparameters across tasks (Balcan et al., 2022).

Taken together, these works suggest that AdvBandit is best viewed as a research program rather than a single model: a family of partial-feedback control problems in which adversariality, structure, and objective design are inseparable. The central technical challenge is no longer merely exploration versus exploitation, but comparator selection under endogenous dynamics, corrupted or delayed feedback, resource exhaustion, and domain-specific control metrics.