Bandit Importance Sampling

• Bandit Importance Sampling is a family of adaptive methods that integrate importance sampling with bandit algorithms to reduce estimator variance using online feedback.
• These techniques are applied in stochastic and finite-sum optimization, graph neural networks, and black-box density estimation to improve convergence and sampling efficiency.
• Theoretical analyses establish variance bounds and regret minimization guarantees, demonstrating BIS's effectiveness in approximating oracle strategies under partial feedback.

Bandit Importance Sampling (BIS) denotes a class of methods that combine importance sampling with bandit-style sequential decision mechanisms in order to reduce estimator variance, improve sample efficiency, or adapt sampling distributions under partial feedback. Across the literature, the term covers several closely related formulations: adaptive non-uniform datapoint sampling for stochastic optimization (Salehi et al., 2017), online learning with bandit feedback for finite-sum optimization and sampling (Hanchi et al., 2021), adversarial-bandit formulations of neighbor sampling in graph neural networks (Liu et al., 2020), selective importance sampling for extreme bandit feedback (Lopez et al., 2020), contextual-bandit selection of subsets in expensive finite-sum optimization (Menickelly, 22 Apr 2026), and a sample-location design method for black-box densities explicitly named “bandit importance sampling” (Matsubara et al., 1 Sep 2025). This suggests that BIS is best understood as a methodological family rather than a single standardized estimator.

1. Conceptual definition and problem setting

In the BIS literature, the common objective is to improve on uniform or static sampling when the variance-relevant structure is unknown a priori, changes during training or optimization, or is only partially observable. In stochastic optimization, the motivating issue is that uniform sampling can yield a gradient estimator with large variance, while classical importance sampling may require access to all gradient norms or reliable upper bounds at each step (Salehi et al., 2017). In finite-sum optimization with decreasing step-sizes, the design of the sampling distribution is formulated as an online learning problem with bandit feedback, where only partial information about the cost is observed at each round (Hanchi et al., 2021). In GNN training, the optimal variance-reduced neighbor sampler depends on node embeddings and learned weights that change during training and are only partially observed when sampled, which makes static derivation of an optimal sampler non-trivial (Liu et al., 2020).

A central feature of BIS is that the importance mechanism is not fixed offline. Instead, it is learned or updated online by a bandit procedure, typically through exploration–exploitation tradeoffs. In datapoint sampling for stochastic optimization, each datapoint is treated as an arm and its selection probability is updated according to observed variance-related feedback (Salehi et al., 2017). In GNN neighbor sampling, each neighbor or edge becomes an arm, and the reward is tied to reduction in sampling variance (Liu et al., 2020). In expensive computational workflows, each component function or local model is an arm, while expert advice and side information can be incorporated through contextual bandits (Menickelly, 22 Apr 2026).

The phrase “importance sampling” itself is also used in more than one sense across these papers. In optimization and GNN work, it typically refers to unbiased reweighting of sampled gradients or messages by inverse sampling probabilities (Salehi et al., 2017, Liu et al., 2020). In “Learning from eXtreme Bandit Feedback,” it refers to off-policy evaluation and policy optimization from logged bandit data, where high variance motivates a selective importance sampling estimator (Lopez et al., 2020). In “Sampling as Bandits,” the method does not adapt a proposal distribution; instead it directly designs the sample sequence and assigns self-normalized importance weights afterward (Matsubara et al., 1 Sep 2025).

2. Canonical mathematical formulations

A recurring formulation in optimization expresses the variance-controllable part of an unbiased estimator as a function of the sampling distribution. For stochastic optimization with objective

$$F(w) = \frac{1}{n}\sum_{i=1}^{n}\phi_i(w) + \lambda r(w),$$

the gradient estimator under non-uniform sampling is

$$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$

and the effective variance is

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$

The choice of $p^t$ is therefore the controllable object, and BIS methods seek to adapt it online (Salehi et al., 2017).

The finite-sum online-learning formulation makes the same structure explicit. For functions of the form $f(x) = \sum_{i=1}^N f_i(x)$, the cost of a distribution $p$ at round $t$ is written as

$$c_t(p) = \sum_{i=1}^N \frac{1}{p_i}\|g_i^t\|_2^2 - \|g^t\|_2^2,$$

where $g_i^t = \nabla f_i(x_t)$ and $g^t = \nabla f(x_t)$. Static regret compares the cumulative cost of the learned sequence to the best fixed distribution in hindsight, whereas dynamic regret compares it to the instantaneous optimum at each round (Hanchi et al., 2021). The latter is stricter because the optimal distribution changes as $$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$0 evolves.

In GNN training, layer-wise sampling uses an unbiased estimator of aggregated neighborhood information,

$$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$1

and the effective variance term is

$$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$2

The optimal distribution is

$$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$3

but this is generally intractable because $$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$4 and $$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$5 are changing during training and only partially observed (Liu et al., 2020).

In batch learning from extreme bandit feedback, the classical importance sampling estimator is

$$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$6

but its variance becomes impractical in extremely large action spaces. The selective importance sampling estimator performs importance sampling only within a small action subset $$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$7:

$$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$8

This is presented as a Rao-Blackwellized estimator with a more favorable bias-variance regime (Lopez et al., 2020).

3. Bandit mechanisms and update rules

The optimization paper “Stochastic Optimization with Bandit Sampling” proposes MABS, an EXP3-inspired method in which each datapoint is an arm and the weights are updated according to sampled variance contributions. With $$\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},$$9 for SGD, the update is

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$0

followed by

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$1

The uniform mixture term ensures exploration (Salehi et al., 2017).

Avare, developed for finite-sum optimization and SGLD with decreasing step-sizes, replaces inaccessible true costs by surrogate costs based on the last observed per-component gradient norm,

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$2

and restricts the distribution to

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$3

The update is

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$4

The restriction prevents degenerate solutions in which some $$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$5, and the method exploits contraction of the iterates under decreasing step-size to make stale observations informative (Hanchi et al., 2021).

In GNN training, the bandit sampler uses rewards defined from the negative gradient of the effective variance. The reward per action is

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$6

The policy is then updated using adversarial multi-armed bandit algorithms such as EXP3 and EXP3.M, yielding methods denoted GNN-BS and GNN-BS.M (Liu et al., 2020). BLISS extends the same general idea as a Bandit Layer Importance Sampling Strategy for GNNs, maintaining edge weights $$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$7, assigning rewards

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$8

and updating weights with the EXP3 rule

$$\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.$$9

with a mixed exploration–exploitation probability

$p^t$0

BLISS is described as layer-wise and architecture-aware, with explicit handling of both fixed aggregation coefficients and attention-based settings (Alsaqa et al., 26 Dec 2025).

A distinct mechanism appears in the black-box density paper that explicitly introduces “bandit importance sampling.” There the bandit acts on a candidate pool of space-filling points rather than on a probability vector over datapoints. A Gaussian process surrogate models a transformed density, and the next point is chosen by maximizing the GP Upper Jensen Bound,

$p^t$1

For $p^t$2,

$p^t$3

After $p^t$4 sequential selections, self-normalized weights are assigned:

$p^t$5

This formulation differs from adaptive importance sampling because it does not adapt the proposal distribution; it directly designs the sample locations (Matsubara et al., 1 Sep 2025).

4. Theoretical properties

A defining theoretical theme in BIS is asymptotic approximation of an intractable variance-optimal strategy under partial feedback. For MABS in stochastic optimization, the main guarantee is

$p^t$6

where $p^t$7 is the best fixed distribution in hindsight. As $p^t$8, the average excess over three times the optimal effective variance vanishes (Salehi et al., 2017). The same factor-of-3 principle appears in the GNN adversarial-bandit formulation, which shows that the cumulative effective variance of the learned sampler is asymptotically within a factor of 3 of the oracle variance, plus a sublinear term (Liu et al., 2020).

Avare sharpens the comparison criterion by targeting dynamic rather than static regret. Under suitable conditions and step-sizes $p^t$9, it achieves $f(x) = \sum_{i=1}^N f_i(x)$0 dynamic regret for SGD and $f(x) = \sum_{i=1}^N f_i(x)$1 dynamic regret for SGLD (Hanchi et al., 2021). This is significant because the optimal sampling distribution is not stationary; it moves with the optimization trajectory. A plausible implication is that regret notions based only on comparison to a fixed sampler may be insufficient when the gradient landscape evolves materially across iterations.

The selective importance sampling estimator in extreme bandit feedback is analyzed through explicit bias and MSE bounds. Its bias satisfies

$f(x) = \sum_{i=1}^N f_i(x)$2

where

$f(x) = \sum_{i=1}^N f_i(x)$3

Its MSE obeys

$f(x) = \sum_{i=1}^N f_i(x)$4

If $f(x) = \sum_{i=1}^N f_i(x)$5 contains all relevant actions, the estimator is unbiased and has lower variance due to conditioning (Lopez et al., 2020).

For expensive finite-sum optimization via contextual bandits, the theoretical comparison is against the best expert in hindsight. Using Exp4, the regret grows as $f(x) = \sum_{i=1}^N f_i(x)$6 with the number of rounds $f(x) = \sum_{i=1}^N f_i(x)$7, and the paper states that expected regret relative to the best expert is bounded and decreases as $f(x) = \sum_{i=1}^N f_i(x)$8 (Menickelly, 22 Apr 2026). In the black-box density setting, the theory is not framed as regret minimization over a probability simplex, but as weak convergence of the weighted empirical measure. With a space-filling proposal sequence such as a scaled Halton sequence,

$f(x) = \sum_{i=1}^N f_i(x)$9

which is stated to be faster than vanilla Monte Carlo’s $p$0 rate (Matsubara et al., 1 Sep 2025).

5. Representative application domains

The published uses of BIS span several distinct application regimes.

Domain	Representative method	Key stated outcome
Stochastic optimization	MABS (Salehi et al., 2017)	Asymptotically approximates the optimal variance within a factor of 3
Finite-sum optimization and sampling	Avare (Hanchi et al., 2021)	$p$1 dynamic regret for SGD and $p$2 for SGLD
Graph neural networks	GNN-BS (Liu et al., 2020), BLISS (Alsaqa et al., 26 Dec 2025)	Lower sampling variance and strong empirical performance on multiple datasets
Extreme bandit feedback / XMC	sIS and POXM (Lopez et al., 2020)	Better bias-variance regime than standard IS in very large action spaces
Expensive finite-sum optimization	Exp4-based subset selection (Menickelly, 22 Apr 2026)	Robust mixing of expert advice, uniform exploration, and side information
Black-box density sampling	BIS with GP-UJB (Matsubara et al., 1 Sep 2025)	Accurate approximations with fewer target evaluations

In optimization, BIS has been paired with SGD, SVRG, SAGA, Adam, and SGLD. “Adam with Bandit Sampling” generalizes Adam by maintaining a distribution over training examples and updating it with an EXP3-style multi-armed bandit rule. The paper states that Adambs improves the convergence rate of Adam—$p$3 instead of $p$4 in some cases—and reports fast convergence in practice on MLP, CNN, LSTM, and RCNN workloads (Liu et al., 2020).

In GNNs, the need for BIS is especially acute because optimal samplers depend on embeddings and learned attention weights. The 2020 bandit sampler paper reports highest or near-highest test F1 scores across datasets and architectures and emphasizes robustness on attentive GNNs where existing variance-reduced samplers are less applicable (Liu et al., 2020). BLISS extends the bandit-layer importance sampling strategy to GraphSAGE and GATv2, reporting higher test F1-score than PLADIES on Citeseer, Pubmed, Cora, and Yelp, while performing nearly identically on Flickr and Reddit (Alsaqa et al., 26 Dec 2025).

In logged bandit learning with extremely large action spaces, POXM uses selective importance sampling with top-$p$5 actions under the logging policy. On EUR-Lex, Wiki10-31K, and Amazon-670K, the reported R@5 values in Table 2 are 44.48, 53.03, and 23.72 for POXM, compared with 13.51, 36.16, and 14.54 for BanditNet (Lopez et al., 2020). In expensive Bayesian computation, BIS for black-box densities reportedly required approximately 5% of the target evaluations of QMC-based IS to achieve the same approximation error on benchmark densities and produced accurate posterior approximations on the Lorenz weather model and the g-and-k model (Matsubara et al., 1 Sep 2025).

6. Distinctions, misconceptions, and related directions

One common misconception is to treat BIS as synonymous with ordinary adaptive importance sampling. The black-box density formulation explicitly distinguishes itself from adaptive importance sampling by stating that it does not optimize a proposal distribution and instead directly designs the samples through a sequential strategy that combines space-filling designs with multi-armed bandits (Matsubara et al., 1 Sep 2025). By contrast, MABS, Avare, Adambs, and the GNN methods do optimize or adapt sampling distributions over datapoints, components, or neighbors (Salehi et al., 2017, Hanchi et al., 2021, Liu et al., 2020, Liu et al., 2020).

A second misconception is that BIS is necessarily unbiased. The literature is more nuanced. In optimization and GNN sampling, unbiasedness is typically preserved through inverse-probability weighting of gradients or messages (Salehi et al., 2017, Liu et al., 2020). In extreme bandit feedback, sIS is deliberately allowed to be slightly biased when the selected subset $p$6 omits relevant actions, in exchange for potentially substantial variance reduction (Lopez et al., 2020). In black-box density sampling, the final estimator uses self-normalized importance weights rather than an i.i.d. proposal-sampling scheme (Matsubara et al., 1 Sep 2025).

A third misconception is that bandit methods are used only when no side information is available. The contextual-bandit formulation for expensive finite-sum optimization instead emphasizes the opposite case: useful side information may come from alternative lower-fidelity simulations, pre-trained emulators, or domain expertise from humans or AI models, and Exp4 is used to mix expert advice with uniform exploration (Menickelly, 22 Apr 2026). This suggests that BIS can function as a mechanism for integrating heterogeneous prior information rather than merely learning from scratch.

Several adjacent directions should also be distinguished from BIS proper. “Importance Sampling with Unequal Support” studies a new variant of importance sampling that can reduce variance by orders of magnitude when the supports of training and testing distributions differ, and provides theoretical analysis of bias and variance relative to ordinary importance sampling (Thomas et al., 2016). That work addresses a closely related variance-reduction problem, but the available abstract does not describe a bandit mechanism. Likewise, “Importance Sampling is All You Need” introduces a prompt-centric evaluation framework called BIS for estimating LLM benchmark performance by reweighting prompt distributions with IWAE-based density models and weight truncation, reporting an average absolute prediction error of 1.1% for code correctness scores and 2.15% for pass@1 (Shi et al., 2 Aug 2025). Despite the acronym, the method is not described in the provided material as a multi-armed bandit procedure.

Taken together, the arXiv literature portrays BIS as an umbrella for sequential importance-weighted sampling schemes that are adaptive, feedback-driven, and often explicitly exploration–exploitation balanced. The unifying idea is not a single formula, but a design principle: variance-relevant sampling decisions are treated as online decisions under uncertainty, and importance weighting is coupled to bandit or bandit-like adaptation to approach intractable oracle strategies with practical feedback budgets (Salehi et al., 2017, Hanchi et al., 2021, Liu et al., 2020, Matsubara et al., 1 Sep 2025).