---
title: Bandit Importance Sampling
url: https://www.emergentmind.com/topics/bandit-importance-sampling-bis
type: topic
---

# Bandit Importance Sampling

Bandit Importance Sampling (BIS) denotes a class of methods that combine importance sampling with bandit-style sequential decision mechanisms in order to reduce estimator variance, improve sample efficiency, or adapt sampling distributions under partial feedback. Across the literature, the term covers several closely related formulations: adaptive non-uniform datapoint sampling for stochastic optimization [1708.02544], online learning with bandit feedback for finite-sum optimization and sampling [2103.12243], adversarial-bandit formulations of neighbor sampling in graph neural networks [2006.05806], selective importance sampling for extreme bandit feedback [2009.12947], contextual-bandit selection of subsets in expensive finite-sum optimization [2604.20657], and a sample-location design method for black-box densities explicitly named “bandit importance sampling” [2509.01437]. This suggests that BIS is best understood as a methodological family rather than a single standardized estimator.

## 1. Conceptual definition and problem setting

In the BIS literature, the common objective is to improve on uniform or static sampling when the variance-relevant structure is unknown a priori, changes during training or optimization, or is only partially observable. In stochastic optimization, the motivating issue is that uniform sampling can yield a gradient estimator with large variance, while classical importance sampling may require access to all gradient norms or reliable upper bounds at each step [1708.02544]. In finite-sum optimization with decreasing step-sizes, the design of the sampling distribution is formulated as an online learning problem with bandit feedback, where only partial information about the cost is observed at each round [2103.12243]. In GNN training, the optimal variance-reduced neighbor sampler depends on node embeddings and learned weights that change during training and are only partially observed when sampled, which makes static derivation of an optimal sampler non-trivial [2006.05806].

A central feature of BIS is that the importance mechanism is not fixed offline. Instead, it is learned or updated online by a bandit procedure, typically through exploration–exploitation tradeoffs. In datapoint sampling for stochastic optimization, each datapoint is treated as an arm and its selection probability is updated according to observed variance-related feedback [1708.02544]. In GNN neighbor sampling, each neighbor or edge becomes an arm, and the reward is tied to reduction in sampling variance [2006.05806]. In expensive computational workflows, each component function or local model is an arm, while expert advice and side information can be incorporated through contextual bandits [2604.20657].

The phrase “importance sampling” itself is also used in more than one sense across these papers. In optimization and GNN work, it typically refers to unbiased reweighting of sampled gradients or messages by inverse sampling probabilities [1708.02544], [2006.05806]. In “Learning from eXtreme Bandit Feedback,” it refers to off-policy evaluation and policy optimization from logged bandit data, where high variance motivates a selective importance sampling estimator [2009.12947]. In “Sampling as Bandits,” the method does not adapt a proposal distribution; instead it directly designs the sample sequence and assigns self-normalized importance weights afterward [2509.01437].

## 2. Canonical mathematical formulations

A recurring formulation in optimization expresses the variance-controllable part of an unbiased estimator as a function of the sampling distribution. For stochastic optimization with objective
$$
F(w) = \frac{1}{n}\sum_{i=1}^{n}\phi_i(w) + \lambda r(w),
$$
the gradient estimator under non-uniform sampling is
$$
\hat{g}(w^t) = \frac{\nabla \phi_{i_t}(w^t)}{n p^t_{i_t}},
$$
and the effective variance is
$$
\mathbb{V}_e^t(w^t, p^t) = \frac{1}{n^2} \sum_{i=1}^n \frac{\|\nabla \phi_i(w^t)\|^2}{p_i^t}.
$$
The choice of $p^t$ is therefore the controllable object, and BIS methods seek to adapt it online [1708.02544].

The finite-sum online-learning formulation makes the same structure explicit. For functions of the form $f(x) = \sum_{i=1}^N f_i(x)$, the cost of a distribution $p$ at round $t$ is written as
$$
c_t(p) = \sum_{i=1}^N \frac{1}{p_i}\|g_i^t\|_2^2 - \|g^t\|_2^2,
$$
where $g_i^t = \nabla f_i(x_t)$ and $g^t = \nabla f(x_t)$. Static regret compares the cumulative cost of the learned sequence to the best fixed distribution in hindsight, whereas dynamic regret compares it to the instantaneous optimum at each round [2103.12243]. The latter is stricter because the optimal distribution changes as $x_t$ evolves.

In GNN training, layer-wise sampling uses an unbiased estimator of aggregated neighborhood information,
$$
\hat{\mu}_i^{(l)} = \frac{1}{k}\sum_{s=1}^{k} \frac{\alpha_{ij_s}}{q_{ij_s}} h_{j_s}^{(l)},
$$
and the effective variance term is
$$
\mathbb{V}_e(q_i) = \sum_{j \in \mathcal{N}_i} \frac{\alpha_{ij}^2}{q_{ij}} \| h_j \|^2.
$$
The optimal distribution is
$$
q_{ij}^* = \frac{\alpha_{ij} \| h_j \|^2}{\sum_{k \in \mathcal{N}_i} \alpha_{ik} \| h_k \|^2},
$$
but this is generally intractable because $\alpha_{ij}$ and $\|h_j\|$ are changing during training and only partially observed [2006.05806].

In batch learning from extreme bandit feedback, the classical importance sampling estimator is
$$
\hat{V}_\text{IS}(\pi) = \frac{1}{n}\sum_{i=1}^n\frac{\pi(y_i \mid x_i)}{\rho(y_i \mid x_i)}r_i,
$$
but its variance becomes impractical in extremely large action spaces. The selective importance sampling estimator performs importance sampling only within a small action subset $\Phi(x)$:
$$
\hat{V}^\Phi_{\text{sIS}(\pi)} = \frac{1}{n} \sum_{i=1}^n \frac{\pi(y_i \mid x_i, y \in \Phi(x_i))}{\rho(y_i \mid x_i)} r_i, \quad y_i \in \Phi(x_i).
$$
This is presented as a Rao-Blackwellized estimator with a more favorable bias-variance regime [2009.12947].

## 3. Bandit mechanisms and update rules

The optimization paper “Stochastic Optimization with Bandit Sampling” proposes MABS, an EXP3-inspired method in which each datapoint is an arm and the weights are updated according to sampled variance contributions. With $a_i^t = \|\nabla \phi_i(w^t)\|^2 / n^2$ for SGD, the update is
$$
w_{i_t}^{t+1} = w_{i_t}^t \cdot \exp\left( \frac{\delta a_{i_t}^t}{(p_{i_t}^t)^3} \right),
$$
followed by
$$
p_i^{t+1} = (1 - \eta) \cdot \frac{w_i^{t+1}}{\sum_j w_j^{t+1}} + \frac{\eta}{n}.
$$
The uniform mixture term ensures exploration [1708.02544].

Avare, developed for finite-sum optimization and SGLD with decreasing step-sizes, replaces inaccessible true costs by surrogate costs based on the last observed per-component gradient norm,
$$
\tilde{c}_t(p) = \sum_{i=1}^N \frac{1}{p_i} (h_i^t)^2,
$$
and restricts the distribution to
$$
\Delta(\varepsilon_t) = \{p \in \mathbb{R}^N \mid p_i \ge \varepsilon_t,\, \sum_{i=1}^N p_i = 1\}.
$$
The update is
$$
p^t = \arg\min_{p \in \Delta(\varepsilon_t)} \tilde{c}_t(p).
$$
The restriction prevents degenerate solutions in which some $p_i^t \to 0$, and the method exploits contraction of the iterates under decreasing step-size to make stale observations informative [2103.12243].

In GNN training, the bandit sampler uses rewards defined from the negative gradient of the effective variance. The reward per action is
$$
r_{ij}(t) = \frac{\alpha_{ij}^{2}}{k\cdot q_{ij}(t)^2}\|h_j(t)\|^2.
$$
The policy is then updated using adversarial multi-armed bandit algorithms such as EXP3 and EXP3.M, yielding methods denoted GNN-BS and GNN-BS.M [2006.05806]. BLISS extends the same general idea as a Bandit Layer Importance Sampling Strategy for GNNs, maintaining edge weights $w_{ij}$, assigning rewards
$$
r_{ij} = \frac{ \alpha_{ij}^2 }{ k \cdot q_j^2 } \| h_j \|_2^2,
$$
and updating weights with the EXP3 rule
$$
w_{ij}^{(t+1)} = w_{ij}^{(t)} \exp\left( \frac{\delta \hat{r}_{ij}^{(t)}}{ |\mathcal{N}_i| } \right),
$$
with a mixed exploration–exploitation probability
$$
q_{ij}^{(t+1)} = (1-\eta) \frac{ w_{ij}^{(t+1)} }{ \sum_{j \in \mathcal{N}_i} w_{ij}^{(t+1)} } + \frac{\eta}{ |\mathcal{N}_i| }.
$$
BLISS is described as layer-wise and architecture-aware, with explicit handling of both fixed aggregation coefficients and attention-based settings [2512.22388].

A distinct mechanism appears in the black-box density paper that explicitly introduces “bandit importance sampling.” There the bandit acts on a candidate pool of space-filling points rather than on a probability vector over datapoints. A Gaussian process surrogate models a transformed density, and the next point is chosen by maximizing the GP Upper Jensen Bound,
$$
U_n(\theta) := \mathbb{E}_{f \sim \mathcal{GP}(m_n, k_n)} \left[ \phi(f(\theta)) \right].
$$
For $\phi(x)=\exp(x)$,
$$
U_n(\theta) = \exp\left( m_n(\theta) + \frac{1}{2} k_n(\theta, \theta) \right).
$$
After $N$ sequential selections, self-normalized weights are assigned:
$$
w_n^* = \frac{q(\theta_n^*) / u(\theta_n^*)}{\sum_{i=1}^N q(\theta_i^*) / u(\theta_i^*)}.
$$
This formulation differs from adaptive importance sampling because it does not adapt the proposal distribution; it directly designs the sample locations [2509.01437].

## 4. Theoretical properties

A defining theoretical theme in BIS is asymptotic approximation of an intractable variance-optimal strategy under partial feedback. For MABS in stochastic optimization, the main guarantee is
$$
\sum_{t=1}^T \mathbb{V}_e^t(p^t) \leq 3 \sum_{t=1}^T \mathbb{V}_e^t(p^\star) + 50 \sqrt{ n^5 T \overline{(a^2)} \ln n },
$$
where $p^\star$ is the best fixed distribution in hindsight. As $T \to \infty$, the average excess over three times the optimal effective variance vanishes [1708.02544]. The same factor-of-3 principle appears in the GNN adversarial-bandit formulation, which shows that the cumulative effective variance of the learned sampler is asymptotically within a factor of 3 of the oracle variance, plus a sublinear term [2006.05806].

Avare sharpens the comparison criterion by targeting dynamic rather than static regret. Under suitable conditions and step-sizes $\alpha_t = \mathcal{O}(1/t)$, it achieves $\mathcal{O}(T^{2/3})$ dynamic regret for SGD and $\mathcal{O}(T^{5/6})$ dynamic regret for SGLD [2103.12243]. This is significant because the optimal sampling distribution is not stationary; it moves with the optimization trajectory. A plausible implication is that regret notions based only on comparison to a fixed sampler may be insufficient when the gradient landscape evolves materially across iterations.

The selective importance sampling estimator in extreme bandit feedback is analyzed through explicit bias and MSE bounds. Its bias satisfies
$$
\left| \mathbb{E}[\hat{V}^\Phi_{\text{sIS}(\pi)}] - V(\pi) \right| \leq \kappa(\pi, \Psi, \Phi),
$$
where
$$
\kappa(\pi, \Psi, \Phi) = \mathbb{E}_{P(x)}\left[ \pi(\Psi(x) \cap \Phi^0(x) \mid x) \right].
$$
Its MSE obeys
$$
\mathrm{MSE}[\hat{V}^\Phi_{\text{sIS}(\pi)}] \leq \mathrm{MSE}[\hat{V}_\text{IS}(\pi)] + 2\sigma^2 \kappa(\pi, \Psi, \Phi) - \frac{\sigma^2}{n} \mathbb{E}_{P(x)} \frac{\pi^2(\Phi^0(x) \mid x)}{\rho(\Phi^0(x) \mid x)}.
$$
If $\Phi(x)$ contains all relevant actions, the estimator is unbiased and has lower variance due to conditioning [2009.12947].

For expensive finite-sum optimization via contextual bandits, the theoretical comparison is against the best expert in hindsight. Using Exp4, the regret grows as $O(\sqrt{p\log(N)K})$ with the number of rounds $K$, and the paper states that expected regret relative to the best expert is bounded and decreases as $1/\sqrt{K}$ [2604.20657]. In the black-box density setting, the theory is not framed as regret minimization over a probability simplex, but as weak convergence of the weighted empirical measure. With a space-filling proposal sequence such as a scaled Halton sequence,
$$
d(p, \delta_N) \leq C_1 \frac{\log(N+M)^d}{N+M} + C_2 \frac{M}{N+M},
$$
which is stated to be faster than vanilla Monte Carlo’s $O(N^{-1/2})$ rate [2509.01437].

## 5. Representative application domains

The published uses of BIS span several distinct application regimes.

| Domain | Representative method | Key stated outcome |
|---|---|---|
| Stochastic optimization | MABS [1708.02544] | Asymptotically approximates the optimal variance within a factor of 3 |
| Finite-sum optimization and sampling | Avare [2103.12243] | $\mathcal{O}(T^{2/3})$ dynamic regret for SGD and $\mathcal{O}(T^{5/6})$ for SGLD |
| Graph neural networks | GNN-BS [2006.05806], BLISS [2512.22388] | Lower sampling variance and strong empirical performance on multiple datasets |
| Extreme bandit feedback / XMC | sIS and POXM [2009.12947] | Better bias-variance regime than standard IS in very large action spaces |
| Expensive finite-sum optimization | Exp4-based subset selection [2604.20657] | Robust mixing of expert advice, uniform exploration, and side information |
| Black-box density sampling | BIS with GP-UJB [2509.01437] | Accurate approximations with fewer target evaluations |

In optimization, BIS has been paired with SGD, SVRG, SAGA, Adam, and SGLD. “Adam with Bandit Sampling” generalizes Adam by maintaining a distribution over training examples and updating it with an EXP3-style multi-armed bandit rule. The paper states that Adambs improves the convergence rate of Adam—$O(\sqrt{\frac{\log n}{T} })$ instead of $O(\sqrt{\frac{n}{T})$ in some cases—and reports fast convergence in practice on MLP, CNN, LSTM, and RCNN workloads [2010.12986].

In GNNs, the need for BIS is especially acute because optimal samplers depend on embeddings and learned attention weights. The 2020 bandit sampler paper reports highest or near-highest test F1 scores across datasets and architectures and emphasizes robustness on attentive GNNs where existing variance-reduced samplers are less applicable [2006.05806]. BLISS extends the bandit-layer importance sampling strategy to GraphSAGE and GATv2, reporting higher test F1-score than PLADIES on Citeseer, Pubmed, Cora, and Yelp, while performing nearly identically on Flickr and Reddit [2512.22388].

In logged bandit learning with extremely large action spaces, POXM uses selective importance sampling with top-$p$ actions under the logging policy. On EUR-Lex, Wiki10-31K, and Amazon-670K, the reported R@5 values in Table 2 are 44.48, 53.03, and 23.72 for POXM, compared with 13.51, 36.16, and 14.54 for BanditNet [2009.12947]. In expensive Bayesian computation, BIS for black-box densities reportedly required approximately 5% of the target evaluations of QMC-based IS to achieve the same approximation error on benchmark densities and produced accurate posterior approximations on the Lorenz weather model and the g-and-k model [2509.01437].

## 6. Distinctions, misconceptions, and related directions

One common misconception is to treat BIS as synonymous with ordinary adaptive importance sampling. The black-box density formulation explicitly distinguishes itself from adaptive importance sampling by stating that it does not optimize a proposal distribution and instead directly designs the samples through a sequential strategy that combines space-filling designs with multi-armed bandits [2509.01437]. By contrast, MABS, Avare, Adambs, and the GNN methods do optimize or adapt sampling distributions over datapoints, components, or neighbors [1708.02544], [2103.12243], [2010.12986], [2006.05806].

A second misconception is that BIS is necessarily unbiased. The literature is more nuanced. In optimization and GNN sampling, unbiasedness is typically preserved through inverse-probability weighting of gradients or messages [1708.02544], [2006.05806]. In extreme bandit feedback, sIS is deliberately allowed to be slightly biased when the selected subset $\Phi(x)$ omits relevant actions, in exchange for potentially substantial variance reduction [2009.12947]. In black-box density sampling, the final estimator uses self-normalized importance weights rather than an i.i.d. proposal-sampling scheme [2509.01437].

A third misconception is that bandit methods are used only when no side information is available. The contextual-bandit formulation for expensive finite-sum optimization instead emphasizes the opposite case: useful side information may come from alternative lower-fidelity simulations, pre-trained emulators, or domain expertise from humans or AI models, and Exp4 is used to mix expert advice with uniform exploration [2604.20657]. This suggests that BIS can function as a mechanism for integrating heterogeneous prior information rather than merely learning from scratch.

Several adjacent directions should also be distinguished from BIS proper. “Importance Sampling with Unequal Support” studies a new variant of importance sampling that can reduce variance by orders of magnitude when the supports of training and testing distributions differ, and provides theoretical analysis of bias and variance relative to ordinary importance sampling [1611.03451]. That work addresses a closely related variance-reduction problem, but the available abstract does not describe a bandit mechanism. Likewise, “Importance Sampling is All You Need” introduces a prompt-centric evaluation framework called BIS for estimating LLM benchmark performance by reweighting prompt distributions with IWAE-based density models and weight truncation, reporting an average absolute prediction error of 1.1% for code correctness scores and 2.15% for pass@1 [2508.01203]. Despite the acronym, the method is not described in the provided material as a multi-armed bandit procedure.

Taken together, the arXiv literature portrays BIS as an umbrella for sequential importance-weighted sampling schemes that are adaptive, feedback-driven, and often explicitly exploration–exploitation balanced. The unifying idea is not a single formula, but a design principle: variance-relevant sampling decisions are treated as online decisions under uncertainty, and importance weighting is coupled to bandit or bandit-like adaptation to approach intractable oracle strategies with practical feedback budgets [1708.02544], [2103.12243], [2006.05806], [2509.01437].

Source: https://www.emergentmind.com/topics/bandit-importance-sampling-bis