---
title: Adaptive Weighted Rejection Sampling
url: https://www.emergentmind.com/topics/adaptive-weighted-rejection-sampling
type: topic
---

# Adaptive Weighted Rejection Sampling

Adaptive weighted rejection sampling encompasses a class of rejection sampling methods that dynamically construct proposal envelopes—often mixtures or piecewise majorizers—to increase acceptance rates when sampling from target densities with intractable or unstandardized form. These techniques generalize classical rejection sampling by introducing adaptive partitions of the support, weighted proposals, and surrogate or data-driven weights, yielding substantial gains in both efficiency and generality, particularly for non-logconcave, multimodal, or constraint-defined targets.

## 1. Fundamentals and Problem Setting

Let the target density be defined as
\[
f(x) = \frac{w(x)g(x)}{Z}, \qquad Z = \int_{\Omega} w(x)g(x)\,d\nu(x),
\]
where $g(x)$ is a tractable base distribution, $w(x)\geq 0$ is a weight function, and $Z$ is typically unknown or intractable. This structure arises in Bayesian hierarchical models (with $g$ a prior, $w$ a likelihood or complex importance weight), constrained autoregressive models, and other settings where the normalization constant cannot be computed efficiently [2401.09696].

Classical rejection sampling draws $x \sim g$, then accepts $x$ with probability $w(x)/M$, requiring construction of $M\geq \sup_x w(x)$ for efficiency and correctness. Adaptive weighted rejection sampling seeks to choose or build proposal distributions $q(x)$ adaptively, so accepted draws are not excessively rare and proposal evaluation/sample generation remains tractable even in complex or high-dimensional settings.

## 2. Adaptive Envelope Construction and Majorization

A central approach is to partition the domain $\Omega$ into $K$ disjoint regions $(R_1,\ldots,R_K)$ and define, on each $R_k$, a simple majorant (envelope) $\overline w_k(x)\ge w(x)$—typically constant or log-linear upper bounds. This yields a composite proposal envelope:
\[
h_0(x) = \sum_{k=1}^K \overline w_k(x)g(x)\mathbf{1}_{R_k}(x),
\]
with associated proposal density $h(x) = h_0(x)/\Psi$, $\Psi = \sum_{k=1}^K \xi_k$, $\xi_k = \int_{R_k} \overline w_k(x)g(x)dx$ [2401.09696].

Sampling proceeds by:

1. Drawing a region index $k \sim \{\pi_k\}$, $\pi_k = \xi_k/\Psi$.
2. Drawing $x \sim g_k(x) \propto \overline{w}_k(x)g(x)$ restricted to $R_k$.
3. Accepting $x$ with probability $w(x)/\overline{w}_k(x)$.

This composite majorization underpins the “vertical weighted strips” (VWS) methodology, which is applicable even when $\log w(x)$ is not concave, and $g(x)$ is not an exponential family [2401.09696, 2509.17155].

## 3. Adaptive Partitioning and Error/Acceptance Rates

Partition refinement is driven by the need to minimize the rejection rate. The VWS method adaptively bifurcates regions based on their contribution to an upper bound for the expected rejection probability. The rejection probability bound is given by:
\[
1-\alpha \leq \frac{\sum_{k=1}^K (\xi_k - \underline{\xi}_k)}{\sum_{k=1}^K \xi_k},
\]
where $\underline{\xi}_k = \int_{R_k} \underline{w}_k(x)g(x)dx$ and $\underline{w}_k(x)\leq w(x)$ is a minorant [2401.09696].

Regions most responsible for poor tightness (as measured by $\rho_k = (\xi_k-\underline{\xi}_k)/\Psi$) are split, most often at their midpoint, with majorizers/minorizers recalculated. Splitting continues until the number of mixture components $K$ or the bound itself meets prescribed thresholds. With moderate $K$ (e.g., 10–50), the acceptance rate often reaches 80–99% for challenging target densities [2401.09696].

For persistent use within iterative samplers (e.g., Gibbs), "self-tuned" VWS maintains and incrementally adapts the mixture proposal as rejections accumulate, periodically pruning components with negligible contribution to the rejection bound [2509.17155].

## 4. Algorithmic Implementation and Pseudocode

The general adaptive weighted rejection sampler implements two central phases:

**A. Offline proposal construction**

```python
Initialize partition {R_1} = {Ω}
Compute ξ_k, underline_ξ_k, ρ_k for each R_k
While len({R_k}) < K_max and sum(ρ_k) > ε:
    Sample region k* ~ {ρ_k}
    Split R_{k*} at midpoint, update ξ_k, underline_ξ_k, ρ_k
Return {R_k}, {overline_w_k}, {π_k}
```

**B. Rejection sampling (online)**

```python
While accepted_samples < n:
    Draw k ~ {π_k}
    Draw x ~ g_k(x)
    Draw u ~ Uniform(0,1)
    If u <= w(x)/overline_w_k(x): accept x
    Else: reject and repeat
```
[2401.09696]

Efficient implementation requires caching truncated base normalizations, efficient component selection (e.g., prefix sums), and numerical optimization for envelope construction.

## 5. Extensions, Theoretical Guarantees, and Empirical Performance

Adaptive weighted rejection sampling subsumes several specialized techniques:

- **Direct Sampling with Adaptive Step Envelopes**: Approximating the required marginalization density by a step-function envelope $h^*(u)\geq p^*(u)$ over an auxiliary variable enables exact sampling with controlled, adaptively reduced rejection rates. Knot selection is driven by the largest discrepancy between $h^*$ and $p^*$, with theoretical total-variation and rejection probability bounds in terms of step rectangles [2203.15852].
- **Alternative Adaptive Schemes**: Generalizations allow for piecewise exponential or convex-concave decompositions, as in adaptive rejection Metropolis and ratio-of-uniforms samplers—particularly for non-logconcave or multimodal targets [1205.5494, 1111.4942]. Each class maintains an envelope $q_n(x)\geq p(x)$, updates support points on rejection, and offers acceptance probabilities approaching unity as envelopes tighten.
- **Weighted Proposals in Approximate Bayesian Computation (ABC) and SMC**: Adaptive weighting, particularly data-driven kernel-based importance weights, dramatically raises acceptance rates in particle systems [1503.07791].

Acceptance rates, sample quality, and computational cost have been benchmarked on variety of distributions, from directional and von Mises–Fisher models [2401.09696] to complex discrete settings and real-world Gibbs samplers [2509.17155]. Linear or log-linear envelope choices within each strip achieve considerable runtime/efficiency improvements, e.g., >95% acceptance in vMF, and effective sample size gains in Bayesian small area estimation scenarios.

## 6. Related Developments and Applications

Adaptive weighted rejection sampling methods have a broad impact across computational statistics:

- **Constrained Language Model Generation**: Adaptive weighted rejection sampling, including prefix-trie adaptive pruning and unbiased importance weighting, outperforms local masking or naive rejection in constrained autoregressive generation, both empirically and in terms of theoretical sample complexity [2504.05410, 2510.01902].
- **Monte Carlo Event Generation with Surrogates**: Two-stage adaptive rejection schemes incorporating neural network surrogates as fast minorants significantly accelerate unweighting in high-energy physics event generation while retaining unbiased sampling [2109.11964].
- **Minimax-Optimal Adaptive Schemes**: Adaptive rejection sampling approaches matching lower bounds for expected rejections in $s$-Hölder function classes have been established [1810.09390].

Theoretical results guarantee that, under mild assumptions, the acceptance rate is non-decreasing and proposals can be made arbitrarily tight at cost proportional to proposal refinement. These methods remain exact, i.e., all accepted samples come from the true target.

## 7. Limitations and High-Dimensional Extensions

Current adaptive weighted rejection sampling theory and practice are most mature for univariate or moderate-dimensional targets. High-dimensional applications (e.g., hyperrectangular VWS, Voronoi partitions) face challenges in exact optimization of strip bounds, efficient sampling from truncated baselines, and envelope construction [2401.09696, 2509.17155]. Direct integration into high-dimensional conditional Gibbs updating remains tractable, as the adaptive mixture-proposal framework is applied to each univariate full conditional separately. Direct multivariate extension of VWS and related algorithms is a subject for ongoing research.

In summary, adaptive weighted rejection sampling unifies and generalizes a body of envelope-based proposal refinement techniques, providing rigorous, efficient, and often near-optimal exact sampling for a wide class of weighted and constrained densities. These methodologies are increasingly central in modern computational statistics, Bayesian inference, and probabilistic generative modeling [2401.09696, 2509.17155, 2203.15852, 2504.05410, 2109.11964, 1810.09390].

Source: https://www.emergentmind.com/topics/adaptive-weighted-rejection-sampling