Efficient Adaptive Rejection Sampling

• EARS is a family of adaptive sampling methods that refines envelopes, proposals, and acceptance thresholds to overcome inefficiencies in classical rejection sampling.
• It employs techniques such as gradient-refined proposals, adaptive masking, and ratio-of-uniforms to efficiently sample from non-standard, multimodal, and high-dimensional distributions.
• EARS provides strong theoretical guarantees and practical computational savings, making it ideal for applications like language decoding and matrix volume sampling.

Efficient Adaptive Rejection Sampling (EARS) refers to a class of methodologies designed to enhance the efficiency and scope of rejection sampling by adaptively constructing proposals, acceptance thresholds, or envelopes. EARS is not a single algorithmic instantiation, but rather a family of adaptive schemes that emphasize minimal tuning, theoretical guarantees, high acceptance rates, and the ability to operate in scenarios where classical rejection sampling, including ARS (Adaptive Rejection Sampling) for log-concave distributions, becomes inefficient, inapplicable, or intractable. Modern EARS variants play a central role in fields ranging from LLM decoding and high-dimensional simulation to matrix volume sampling, stochastic filtering, and constrained generation.

1. Foundational Principles and Variants

At its core, EARS addresses the rejection rate bottleneck and proposal inefficiency of classical rejection sampling. Various EARS-type schemes adapt either the envelope (“hat” function), the proposal, or the acceptance criterion based on online observations, preceding samples, local geometry, or model uncertainty. Notable implementations include:

• Marginal-potential and ratio-of-uniforms adaptation: As in Martino and Míguez’s two EARS schemes, envelope construction and geometric adaptation enable efficient exact sampling from non-log-concave, multimodal, and log-convex-tailed distributions (Martino et al., 2011).
• Gradient-refined proposals: Parameterized mixtures, empirically optimized via mini-batch softmax losses, make the proposal converge to the target density without requiring analytic suprema (Raff et al., 2023).
• Adaptive reweighting and deterministic removal: For discrete domains, such as LLM output supports, EARS can adaptively mask previously-rejected tokens, thus avoiding repeat computation and achieving order-of-magnitude runtime improvements even for large support sets (Lipkin et al., 7 Apr 2025).
• Uncertainty-based relaxation: In speculative decoding for autoregressive models, EARS dynamically relaxes the acceptance criterion based on target model uncertainty, reducing random rejections and accelerating throughput (Sun, 15 Dec 2025).

2. Theoretical Guarantees and Minimax Bounds

EARS methodology is often justified via strong theoretical risk and loss bounds under weak regularity assumptions. For instance, the Nearest-Neighbor Adaptive Rejection Sampling (NNARS), sometimes labelled as EARS, achieves the minimax optimal rejection rate among adaptive samplers for Hölder-smooth densities:

• Problem setup: For $f : [0,1]^d \to \mathbb{R}_+$ with $s$-Hölder smoothness and $n$ sampling budget, NNARS achieves a rejection loss $L_n = n-\hat n$ with $E[L_n] = O(n^{1-s/d}\log^2 n)$, matching the lower bound up to logarithmic factors (Achdou et al., 2018).
• Key proof technique: Piecewise-constant proposals on adaptively refined grids, using approximate nearest neighbors, simultaneously guarantee envelope validity and proposal coverage under smoothness and boundedness.
• Contrast with classical ARS: NNARS generalizes ARS to higher dimensions, arbitrary smoothness, and multimodality, while providing explicit non-asymptotic guarantees.

3. Algorithmic Design: Proposals, Thresholds, and Envelopes

EARS designs are unified by adaptive mechanisms that refine the proposal or acceptance region:

• Stepwise marginal-potential envelopes (EARS-I): Partition the domain into intervals with locally tight piecewise-constant envelopes based on the decomposition of the potential. Upon rejection, new support points are inserted, tightening the envelope and improving future acceptance rates. This procedure is robust to non-concavity and log-convex tails as long as one factor decays sufficiently (Martino et al., 2011).
• Ratio-of-uniforms (EARS-II): Constructs an adaptive geometric envelope in the $(u,v)$ plane by triangulating the region implied by the ratio-of-uniforms theorem. Upon rejection, triangles are split, strictly shrinking the envelope (Martino et al., 2011).
• Gradient-refined mixture proposals: In continuous domains, acceptance ratios $f(x)/g(x;\theta)$ are minimized in a smooth-softmax sense via autodifferentiation, using GMMs as flexible proposals, and updating empirical suprema to maintain correctness (Raff et al., 2023).
• Adaptive masking in discrete supports: For constrained language generation, EARS adaptively removes rejected tokens from the proposal, so each forbidden candidate is tested only once; acceptance is provably exact in the locally masked subspace (Lipkin et al., 7 Apr 2025).
• Uncertainty-tolerant thresholds: In speculative decoding, EARS dynamically adapts the acceptance threshold based on model uncertainty, measured as $u_t = 1 - \max_v P_t(v)$, and relaxes the acceptance via a tolerance $\tau_t = \beta u_t$ (Sun, 15 Dec 2025).

4. Computational Complexity and Efficiency Analysis

The computational savings of EARS arise from the local adaptation and minimal redundancy:

• Per-iteration cost: In stepwise and RoU EARS, the dominant cost is $O(\log m_t)$ table or support search (via priority queues/balanced trees), with typically $O(1)$ cost for proposal drawing and envelope update due to localized refinement (Martino et al., 2011).
• Acceptance rates: Empirically, after a small number of adaptations, EARS schemes routinely reach acceptance rates $R_t > 0.95$, even on severe multimodal or log-convex-tailed targets; further improvements to near-unity can be obtained with incremental interval or triangle splitting (Martino et al., 2011).
• NNARS and high dimensions: Piecewise-constant grid-based proposals retain $O(n \log n)$ overall complexity and achieve minimax acceptance for $d$-dimensional Hölder classes (Achdou et al., 2018).
• Gradient-refined ERS (GMM-based): Dominant cost is GMM sampling ($O(B d K)$ per batch) and infrequent mixture refits or refinements ($O(N K' d I_{em})$ per EM). Batch refinement during suprema updates ensures empirical error decay $O(1/n)$ (Raff et al., 2023).
• Controlled generation (large $|V|$): ARS removes redundant evaluations (each forbidden token is tested once). Empirical speedups of up to $50\times$ over naive masking are demonstrated, with cost closely tracking the KL divergence between constrained and unconstrained distributions (Lipkin et al., 7 Apr 2025).

5. Integration in Modern Applications

EARS techniques are widely adopted in both classical simulation and modern ML contexts:

• Speculative decoding for LLMs: Integration requires no architectural changes; EARS modules intervene via the acceptance threshold, yielding up to $+18.12\%$ throughput and $–4.08\%$ latency, with only $–0.84\%$ accuracy drop on GSM8K reasoning benchmarks (Sun, 15 Dec 2025). The entire adaptation pipeline can be fused into a custom logits-processor compatible with standard inference toolkits.
• Volume sampling in matrix algorithms: EARS-based rejection sampling enables $O(mk + k^3\log k)$ complexity for pivot set selection in adaptive randomized pivoting (ARP), producing volume-sampled columns/rows exactly, and guaranteeing active learning and matrix approximation fidelity (Epperly, 2 Oct 2025).
• Constrained and controlled decoding: In language and discrete structure modeling, EARS offers exact local sampling under hard constraints, and weighted variants provide unbiased SMC weights, resolving the myopic behavior of locally constrained proposals (Lipkin et al., 7 Apr 2025).

6. Empirical Performance and Benchmarking

Empirical assessments consistently demonstrate the practical impact of EARS:

Setting	EARS Variant	Notable Metrics & Results	Reference
LLM decoding	Uncertainty-threshold	+18.12% throughput, –0.84% acc. cost, <1% overhead	(Sun, 15 Dec 2025)
Controlled LM	ARS/AWRS	50× runtime over masking, ~5–10% accuracy gain SMC-AWRS	(Lipkin et al., 7 Apr 2025)
Density sim	ERS (GMM-proposal)	2–7× acceptance, 2–4× speedup vs. prior general schemes	(Raff et al., 2023)
Classic filter	EARS-II (RoU)	Acceptance >0.4 for stochastic volatility, MSE≈1.48	(Martino et al., 2011)
Matrix ARP	EARS volume-sampling	O(k/ log k) speedup, preserves exact statistical law	(Epperly, 2 Oct 2025)

In all cases, adaptation to local structure, constraints, or uncertainty is key to EARS’s performance benefit, yielding orders-of-magnitude gains or achieving theoretical optima where prior methods degrade severely.

7. Limitations, Open Problems, and Future Directions

Several limitations, caveats, and research directions arise:

• Smoothness and domain assumptions: Piecewise-constant methods rely on sufficient regularity; proposals need to be bounded below everywhere, and heavy or ultra-slow-decaying tails require explicit treatment (e.g., generalized RoU for $1/x^r$ tails) (Martino et al., 2011).
• Uncertainty measures: In speculative decoding, $u_t=1-\max_v P_t(v)$ is a crude entropy proxy; fine-grained entropy- or variance-based tolerances may yield further performance gains or lower risk of semantic drift (Sun, 15 Dec 2025).
• Hyperparameter tuning: Parameters such as the tolerance aggression $\beta$, batch sizes, or the number of mixtures/components require some per-task tuning, though empirical results suggest insensitivity across a wide range (Raff et al., 2023, Sun, 15 Dec 2025).
• Extension to interacting drafts/sequences: Extending EARS to multi-sequence environments (e.g., multi-draft speculative decoding, chain-of-thought reasoning) is not yet fully explored (Sun, 15 Dec 2025).
• Curse of dimensionality: Despite minimax-optimal performance, acceptance rates of all adaptive samplers decay with dimension unless the density is sufficiently regular or structured (Achdou et al., 2018, Raff et al., 2023).
• Implementation details: Maintaining robust and numerically stable updates, especially in the vicinity of near-flat or vanishing gradients, requires careful engineering—particularly for GMM-based and geometric envelope schemes (Martino et al., 2011, Raff et al., 2023).

A plausible implication is that future research will increasingly favor EARS-type samplers for both classical and modern large-scale inference, provided that technical challenges in modeling uncertainty, tailoring envelopes, and scaling to extreme dimensions are addressed with novel adaptation architectures.