Entropy-Guided Rejection Sampling

• Entropy-guided rejection sampling is a method that balances entropy cost with proposal design to achieve near-optimal exact sampling and channel simulation.
• It employs strategies such as ALDR and bits-back rejection schemes to minimize expected random bit usage while maintaining practical runtime and space requirements.
• The approach effectively navigates tradeoffs between space, time, and entropy, achieving performance bounds close to theoretical limits like Shannon entropy and mutual information.

Entropy-guided rejection sampling refers to a class of exact sampling and channel simulation algorithms that use rejection sampling mechanisms with entropy-theoretic guidance to achieve minimax or optimality with respect to the information-theoretic lower bounds for randomness (entropy) or communication in sampling and coding tasks. Such methods explicitly balance the entropy cost of sampling with the structure of the distribution or channel, yielding algorithms that in many regimes attain or closely approach the theoretical limits set by Shannon entropy, mutual information, or the functional information of the problem.

1. Shannon-Theoretic Optimality in Discrete Sampling

Consider sampling $X$ from a finite discrete distribution $P = (p_1, \dots, p_n)$, with each $p_i = a_i / m$ rational and given access only to i.i.d. $\mathrm{Bern}(1/2)$ coin flips. The fundamental lower bound on randomness consumption for any exact sampling procedure is the Shannon entropy: $H(P) = -\sum_{i=1}^n p_i \log_2 p_i$ Knuth and Yao established that entropy-optimal algorithms exist, with expected random bit usage in $[H(P), H(P)+2)$ (Draper et al., 5 Apr 2025). However, classical efficient samplers such as Walker's alias method, while runtime-optimal, are far from entropy-optimal in the presence of highly skewed $P$, often requiring $\log_2 n + O(1)$ bits per sample, substantially above $H(P)$.

2. Entropy-Guided Rejection Sampling Algorithms

The core strategy of entropy-guided rejection sampling is to carefully design the proposal distribution, acceptance rule, and data structure so that the expected cost in random bits remains near the entropy lower bound, but the state and time complexity are practical. A central example is the family of "Amplified Loaded Dice Roller" (ALDR) algorithms (Draper et al., 5 Apr 2025):

• Proposal Construction: Starting from the $m$-type rational distribution, ALDR selects a proposal $P = (p_1, \dots, p_n)$0 with all probabilities as integer multiples of $P = (p_1, \dots, p_n)$1, for $P = (p_1, \dots, p_n)$2, allowing exact implementation via a fixed-depth Knuth–Yao tree.
• Rejection Criterion: Index $P = (p_1, \dots, p_n)$3 is accepted if $P = (p_1, \dots, p_n)$4 (where 0 indexes extra proposal mass not corresponding to any $P = (p_1, \dots, p_n)$5), ensuring rejection probability and acceptance rates are controlled by the amplification parameter $P = (p_1, \dots, p_n)$6.
• Entropy Analysis: As $P = (p_1, \dots, p_n)$7 increases relative to $P = (p_1, \dots, p_n)$8, the expected coin flips per accepted sample approaches $P = (p_1, \dots, p_n)$9, with the definitive bound $p_i = a_i / m$0 achieved for $p_i = a_i / m$1.
• Complexity: ALDR achieves $p_i = a_i / m$2 (linearithmic) space and preprocessing time, and $p_i = a_i / m$3 expected sample runtime for fixed-precision $p_i = a_i / m$4.

These algorithms bridge the exponential-space gap of general entropy-optimal samplers and the entropy gap of practical table-based samplers like alias (Draper et al., 5 Apr 2025).

3. Optimal Channel Simulation and Relative Entropy Coding

In the context of channel simulation—drawing $p_i = a_i / m$5 given $p_i = a_i / m$6—entropy-guided rejection sampling appears in the construction of relative-entropy codes, where one targets a sample $p_i = a_i / m$7 at the receiver using minimal average description length (bits) from the sender. The lower bound is mutual information $p_i = a_i / m$8, but achieving $p_i = a_i / m$9 exactly is only possible up to an additive logarithmic redundancy for general channels (Hill et al., 25 Apr 2026).

The ring-toss code, introduced in (Hill et al., 25 Apr 2026), applies an entropy-guided rejection strategy whereby the acceptance index $\mathrm{Bern}(1/2)$0 (first successful trial under rejection sampling) is encoded under its conditional distribution (guided by channel structure and shared randomness) rather than naively, thereby reducing redundancy. The result demonstrates that: $\mathrm{Bern}(1/2)$1 where $\mathrm{Bern}(1/2)$2 (functional information) refines mutual information to the tightest possible form for channel simulation. For singular channels, the redundancy over $\mathrm{Bern}(1/2)$3 drops to $\mathrm{Bern}(1/2)$4 bits, and the methods are tight in both one-shot and asymptotic regimes (Hill et al., 25 Apr 2026).

4. Bits-Back Rejection Schemes in Singular Channels

Recent work on bits-back rejection sampling (BBRS) in (Flamich et al., 7 Apr 2026) further advances entropy-guided approaches, specifically for singular channels $\mathrm{Bern}(1/2)$5 where the conditional law $\mathrm{Bern}(1/2)$6 depends only on $\mathrm{Bern}(1/2)$7. BBRS employs:

• Stage 1: Quantization of the log-density ratio and greedy rejection sampling for $\mathrm{Bern}(1/2)$8, encoding acceptance index $\mathrm{Bern}(1/2)$9 in a manner that enables bits-back recovery at the decoder.
• Stage 2: Standard rejection to sample $H(P) = -\sum_{i=1}^n p_i \log_2 p_i$0 conditioned on the quantized value $H(P) = -\sum_{i=1}^n p_i \log_2 p_i$1.
• Efficiency: The achieved expected bits per symbol is

$H(P) = -\sum_{i=1}^n p_i \log_2 p_i$2

with asymptotic redundancy $H(P) = -\sum_{i=1}^n p_i \log_2 p_i$3 for singular channels, outperforming previous schemes in both constants and implementability.

This construction generalizes entropy-guided rejection ideas to full relative entropy coding for channels, integrating quantization, greedy rejection, and bits-back codes (Flamich et al., 7 Apr 2026).

5. Space-Time–Entropy Tradeoffs and Comparison with Standard Methods

A central property of entropy-guided rejection samplers is the explicit navigation of tradeoffs between entropy cost, space complexity, and runtime. The ALDR method uniquely achieves linearithmic space, $H(P) = -\sum_{i=1}^n p_i \log_2 p_i$4 expected time per sample, and ensures expected random bits consumed within $H(P) = -\sum_{i=1}^n p_i \log_2 p_i$5 of the entropy lower bound, even for large, highly skewed distributions (Draper et al., 5 Apr 2025).

The table below summarizes key distinguishing features of several prominent algorithms:

Algorithm	Entropy Cost	Space Complexity	Runtime per Sample
Knuth–Yao tree	$H(P) = -\sum_{i=1}^n p_i \log_2 p_i$6	Exponential	$H(P) = -\sum_{i=1}^n p_i \log_2 p_i$7
Alias method	$H(P) = -\sum_{i=1}^n p_i \log_2 p_i$8	$H(P) = -\sum_{i=1}^n p_i \log_2 p_i$9	$[H(P), H(P)+2)$0
ALDR	$[H(P), H(P)+2)$1	$[H(P), H(P)+2)$2	$[H(P), H(P)+2)$3 (expected)
Ring-toss code	$[H(P), H(P)+2)$4	N/A (stream coding)	$[H(P), H(P)+2)$5 (expected)
BBRS	$[H(P), H(P)+2)$6 (singular)	As coding stream	Practical (stream)

The empirical performance of these algorithms corroborates the entropy-efficiency benefits: ALDR uses $[H(P), H(P)+2)$7 bits per sample while the alias method's cost can be up to an order of magnitude larger. For large $[H(P), H(P)+2)$8 and skewed $[H(P), H(P)+2)$9, this directly translates into both faster sampling and lower cryptographically-secure entropy usage (Draper et al., 5 Apr 2025).

6. Mechanisms Underlying Entropy Guidance

Across both discrete sampling and channel simulation, two technical principles are fundamental:

• Dyadic Proposal Design: Ensuring proposal probabilities align with powers of two simplifies Knuth–Yao tree implementations, enables fixed-depth representations, and prevents exponential blowup.
• Amplification and Quantization: Carefully chosen amplification (in ALDR) or quantization steps (in BBRS) ensure the entropy gap (Knuth–Yao toll, bits-back recovery) remains below strict analytical thresholds, guaranteeing tightness of the entropy upper bound.

This design philosophy enables shifting between space-optimal (large $P$0) and entropy-optimal (large amplification or quantization resolution) regimes as dictated by application needs (Draper et al., 5 Apr 2025, Flamich et al., 7 Apr 2026).

7. Asymptotic Regimes, Tightness, and Outlook

The theoretical bounds achieved by entropy-guided rejection sampling are tight: the redundancy additive terms (e.g., $P$1 or $P$2 in the ring-toss code) are proven necessary in general (Hill et al., 25 Apr 2026). For singular channels, both BBRS and the ring-toss code reach zero log-factor redundancy and are thus optimal for both finite and infinite block length.

Entropy-guided rejection sampling therefore constitutes a unified paradigm bridging information theory, random variate generation, and practical coding by marrying fine-grained entropy control with efficient rejection strategies. It has set new benchmarks in exact sampling and channel simulation, both in space–time–entropy optimality and practical implementability, and continues to inform refinements in functional information analysis and channel coding theory (Draper et al., 5 Apr 2025, Flamich et al., 7 Apr 2026, Hill et al., 25 Apr 2026).