Online Random Sampling with Real Probabilities

Abstract: We develop an efficient online algorithm to sample a sequence of discrete random variables using an entropy source of i.i.d. fair coin flips, in a standard model of real computation where real-valued probabilities are represented by rational approximations. For any sequence $F_1, F_2, \dots$ of probability distributions, our sampler generates $n$ outputs $X_1 \sim F_1, \dots, X_n \sim F_n$ using at most $$\mathbb{E}\left[H(F_1) +\dots + H(F_n)\right] + O(\log n)$$ coin flips in expectation while carrying $O(\log n)$ bits of persistent space, where $H$ is the Shannon entropy. Under standard assumptions, we prove that the space used by our sampler to achieve this information-theoretically optimal entropy rate is asymptotically optimal. The key idea is to replace the global arithmetic-decoding sampling scheme of Han and Hoshi (1997) with a local discrete uniform state, yielding an exponential reduction in space for a given entropy loss. Our approach applies to distributions with irrational probabilities and countably infinite supports, generalizing recent randomness-recycling methods beyond finite rational distributions with bounded denominator.

• The paper achieves information-theoretic optimality by consuming an expected entropy within an additive O(log n) term using only independent coin flips.
• The method leverages a local randomness-recycling mechanism to maintain a compact persistent state of O(log n) bits, avoiding the heavy state overhead of traditional methods.
• It supports arbitrary computable discrete distributions, including those with irrational probabilities and infinite support, optimizing both space and runtime efficiency.

Online Random Sampling with Real Probabilities: An Authoritative Summary

Problem Setting and Motivation

The paper "Online Random Sampling with Real Probabilities" (2607.13828) addresses the fundamental algorithmic task of sampling from a sequence of arbitrary (potentially countably infinite, and supported on irrational probabilities) discrete distributions in an online manner, where distributions are revealed one at a time and each output variable must be exact and potentially dependent on previous outcomes. The sampling algorithm must utilize only an i.i.d. entropy source—namely, fair coin flips—while minimizing (i) expected entropy loss (the excess entropy consumed beyond the Shannon entropy of the output), (ii) persistent and temporary space, and (iii) running time, in a model where the only available access to distributions is via rational-approximation CDF oracles.

This setting extends prior works that either assumed access to finite rational distributions or required unbounded, often impractical global state sizes. Practical applications of such online samplers appear in cryptographic, stochastic simulation, and randomized algorithmic engines where entropy is at a premium and distributions are non-stationary or adaptive.

Main Contributions

The central contribution is an online random sampling algorithm that attains information-theoretically optimal entropy usage—that is, the expected number of coin flips is within an additive $O(\log n)$ term of the total output entropy after $n$ samples—using only $O(\log n)$ bits of persistent state. The algorithm is applicable to arbitrary computable discrete distributions, including those with irrational probabilities and infinite support.

A novel aspect is the local randomness-recycling mechanism: instead of carrying forward a potentially large global state (as in arithmetic coding-based approaches), the algorithm maintains a compact discrete uniform state $(Z, M)$, encoding the leftover randomness from previous rounds. Sampling from the next distribution uses this state together with fresh coin flips as needed. After each draw, a new pair $(Z', M')$ is extracted by a precise re-encoding procedure, ensuring unbiasedness, exact sampling, and state size bounded by a chosen parameter $\Delta$.

By tuning $\Delta$ to $O(i \log i)$ at step $i$, the per-round persistent state and entropy loss remain $O(\log n)$ after $n$0 samplings. Moreover, the algorithm interpolates between linear and logarithmic space-entropy trade-offs and supports any desired entropy gap $n$1 using only $n$2 persistent space.

Algorithmic Mechanics and Theoretical Analysis

The algorithm's operation hinges on simulating the effect of using an infinitely precise random real $n$3 (encoded by the binary fraction expansion of coin flips), which would suffice for exact inversion sampling. Given the sampled value $n$4 (such that $n$5), the conditional randomness $n$6— that is, the "position" of $n$7 inside this interval—is recycled into a new discrete uniform state via a rounding/projection mechanism that preserves exactness and independence.

Detailed technical results establish the following:

• Entropy Optimality: The algorithm achieves expected entropy use $n$8 without knowledge of $n$9 in advance. This is exponentially better in space for a given entropy loss than previous global-state solutions [han1997].
• Space Complexity: Persistent state required after $O(\log n)$0 samples is $O(\log n)$1 bits, optimal up to constant factors by a lower bound based on oracle access requirements. The expected overall space needed during execution is also $O(\log n)$2, where $O(\log n)$3 bounds the (conditional) means of the sampled variables.
• Algorithmic Flexibility: Space-entropy tradeoffs are supported: for any $O(\log n)$4, the entropy loss can be made $O(\log n)$5 with $O(\log n)$6 persistent state—an exponential improvement over batching and prior recycling methods.
• Generalization: The method accommodates infinitely supported and irrational-probability distributions, efficiently querying only a CDF oracle to any desired accuracy.
• Computational Bounds: Assuming efficient CDF oracles (where cost is polynomial in access precision and index), the expected time per step is $O(\log n)$7 operations, and expected temporary space per step is likewise logarithmic in $O(\log n)$8 and the typical output value.

Crucially, compared to the arithmetic coding (interval) method [han1997], the approach avoids the need to query past CDFs with increasing precision, and compared to rational-probability recycling [draper2025rr], it removes dependencies on the denominator structure and achieves more generality and efficiency.

Technical Results and Numerical Guarantees

Key theoretical guarantees include:

• Entropy Loss: For $O(\log n)$9 outputs, $(Z, M)$0, with $(Z, M)$1 arbitrarily small for suitable choice of state parameter $(Z, M)$2.
• Persistent State: $(Z, M)$3 bits with deterministic bounds, as opposed to only expected bounds in prior art.
• Lower Bound Matching: It is shown that any CDF-oracle-based online sampler must pay at least $(Z, M)$4 space; thus, the proposed scheme is optimal up to constants for all algorithms in the considered model.
• Runtime: For sufficiently efficient CDF oracles, expected time per sample is $(Z, M)$5.
• Robustness: The exactness of sampling and the efficiency of resource usage hold even when distributions and their probabilities adapt adversarially between rounds.

Comparisons and Broader Context

The work systematically advances beyond the following prior lines:

• Interval (Arithmetic Coding) Methods: Reduces persistent/temporary space from linear in $(Z, M)$6 to logarithmic in exchange for only an additive logarithmic entropy loss, and avoids precision creep in oracle queries [han1997].
• Randomness Recycling for Rational Distributions: Removes dependence on maximal denominator, supports arbitrary distributions, and compresses persistent state from $(Z, M)$7 to $(Z, M)$8 [draper2025rr].
• Batching and Batched I.I.D. Techniques: Achieves exponentially improved space for a given entropy loss compared to batch-based entropy-recycling [Kozen2022].
• Ad-hoc and Distribution-Specific Algorithms: Provides a generic online algorithm with near-optimal resource bounds applicable to all discrete distributions with computable CDFs, superseding distribution- or support-specific engines.

Implications and Future Directions

The established algorithm sets a new technical benchmark for entropy-efficient online random sampling under the strict CDF-oracle access model, particularly relevant in settings where (i) distributions have infinite or irrational support, (ii) outputs are non-i.i.d. or adaptively defined, or (iii) resource constraints on randomness and memory are significant.

Practically, this type of algorithm is likely to be important in the internals of future cryptographic primitives, embedded systems where randomness must be rationed, and massive Monte Carlo simulations where high-fidelity entropy management directly impacts throughput and hardware performance. The separation of persistent state from temporary space also affords better compositionality in algorithmic systems and allows rare flushes or resets without significant loss.

Theoretically, the work suggests further exploration of local recycling in other random transformation tasks beyond sampling (e.g., online data anonymization, Markov chain simulation, randomized rounding in streaming), as well as extensions to continuous-distribution and multi-dimensional settings. The necessity of $(Z, M)$9 persistent space for online CDF-oracle sampling may also have implications for lower bounds in streaming or communication complexity for related tasks.

Conclusion

"Online Random Sampling with Real Probabilities" (2607.13828) resolves a key open direction in the theory of random sampling, providing an entropy-loss-optimal and space-optimal online sampling scheme for arbitrary discrete distributions accessible only through approximation oracles. The paradigm of local randomness recycling yields both practical and theoretical advances over prior global-state and rational-case algorithms, and its implications are immediate for applications demanding both precision and efficiency in entropy usage and memory resource management. The optimality results and established trade-offs mark a definitive progress point in the domain, informing both future implementations and theoretical developments in random sampling methodology.