---
title: Random Float Sampling (RFS)
url: https://www.emergentmind.com/topics/random-float-sampling-rfs
type: topic
---

# Random Float Sampling (RFS)

Random Float Sampling (RFS) is a polysemous term in recent technical literature. In statistical computing, it commonly denotes the practice of generating discrete random outcomes from floating-point uniform variates, especially via the map \(Y = 1 + \lfloor mX \rfloor\), a construction shown to induce quantization bias under finite precision [1809.06520]. In differential privacy, the same phrase is used for secure floating-point sampling schemes that combine multiple independent uniforms so that sampled noise cannot be efficiently inverted back to its source randomness [2107.10138]. In Transformer research, RFS denotes a position-indexing strategy in which tokens are assigned randomly sampled continuous values during training to improve length generalization [2602.14050]. A further, hardware-oriented usage concerns direct uniform Float16 sampling with stochastic magnetic tunnel junctions (s-MTJs) [2501.00015]. These usages share an emphasis on float-valued randomness, but they address different problems, assumptions, and failure modes.

## 1. Scope and principal usages

In the cited literature, “Random Float Sampling” refers to several distinct mechanisms rather than a single canonical algorithm. The common element is that floating-point random values are treated as primary computational objects, but the objectives range from integer generation to privacy-preserving noise injection, positional indexing, and hardware-native sampling.

| Usage of RFS | Core mechanism | Representative source |
|---|---|---|
| Statistical integer generation | Multiply a random float by \(m\), take the floor, and add \(1\) | [1809.06520] |
| Secure random sampling in differential privacy | Combine multiple independent uniform variates through infinitely divisible decompositions | [2107.10138] |
| Transformer position indexing | Sample continuous position values in \([0,1)\), sort them, and use them as indices | [2602.14050] |
| Uniform Float16 generation | Configure bitwise Bernoulli probabilities for IEEE Float16 output using s-MTJs | [2501.00015] |

The terminological breadth matters because the mathematical critique applicable to multiply–floor integer generation does not apply to the differential-privacy or Transformer usages. Conversely, the security arguments developed for differential privacy do not address the positional out-of-distribution problem studied in sequence models.

## 2. Statistical-computing RFS: multiply–floor sampling and quantization bias

A central usage of RFS arises in software that generates a random integer uniformly on \(\{1,\ldots,m\}\) by drawing a pseudorandom float \(X\) on \([0,1)\) and returning
\[
Y = 1 + \lfloor mX \rfloor.
\]
R, in Version 3.5.1 patched, was described as using this strategy for random integers, and because the `sample` function relies on generating random integers, random sampling in R is biased [1809.06520].

The problem is finite-precision quantization. In practice, \(X\) is not truly continuous, but takes values
\[
X_k = \frac{k}{2^w}, \qquad k = 0,\ldots,2^w-1,
\]
typically with \(w=32\). The interval \([0,1)\) is partitioned into
\[
I_k = \left[\frac{k-1}{m}, \frac{k}{m}\right), \qquad k=1,\ldots,m,
\]
but the number of quantized float values falling into each \(I_k\) is generally unequal. Consequently,
\[
P(Y=k)=\#\{i : v_i \in I_k\}/2^w
\]
is not exactly uniform [1809.06520].

The distortion can be analyzed through the ratio of the largest and smallest output probabilities. For some \(m<2^w\),
\[
\frac{p_+(m)}{p_-(m)} = 1 + m2^{-w+1},
\]
to first order, where \(p_+(m)=\max_k \Pr(Y=k)\) and \(p_-(m)=\min_k \Pr(Y=k)\) [1809.06520]. The dependence on \(m\) is substantive rather than asymptotically negligible. With \(w=32\) and \(m=10^6\), the ratio is approximately \(1.00047\), corresponding to a \(0.047\%\) difference between the most likely and least likely outcomes. As \(m\) approaches \(2^{31}\), the ratio approaches \(2\), so one output can be about twice as likely as another [1809.06520].

The broader critique is not limited to one implementation. “Random Sampling: Practice Makes Imperfect” argues that many packages rely on the false assumption that PRNG outputs are IID \(U[0,1)\) variables, whereas actual PRNG outputs are discrete and generated from limited state spaces [1810.10985]. In that analysis, mapping discrete PRNG outputs to \(\{1,\ldots,m\}\) by multiplying and rounding can never be exactly uniform unless \(m\) is a power of \(2\). When \(m>2^w\), at least \(m-2^w\) values cannot be selected at all. The paper also gives a concrete example: for \(m=(2/5)\times 2^{32}=1,717,986,918\), R’s `sample()` function generates about \(40\%\) even numbers and \(60\%\) odd numbers [1810.10985].

These results isolate a structural flaw: even if the underlying PRNG were otherwise satisfactory, constructing integers through random floats can still produce biased sampling. The bias is a property of the mapping, not merely of the generator.

## 3. Direct integer generation and related algorithmic alternatives

The standard fix is to construct random integers directly from random bits rather than from floating-point multiplication. One formulation computes a minimal bit length \(b\) such that \(2^b \ge m\), generates an integer \(M \in \{0,\ldots,2^b-1\}\), and returns \(M+1\) only if \(M<m\); otherwise it rejects and retries [1809.06520]. Because every accepted bit-string corresponds to exactly one output in \(\{1,\ldots,m\}\), the resulting distribution is uniform if the input bits are independent and fair.

The method is explicitly described as the strategy taken in Python’s `numpy.random.randint()` function, and related descriptions identify the same rejection-based construction in C++11 `std::uniform_int_distribution` and the Mersenne Twister documentation [1809.06520]. The trade-off is that some draws are rejected; if \(m\) is not a power of \(2\), the rejection probability can be substantial, approaching \(50\%\) in the worst case [1809.06520].

The critique in [1810.10985] extends from integer generation to sampling workflows. It recommends avoiding methods that assume PRNG outputs are IID \(U[0,1)\), including generating a random sample by permuting the population and taking the first \(k\) items, or generating random integers by multiplying a pseudo-random binary fraction or float by a constant and rounding the result. More accurate methods are described as available, including bit-masking with rejection sampling for integers, reservoir sampling, and Fisher–Yates or Knuth shuffle for permutations [1810.10985].

That paper also links RFS-style failures to state-space limitations. A 32-bit PRNG cannot generate all permutations of \(13\) items, and even Mersenne Twister cannot generate all permutations of \(2084\) items [1810.10985]. This suggests that multiply–floor bias is one component of a larger issue: software can simultaneously suffer from quantization artifacts, insufficient generator state, and algorithmic overuse of the available randomness.

## 4. Secure random float sampling in differential privacy

In differential privacy, the phrase Random Float Sampling refers to a different problem. Mechanisms such as Laplace or Gaussian noise addition are often implemented by inverse transform sampling,
\[
x = F^{-1}(U),
\]
with \(U\) a floating-point approximation to a uniform draw on \([0,1)\). Because floating-point representations are finite, the mapping from \(U\) to \(x\) can be inverted, enabling attacks that recover the original value with non-negligible advantage [2107.10138].

The proposed remedy is to make output generation non-injective by combining multiple independent uniform variates and exploiting infinite divisibility. A distribution is infinitely divisible if, for any \(n\), it can be expressed as the sum of \(n\) IID random variables with the original law recovered after aggregation. The paper states that both Laplace and Gaussian are infinitely divisible, and uses this fact to construct secure samplers whose outputs have large preimages in the space of source uniforms [2107.10138].

For Gaussian sampling, one decomposition is
\[
X = \frac{1}{\sqrt{n}} \sum_{i=1}^n N_i \sim N(0,1).
\]
For Laplace sampling, the paper lists several equivalent constructions, including
\[
E_1 - E_2 \sim \Lap(0,1), \qquad E_i \sim \mathrm{Exp}(1),
\]
\[
N_1N_2 - N_3N_4 \sim \Lap(0,1),
\]
and the implementation-oriented expression
\[
\log(1-U_1)\cos(\pi U_2) + \log(1-U_3)\cos(\pi U_4) \sim \Lap(0,1),
\]
with independent \(U_i \sim U(0,1)\) [2107.10138].

The security argument is combinatorial. Standard inverse transform sampling may map one output back to a unique source float from a pool of \(2^{53}\) possibilities. Under the proposed construction, each output can arise from a much larger set of tuples \((U_1,\ldots,U_n)\), with attack cost scaling as \(2^{pn}\); for 53-bit floats and \(n=4\), the paper gives \(2^{128}\) as the attack complexity [2107.10138]. The method is described as generalisable to any infinitely divisible probability distribution and as designed to make side channel attack infeasible because brute-force attacks are inherently exponential in the size of the domain [2107.10138].

This usage of RFS is therefore not a critique of floating-point sampling per se. Rather, it is a reconstruction of float-based sampling so that finite-precision outputs preserve the intended statistical law while becoming computationally non-invertible.

## 5. RFS as continuous position indexing in Transformers

A third usage appears in sequence modeling, where Random Float Sampling is introduced as a position encoding strategy for improving length generalization in Transformers [2602.14050]. Instead of assigning tokens discrete indices from a predefined set, RFS assigns randomly sampled continuous values during training.

For a sequence of length \(n_{\mathrm{tr}}\), training samples
\[
\tilde p_i \sim \mathrm{Uniform}(0,1), \qquad i=1,\ldots,n_{\mathrm{tr}},
\]
and then sorts the sampled floats to obtain
\[
p_1 < p_2 < \cdots < p_{n_{\mathrm{tr}}}.
\]
At inference, for context length \(n_{\mathrm{in}}\), positions are assigned deterministically by even partitioning:
\[
p_i = \frac{2i-1}{2\cdot \max(n_{\mathrm{tr}}, n_{\mathrm{in}})}, \qquad i=1,\ldots,n_{\mathrm{in}}.
\]
All indices are additionally multiplied by a scaling factor \(L\), typically \(1000\) [2602.14050].

These float positions can replace integer indices in absolute sinusoidal encoding, RoPE, and ALiBi. For the sinusoidal case, the paper gives
\[
\mathbf{p}_{i,2k} = \sin\left(\frac{p_i}{10000^{2k/d}}\right), \qquad
\mathbf{p}_{i,2k+1} = \cos\left(\frac{p_i}{10000^{2k/d}}\right).
\]
The stated motivation is to avoid out-of-distribution issues on unseen lengths by exposing the model to diverse indices during training [2602.14050].

The empirical claims are specific. On copy tasks, RFS achieves about \(80\%\) accuracy on sequences twice as long as those seen during training, while the best alternative, NoPE, drops to about \(20\%\) accuracy at only \(1.5\times\) longer lengths [2602.14050]. The paper further reports superior performance on reverse, sort, and other algorithmic sequence tasks, as well as higher out-of-distribution accuracy on zero-shot commonsense reasoning benchmarks including HellaSwag, RACE, ARC-e, ARC-c, OpenBookQA, WinoGrande, and BoolQ [2602.14050].

A notable analytical claim is that simple extension of standard positional encodings leads to a rank increase in the position encoding matrix, contaminating semantic subspaces, whereas RFS keeps the rank of the position matrix stable regardless of length [2602.14050]. This suggests that the method’s benefit is not merely stochastic regularization, but a modification of the geometry of positional representations under extrapolation.

## 6. Hardware-native uniform float sampling

A fourth line of work addresses direct hardware generation of floating-point random samples. “Energy-Efficient Sampling Using Stochastic Magnetic Tunnel Junctions” introduces an algorithm for uniform Float16 sampling using room-temperature s-MTJ devices [2501.00015]. The device-level objective is to generate truly random IEEE Float16 values by assigning Bernoulli probabilities \(p_i\) to the output bits:
\[
C = \{ (b_i, p_i)\mid p_i \in [0,1],\ b_i \in \{b_0,\dots,b_{15}\}\}.
\]

For Float16, the sign and mantissa bits use \(p_i=0.5\), while exponent-bit probabilities are chosen so that the resulting 16-bit word, when interpreted as an IEEE Float16 number, is uniform over the representable range. The desired condition is written as
\[
\lim_{n \to \infty} P(B_n=b \mid C) = D(b), \qquad D = \mathrm{Uniform}(-65504,65504).
\]
After sampling, values can be mapped to an interval \((a,b)\) by
\[
s' = a + \frac{(s + 65504)\cdot (b-a)}{131008},
\]
with the note that the normalization should be performed in Float32 or higher for precision [2501.00015].

The paper reports concrete energy figures. Biasing the exponent bits requires a total of \(20.86\,\mu\mathrm{W}\) at \(1\) MHz sampling, corresponding to \(20.86\) picojoules per sample; reading all s-MTJs costs \(16\) femtojoules per sample. For \(2^{30}\) Float16 samples, total energy is about \(22.42\) mJ for raw sampling and \(23.22\) mJ including normalization arithmetic [2501.00015]. These values are compared against software generators: a minimum factor of \(9721\) improvement relative to Mersenne-Twister and a factor of \(5649\) relative to PCG [2501.00015].

Beyond uniform Float16 sampling, the same work represents an arbitrary 1D distribution as a finite mixture of non-overlapping uniform distributions,
\[
D(x)=f_U(x)=\sum_{i=1}^k w_i f_{U_i}(x),
\]
and uses this representation for sampling, convolution, and prior-likelihood operations. Reported approximation errors are given as KL-divergence \(0.0343 \pm 0.1473\) for convolutions and \(0.0141 \pm 0.1073\) for prior-likelihood multiplication [2501.00015]. In this sense, RFS becomes a hardware-supported primitive for broader probabilistic computation.

## 7. Terminological ambiguity and the unrelated “RFS” of tracking theory

A persistent source of confusion is that the acronym RFS is also entrenched in estimation and tracking as Random Finite Set. In that literature, RFS refers to finite-set-valued stochastic modeling of targets, landmarks, and measurements, not to float sampling. Examples include arithmetic average fusion for unlabeled and labeled RFS densities [2209.10433], heterogeneous unlabeled and labeled RFS filter fusion [2303.09401], and PMBM-based data-association sampling for batch SLAM [2407.11643].

The mathematical vocabulary of that domain—PHD consistency, MB, MBM, PMBM, GLMB, LMB, Gibbs sampling over association hypotheses, and GraphSLAM back-ends—is unrelated to the floating-point sampling problems discussed above. The overlap is purely acronymic. This distinction is operationally important because searches for “RFS” in arXiv-scale corpora will retrieve both families of work, but the technical content is disjoint.

A second misconception concerns the phrase “random float sampling” itself. In statistical computing, the term is often used critically, because constructing integers by multiplying floats and rounding is shown to be biased under finite precision [1809.06520]. In differential privacy and Transformer research, by contrast, random float sampling is a constructive technique: in one case to make floating-point noise generation non-invertible [2107.10138], and in the other to expose a model to diverse continuous position indices for better out-of-distribution behavior [2602.14050]. The same wording therefore spans both a documented failure mode and two affirmative design patterns.

Taken together, the literature presents Random Float Sampling not as a single settled method but as a cluster of domain-specific techniques centered on float-valued randomness. Its technical meaning depends entirely on context: in legacy integer generation it is a source of measurable bias; in privacy mechanisms it is a route to secure non-injective sampling; in sequence modeling it is a continuous indexing scheme for extrapolation; and in hardware sampling it is an energy-efficient way to generate floating-point randomness directly.

Source: https://www.emergentmind.com/topics/random-float-sampling-rfs