Stochastic Rounding: Concepts & Applications

• Stochastic rounding is a method that rounds a real value to its two closest representable numbers based on proximity, ensuring an unbiased expectation.
• It introduces controlled variance to improve error cancellation and stability in iterative computations, especially in low-precision settings.
• It is widely applied in neural network training, PDE solvers, and large-scale matrix computations, requiring careful hardware and PRNG design to balance bias and variance.

Stochastic rounding (SR) is a class of probabilistic rounding schemes in which a real or fixed-point value is rounded to either of its two nearest representable numbers, with probabilities determined by its proximity to each neighbor. Unlike conventional deterministic rounding methods such as round-to-nearest or truncation—where the choice of rounding direction is fixed—SR ensures that, in expectation, the rounding result is identical to the original value. This intrinsic unbiasedness leads to distinctive and beneficial statistical and numerical properties, especially in low-precision and large-scale computations.

1. Formal Definitions and Mechanisms

SR is defined with respect to a real value $x$ and a discretization grid (e.g., a fixed- or floating-point system) %%%%1%%%%. For $x \in [f_j, f_{j+1}] \subset \mathcal{F}$, the canonical SR-nearness scheme prescribes: $$\operatorname{SR}(x) = \begin{cases} f_j & \text{with probability } p_j = \frac{f_{j+1} - x}{f_{j+1} - f_j} \ f_{j+1} & \text{with probability } 1 - p_j = \frac{x - f_j}{f_{j+1} - f_j} \end{cases}$$ This ensures $E[\operatorname{SR}(x)] = x$ (unbiasedness). An alternative, SR-up-or-down, uses $p_j = 0.5$, losing this unbiasedness except at midpoints.

The mechanics can be adapted for fixed-point or floating-point hardware. In fixed-point, with step $\varepsilon = 2^{-p}$, one sets: $$\operatorname{SR}(x) = \begin{cases} \lfloor x \rfloor & \text{if } P \geq \frac{x - \lfloor x \rfloor}{\varepsilon} \ \lfloor x \rfloor + \varepsilon & \text{otherwise} \end{cases}$$ where $P$ is drawn uniformly from $[0,1)$ (Mikaitis, 2020).

2. Unbiasedness, Variance, and Trade-offs

SR's unbiasedness follows directly from the definition. For $x$ and neighbors $a < x < b$: $$E[\operatorname{SR}(x)] = a \frac{b - x}{b - a} + b \frac{x - a}{b - a} = x$$ This property is pivotal in ensuring that, over sequences of operations, systematic error accumulation is eliminated, in contrast to deterministic rounding where bias can build up ($\mathcal{O}(n u)$ for $n$ operations and unit roundoff $u$).

However, SR introduces increased variance: $$V[\operatorname{SR}(x)] = (a - x)^2 \, p_j + (b - x)^2 (1-p_j)$$ Upper bounds on variance are provided in (Xia et al., 2020): when rounding to $n$ fractional bits, the variance per operation obeys $V \leq (1/(2\theta))^2$, with $\theta$ the scaling factor.

Recent work expands SR to allow alternative probability laws, introducing a parameter $p \in [0,1]$ to achieve bias–variance trade-offs. The optimal choice of $p$ can be cast as a multi-objective optimization balancing bias $B(p)$ and variance $V(p)$, leading to variants (D1, D2) with tunable properties (Xia et al., 2020).

3. Error Propagation and Probabilistic Bounds

Unlike round-to-nearest (RN), which accumulates local errors deterministically, SR errors are random, mean-zero, and (often) mean-independent. This property enables the construction of powerful martingale-based concentration inequalities (see also Azuma–Hoeffding), yielding probabilistic error bounds (Arar et al., 2022, Castro et al., 19 Nov 2024, Arar et al., 2022). For a cumulative sum or inner product of $n$ terms: $$\text{RN:} \quad |\text{error}| \lesssim n u \qquad \text{SR:} \quad |\text{error}| \lesssim \sqrt{n} u$$ with high probability. Similar improvements are realized for nonlinear algorithms, such as Horner polynomial evaluation or pairwise summation, and in computations of statistical moments (Arar et al., 2023).

In the context of non-linear variance algorithms (textbook or two-pass), SR was shown to yield error bounds scaling as $O(\sqrt{n}u)$ instead of $O(nu)$ with deterministic rounding (Arar et al., 2023).

4. Hardware Implementations and Limited-Precision Randomness

SR algorithms require a high-quality PRNG. In dedicated hardware (e.g., SpiNNaker2), specialized PRNG cores (e.g., JKISS32) are embedded to enable single-cycle stochastic rounding with saturation and overflow handling for fixed-point multipliers (Mikaitis, 2020, Ali et al., 22 Apr 2024). Hardware-optimized approaches (eager vs. lazy SR) trade PRNG bitwidth, area, and latency.

In practical settings, only a finite number of random bits are available. Limited-precision SR (SR$_{p,r}$) uses $r$ random bits, leading to a bias bounded by $O(n u_{p+r})$ for $n$ operations, with $u_{p+r}$ the step at the higher precision (Arar et al., 6 Aug 2024). Excessive limitation in the bitwidth induces systematic bias, which accumulates, contradicting theoretical unbiasedness; consequently, SR implementations require careful configuration and bias correction strategies (e.g., SRC scheme (Fitzgibbon et al., 29 Apr 2025)).

A summary of bias in few-bit stochastic rounding (FBSR):

Implementation	Bias (infinite-precision)	Recommended Use
SRFF	$-2^{-(N+1)}$	Not for ML, accumulates bias
SRF	$0$ (infinite-precision)	Use when $N \approx D$
SRC	$0$	Bias-corrected, preferred

$N$ is the number of random bits, $D$ excess input bits (Fitzgibbon et al., 29 Apr 2025).

5. Applications: Neural Networks, ODE/PDE Solvers, and Large-Scale Training

Neural Network Training: SR is widely adopted in low-precision neural networks, preventing vanishing gradient issues caused by zeroing small updates (Xia et al., 2021, Xia et al., 2022, Xia et al., 2023). Experiments consistently show that unbiased SR preserves trainability under aggressive quantization (down to ternary weights) and achieves accuracy nearly comparable to high-precision baselines, outperforming deterministic rounding (Zhao et al., 6 Dec 2024). Novel DQTs even eliminate the need for full-precision weights during training, slashing memory use (Zhao et al., 6 Dec 2024).

Gradient Descent Convergence: While SR guarantees zero expectation in rounding error, introducing a controlled bias (e.g., in "signed-SR") can further accelerate convergence for PL or convex objectives by nudging the update direction in coordinate descent (Xia et al., 2022, Xia et al., 2023).

PDEs and Climate Models: In low precision PDE solvers, RN causes stagnation and $O(u/\Delta t)$ global error growth; SR produces zero-mean, "decorrelating" errors scaling as $O(u/\Delta t^{1/4})$ in 1D and $O(u)$ in higher dimensions, avoiding stagnation and maintaining trajectory fidelity over long integrations. This robustness extends to next-generation climate models where half-precision with SR can maintain climatic mean and variability within physically negligible bounds over 100 year integrations (Croci et al., 2020, Paxton et al., 2021, 2207.14598).

Matrix Regularization: In large-scale ML, SR-nearness not only mitigates bias but also acts as an implicit random regularizer for tall-and-thin matrices, ensuring full column rank post-quantization even under severe ill-conditioning. This phenomenon quantitatively enhances the minimum singular value, scaling with $\sqrt{n}$, supporting numerical stability (Dexter et al., 18 Mar 2024).

6. Complexity Analysis and Theoretical Directions

SR, via its intrinsic error cancellation, allows for more realistic average-case complexity analysis analogous to "smoothed complexity." For an algorithm with $n$ rounding steps: $$\text{Under SR:}\;\; \max |S_n - \mathbb{E}S_n| \lesssim \sqrt{n} u$$ SR can thus be treated as a noise model in algorithm analysis, providing a foundational tool for probabilistic error and complexity studies (Drineas et al., 14 Oct 2024).

Future directions include:

• Improving error bounds using non-asymptotic random matrix theory,
• Designing optimized PRNGs for hardware implementations,
• Quantifying the stability of algorithms under few-bit SR,
• Extending analysis to nonlinear and adaptive algorithms (Doob–Meyer decomposition) (Castro et al., 19 Nov 2024).

7. Observed Limitations and Open Problems

• Random Bitwidth: Limited random bits induce bias, which can be significant for large-scale computations if not mitigated (see SRC bias-corrected methods) (Fitzgibbon et al., 29 Apr 2025).
• Variance Control: Standard SR increases variance, which may destabilize some learning algorithms or iterative methods. Tuning or "designer" SR schemes (multi-objective bias–variance optimization) may ameliorate these effects in sensitive domains (Xia et al., 2020).
• Hardware Constraints: Efficient, synchronous distribution of PRNG streams is critical in distributed or accelerator architectures, otherwise deterministic artifacts or model drift may emerge (Ozkara et al., 27 Feb 2025).

---

Stochastic rounding, through unbiased probabilistic selection between adjacent representable values, enables error cancellation, robust numerical behavior in low-precision settings, resilient neural network and PDE/ODE solvers, and regularization effects in large-scale matrix computations. While its theoretical advantages are now well-understood—probabilistic error bounds of $O(\sqrt{n}u)$, preservation of meaningful low-level updates, and implicit regularization—practical deployment mandates careful consideration of PRNG sources, bias introduction when using few random bits, and hardware/software co-design for high-performance implementations. The methodology continues to impact and reshape algorithmic design paradigms across numerical computation, scientific simulation, and machine learning at scale.