---
title: Quantized Random Rounding in Low-Precision Arithmetic
url: https://www.emergentmind.com/topics/quantized-random-rounding
type: topic
---

# Quantized Random Rounding in Low-Precision Arithmetic

Quantized random rounding, often referred to as stochastic rounding (SR) with explicit discretization of both the randomization and quantization grids, is a probabilistic rounding scheme that targets unbiasedness and superior error scaling for low-precision arithmetic, particularly in the context of machine learning, numerical linear algebra, and scientific computing. SR and its quantized variants are distinguished by their ability to deliver expectation-preserving quantization even at low bit-widths, facilitate rigorous error control, and bridge algorithmic and hardware requirements for efficient deployment.

## 1. Formal Definitions and Mathematical Foundations

In quantized random rounding, the real-valued input $x$ is mapped to a discrete quantization grid $\mathbb{F}$ (floating- or fixed-point). For uniform quantization step $\Delta>0$, let $\lfloor x \rfloor_\Delta = \Delta \cdot \lfloor x/\Delta \rfloor$ and $\lceil x \rceil_\Delta = \Delta \cdot \lceil x/\Delta \rceil$. The canonical stochastic rounding operator selects either the lower or upper neighbor with probability proportional to their proximity:
\[
\operatorname{SR}_\Delta(x) = \begin{cases}
\lfloor x \rfloor_\Delta \quad & \text{with probability } 1 - \frac{x - \lfloor x \rfloor_\Delta}{\Delta} \\
\lceil x \rceil_\Delta         & \text{with probability }     \frac{x - \lfloor x \rfloor_\Delta}{\Delta}
\end{cases}
\]
so that $\mathbb{E}[\operatorname{SR}_\Delta(x)] = x$ for $x\in\mathbb{R}$ [2511.00874, 2603.06060].

*Limited-precision* or *quantized* SR, denoted $SR_p^{(r)}$, further quantizes randomization: $x$ is first rounded to $p+r$ bits, the discretized fractional part is extracted, and the random decision is made with $r$ random bits to approximate the ideal probability. This results in a rounding operator
\[
SR_p^{(r)}(x) \coloneqq SR_p(\text{round}_{p+r}(x))
\]
where $SR_p$ acts on the $p$-bit representation and the stochastic decision is made from a uniformly random integer $R \in \{0, ..., 2^r - 1\}$ compared to the discretized threshold. For architectures, typical $r$ values are $8$–$24$, balancing statistical fidelity and random number generation overhead [2603.06060].

## 2. Statistical Properties and Error Bounds

The key statistical property of quantized random rounding is expectation-preservation:
\[
\mathbb{E}[\operatorname{SR}_\Delta(x)] = x,
\]
ensuring that rounding is unbiased in the mean. When used in summation and reduction kernels, the error $\epsilon_k$ at step $k$ forms a bounded martingale difference sequence, leading to Azuma–Hoeffding and Chebyshev-type high-probability bounds:
\[
|S_n - \sum_{k=1}^n x_k| \leq u M \sqrt{2 n \log(2/\delta)} \qquad \text{with prob. } 1-\delta,
\]
where $u$ is the unit roundoff ($\sim2^{-p}$), $M$ bounds partial sum magnitudes, and $n$ is the accumulation length [2603.06060].

This $O(u\sqrt{n})$ scaling (as opposed to $O(nu)$ for round-to-nearest) is corroborated for arbitrary discrete grids and higher moments:
- For a general quantizer $\mathbb{F}$ and $\operatorname{rd}(X)$ as the stochastically rounded version, the $k$-th moment deviation satisfies
  \[
  |\mathbb{E}[X^k] - \mathbb{E}[\operatorname{rd}(X)^k]| \leq C\epsilon^2
  \]
  for a universal constant $C$ depending on the quantization envelope and the probability law of $X$ [2007.11041].
- Variance and second-moment loss can be rigorously bounded using variance-informed probabilistic rounding models, yielding problem-size constants $\hat{\gamma}_n$ scaling as $O(\sqrt{n})$ and empirically validated to outperform classical deterministic error growth by several orders of magnitude for large $n$ [2404.12556].

In practical scenarios with randomly quantized discretization and $r$-bit precision in the stochastic decision, the bias is $O(2^{-r})$, negligible in high-$r$ regimes, but degrading to standard biased rounding for very small $r$ [2603.06060].

## 3. Theoretical Analysis in SGD and Optimization

Quantized random rounding is analytically advantageous in mini-batch SGD and low-precision neural training:
- **Variance Decomposition:** In SR-quantized SGD, gradient estimator variance has two parts—sampling variance ($\sigma_S^2$) and quantization variance ($\sigma_Q^2\sim 2^{-2B}$). The total variance per entry decays as $1/b$ with batch size $b$:
  \[
  T^Q_{ij} \leq \frac{C}{b} 2^{-2B}
  \]
  so larger batches linearly suppress quantization noise [2511.00874].
- **Unbiasedness and Convergence:** SR maintains unbiasedness in the stochastic gradient, ensuring that SGD recovers standard convergence rates (for smooth, bounded-variance objectives). The deterministic round-to-nearest (RTN) is systematically biased, imparting an irreducible error term in SGD convergence, whereas the bias from SR vanishes as batch size increases [2511.00874].
- **Precision–Batch Trade-Off:** Reducing mantissa bit-width $B$ by $\Delta B$ quadruples ($4^{\Delta B}$) the quantization variance, which can be offset by increasing batch size by the same factor. This relationship is pivotal for low-precision training on resource-constrained devices.

Frameworks such as LOTION apply SR as a smoothing primitive to the quantized loss landscape, yielding a differentiable surrogate and preserving global minima of the original quantized objective. This allows rigorous application of stochastic optimization (SGD/Adam) without off-manifold bias, in contrast to Straight-Through Estimators, which are generally biased and lack convergence guarantees for non-convex, discontinuous objectives [2510.08757].

## 4. Moment Preservation and Limiting Behavior

Quantized random rounding exhibits precise control over moment distortion:
- **First moment:** Always preserved (unbiasedness).
- **Higher raw moments:** The error in the $k$-th moment is $O(\epsilon^2)$, and the absolute moment error is $O(\epsilon)$. The constants linking quantization error and higher moments may grow rapidly with $k$, implying that deterministic rounding may be preferable where faithful preservation of variance, skewness, or kurtosis is critical [2007.11041].
- **Sheppard-type Corrections:** For uniformly spaced quantization, stochastic rounding uniquely avoids the systematic bias (Sheppard correction) present in deterministic schemes; for grid, lattice, or block-form quantizers these corrections can be written explicitly [2504.06601].

Moment analysis extends to discrete random variables, where randomized rounding (probability matched to the fractional part) achieves exact expectation preservation and, for many practical cases, minimal MSE among rounding schemes.

## 5. Practical Implementation, Hardware, and Guidelines

Quantized random rounding is efficient to implement and widely adopted across hardware:
- **Random Bit Resource:** For sequence lengths $n$ (e.g., in reductions), $r \sim \lceil 0.5 \log_2 n \rceil$ suffices for negligible bias. PRNGs such as LFSRs can produce random bits on-chip with very little area overhead; industry platforms typically set $r = 8$–$24$ depending on the operation and performance target [2603.06060].
- **Placement in Kernels:** Best practice is to employ SR when casting high-precision intermediate results to lower-precision outputs (e.g., during accumulator flushes or gradient write-back). For inference, deterministic modes may be preferred for reproducibility, while for training and accumulation, SR gives optimal error scaling [2603.06060].
- **Hardware Integration:** Modern AI/ML accelerators (NVIDIA, AMD, Graphcore) provide hardware support for limited-precision random rounding, enabling aggressive quantization (INT4/INT8) without stagnation or bias-driven accuracy loss.
- **Microbatching:** In edge training (LLMs and vision models), smaller mantissa can be compensated by increasing the microbatch size; this unlocks high compute density without sacrificing convergence or statistical fidelity [2511.00874].

## 6. Applications and Empirical Validation

Quantized random rounding is validated in diverse settings:
- **Deep Learning:** SR enables stable low-precision training, especially at INT4/INT8 formats. Experiments confirm that with correct batch/precision scaling, accuracy typical of full-precision models is recoverable (e.g., BERT-GLUE/LLM finetuning) [2511.00874].
- **Numerical Linear Algebra:** Summation, dot-product, matrix-vector multiplication, and LU solutions using SR exhibit rounding error growth $O(\sqrt{n})$, several orders smaller than classical error bounds. Calibration on hardware matches predicted constants by variance-driven bounds [2404.12556].
- **Scientific Modeling:** Climate simulations and long-time evolution tasks maintain statistical properties when using SR, as opposed to loss of fidelity with deterministic round-to-nearest; this is crucial for modeling reliability in predictive and chaotic systems [2603.06060].

Empirical validation extends to statistical kernels, PDE solvers, and signal processing, with moment error always consistent with theoretical bounds, and in large-scale reductions SR outperforms all deterministic modes in bias and variance [2504.06601, 2007.11041].

## 7. Comparison to Alternative Rounding Schemes

A comparative summary:

| Rounding mode        | Unbiased?        | Error growth      | Hardware cost      |
|----------------------|------------------|-------------------|--------------------|
| Round-to-nearest     | No               | $O(nu)$           | None               |
| Stochastic (quantized) | Yes ($r$ large) | $O(u\sqrt{n})$    | $r$-bit PRNG, LFSR |
| Floor/Ceil           | No; one-sided    | $O(nu)$, biased   | None               |
| Data-dependent (DiscQuant) | Yes (on calibration set) | $O(\sqrt{n})$, low discrepancy | PRNG, calibration |
| Classical randomized | Yes (coordinate-wise) | $O(\sqrt{n})$ | PRNG              |

Quantized random rounding maintains bias-free summation properties; deterministic alternatives accumulate systematic error (bias), degrade for deep reductions, and may stagnate when adding sub-threshold summands (causing loss of signal in neural gradients and physical simulations) [2603.06060, 2511.00874].

## References

- "Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding" [2511.00874]
- "Limited-Precision Stochastic Rounding" [2603.06060]
- "Non-asymptotic moment bounds for random variables rounded to non-uniformly spaced sets" [2007.11041]
- "Exploiting Higher-Order Statistics for Robust Probabilistic Rounding Error Analysis" [2404.12556]
- "Rounding of discrete variables" [2504.06601]
- "LOTION: Smoothing the Optimization Landscape for Quantized Training" [2510.08757]
- "DiscQuant: A Quantization Method for Neural Networks Inspired by Discrepancy Theory" [2501.06417]

Source: https://www.emergentmind.com/topics/quantized-random-rounding