---
title: Block-Floating and Posit Number Systems
url: https://www.emergentmind.com/topics/block-floating-and-posit
type: topic
---

# Block-Floating and Posit Number Systems

Block-floating point and posit number systems are alternative numerical formats developed to address the challenges posed by traditional IEEE-754 floating-point arithmetic, especially for reduced-precision applications. These formats play a significant role in efficient deep neural network (DNN) training, memory and bandwidth reduction, and custom hardware acceleration. The posit system, as a type-3 universal number (Unum) format, replaces fixed-width exponent representation with a dynamic regime field, enabling superior trade-offs between precision and dynamic range, particularly near unity. Block-floating formats, by contrast, share an exponent among all elements of a block, offering lower storage overhead but coarser granularity. Recent research has demonstrated that, with suitable methodology, posits can enable full-scale DNN training with 16-bit precision and no accuracy loss, while also providing substantial power and area advantages when implemented in hardware [1909.03831].

## 1. Structure and Decoding of the Posit Number System

An $(n, es)$ posit is composed sequentially of a sign bit $s$, a variable-length regime field $r$ encoded in signed unary, $es$ exponent bits $e$, and remaining fraction bits $f$. The regime directly encodes a signed integer $k$:

- A run of $k+1$ ones followed by a zero denotes regime value $+k$.
- A run of $k$ zeros followed by a one denotes regime value $-k$.

The useed, defined as $\mathrm{useed} \triangleq 2^{2^{es}}$, specifies regime scaling and ultimately dynamic range:
- Positive maximal value $\approx \mathrm{useed}^{n-2}$
- Positive minimal value $\approx \mathrm{useed}^{2-n}$

The value $x$ represented by a posit bitstring $p$ is decoded as:
$$
x =
  \begin{cases}
    0,                   & p = 00\ldots0 \\
    \pm\infty,           & p = 10\ldots0 \\
    (-1)^s\ \mathrm{useed}^k 2^e (1 + f), & \text{otherwise}
  \end{cases}
$$

The variable-length regime enables a dynamic allocation of bits between precision and range. Near unity, more bits are devoted to fraction (higher precision); for large $|x|$, more bits are allocated to range.

## 2. Comparison to IEEE-754 Floating-Point and Block-Floating Formats

IEEE-754 floating-point uses fixed-width exponent and mantissa fields per value, resulting in uniform precision within an exponential segment, but consistent bit allocation regardless of magnitude. By contrast, block-floating formats save storage by sharing the exponent among a block of values, reducing per-value overhead at the expense of per-value range control. The posit format differs fundamentally:

- No explicit encoding for NaN or subnormal values; only explicit $\pm\infty$ and zero.
- Tapered precision: maximum representational accuracy near $|x|=1$, reducing at extreme magnitudes.
- Unique regime field per value confers individual dynamic scaling.
- Adapts granularity on a per-value basis, in contrast to block-floating, where all block elements share a single exponent.

## 3. Reduced-Precision DNN Training: Challenges and Methodology

Reduced-precision DNN training with posits confronts several well-characterized obstacles:

- In early epochs, weight and activation distributions (notably in BatchNorm layers) can be highly non-stationary; low-precision regimes can impair convergence.
- n-bit representations do not guarantee alignment of peak precision with data distribution modes.
- Tensors (activations, gradients, errors) have disparate dynamic ranges; using a single $es$ parameter is insufficient.

To address these challenges, the following approach is taken [1909.03831]:

1. **Warm-Up Training:** The first 1–5 epochs are conducted in FP32 to permit stabilization of layer statistics. Distribution information is collected post warm-up using log₂ domain centrality.
2. **Layer-Wise Scaling:** For each tensor $X$, compute a scale factor $S_f = 2^{\mathrm{center} + \sigma}$, where $\mathrm{center} = \mathrm{round}\left(\mathrm{mean}( \log_2|X| )\right)$ and $\sigma=2$. Data is normalized to leverage the highest fraction-bit density ($X / S_f$), quantized to posit, then rescaled: $X_p = P_{n,es}( X / S_f ) \cdot S_f$.
3. **Adaptive $es$ per Tensor:** $es=1$ for weights/activations (favoring fraction bits), $es=2$ for gradients/errors (favoring dynamic range).

The posit conversion operator $P_{n,es}(x)$ involves: clipping input to $[\mathrm{useed}^{2-n}, \mathrm{useed}^{n-2}]$, extracting sign and exponent, arithmetic for regime and local exponent, bit-allocation for exponent/fraction, followed by quantization and repacking procedures.

## 4. Empirical Results: DNN Training and Model Efficiency

On the ImageNet classification task with ResNet-18, Lu et al. implement the methodology as follows:

- **Optimizer:** SGD with momentum 0.9
- **Learning Rate:** Initial 0.1, divided by 10 every 30 epochs
- **Epochs:** Total 90, batch size 512
- **Format Usage:** Forward/weight updates in $(16,1)$; backward pass in $(16,2)$
- **Warm-Up:** First 5 epochs in FP32

The outcome is a Top-1 accuracy of 71.09%, matching the FP32 baseline of 71.02%. Model size is halved (16-bit vs 32-bit), off-chip memory bandwidth is also halved, and data movement/computation yields approximately 2× energy savings due to reduced precision. This demonstrates that, under the stated methodology, 16-bit posit arithmetic can support large-scale DNN training with no accuracy penalty [1909.03831].

## 5. Energy-Efficient Posit Hardware Design

A hardware multiply-and-accumulate (MAC) architecture leveraging posit offers the following block structure: [Posit Decode] → [FP Multiply-Accumulate] → [Posit Encode].

- **Decoder:** Converts $(n,es)$ posit into sign, regime (decoded via leading-zero/one detectors), and effective exponent+mantissa. An optimization eliminates explicit adders in the critical path by duplicating barrel shifters, with empirical speedup of 15–30%.
- **Encoder:** Reconstitutes a $2n$-bit representation post-MAC, fills regime bits, and right-shifts by $k$ or $k+1$. The same split-barrel shifter trick is used, with speedups of 25–35%.
- **Resource Utilization and Power:** In TSMC28nm at 750 MHz:
  - FP32 MAC: 2.52 mW, 4,322 $\mu$m²
  - Posit(8,1): 0.45 mW (↓82%), 1,208 $\mu$m² (↓72%)
  - Posit(16,1): 1.77 mW (↓30%), 4,079 $\mu$m² (↓6%)
  - Posit(16,2): 1.60 mW (↓37%), 3,897 $\mu$m² (↓10%)

Decode and encode steps together account for approximately 40% of pipeline latency, substantially reduced by listed optimizations.

## 6. Implementation Considerations and Limitations

Deploying posit arithmetic in DNN accelerators entails:

- **Custom Encode/Decode Pipelines:** These add some depth compared to IEEE-754, but pipeline/area overhead is minor post-optimization, with potential net gains.
- **Necessary Remedies for Training Stability:** Use of per-tensor scaling factors and an initial FP32 warm-up is essential.
- **Empirical Scope:** Only select $(n,es)$ configurations have been validated at scale; extreme low-bit precision (e.g., 8-bit) for large models remains an open area of research.
- **System-Level Benefits:** Memory and bandwidth requirements reduced by 2–4×; MAC power and area cut by 20–80%. Per-layer autotuning of scaling and $es$ can be compiler-integrated.

## 7. Broader Impact on DNN Accelerator Design

The dynamic regime field in the posit format enables a better precision–range compromise for DNN training than fixed-structure formats. With a combination of initial FP32 warm-up, per-layer log-domain scaling, and per-tensor $es$, full-scale ImageNet training at 16-bit posit is attainable without loss of accuracy, as established by Lu et al. The described hardware architecture—using optimized encode/decode logic and a conventional FP MAC—yields end-to-end units that are more area- and energy-efficient than IEEE-754 implementations, making posit arithmetic a viable candidate for next-generation low-power DNN training ASICs [1909.03831].

Source: https://www.emergentmind.com/topics/block-floating-and-posit