Block-Floating and Posit Number Systems

• Block-floating and posit number systems are alternative formats that address IEEE-754 limitations by dynamically allocating bits for improved precision and range.
• They employ techniques such as FP32 warm-up and per-tensor scaling to enable full-scale DNN training at reduced precision without accuracy loss.
• Optimized hardware designs with specialized encode/decode pipelines deliver significant energy, area, and speed improvements over conventional FP32 implementations.

Block-floating point and posit number systems are alternative numerical formats developed to address the challenges posed by traditional IEEE-754 floating-point arithmetic, especially for reduced-precision applications. These formats play a significant role in efficient deep neural network (DNN) training, memory and bandwidth reduction, and custom hardware acceleration. The posit system, as a type-3 universal number (Unum) format, replaces fixed-width exponent representation with a dynamic regime field, enabling superior trade-offs between precision and dynamic range, particularly near unity. Block-floating formats, by contrast, share an exponent among all elements of a block, offering lower storage overhead but coarser granularity. Recent research has demonstrated that, with suitable methodology, posits can enable full-scale DNN training with 16-bit precision and no accuracy loss, while also providing substantial power and area advantages when implemented in hardware (Lu et al., 2019).

1. Structure and Decoding of the Posit Number System

An $(n, es)$ posit is composed sequentially of a sign bit $s$, a variable-length regime field $r$ encoded in signed unary, $es$ exponent bits $e$, and remaining fraction bits $f$. The regime directly encodes a signed integer $k$:

• A run of $k+1$ ones followed by a zero denotes regime value $+k$.
• A run of $k$ zeros followed by a one denotes regime value $-k$.

The useed, defined as $\mathrm{useed} \triangleq 2^{2^{es}}$, specifies regime scaling and ultimately dynamic range:

• Positive maximal value $\approx \mathrm{useed}^{n-2}$
• Positive minimal value $\approx \mathrm{useed}^{2-n}$

The value $x$ represented by a posit bitstring $p$ is decoded as:

$$x = \begin{cases} 0, & p = 00\ldots0 \ \pm\infty, & p = 10\ldots0 \ (-1)^s\ \mathrm{useed}^k 2^e (1 + f), & \text{otherwise} \end{cases}$$

The variable-length regime enables a dynamic allocation of bits between precision and range. Near unity, more bits are devoted to fraction (higher precision); for large $|x|$, more bits are allocated to range.

2. Comparison to IEEE-754 Floating-Point and Block-Floating Formats

IEEE-754 floating-point uses fixed-width exponent and mantissa fields per value, resulting in uniform precision within an exponential segment, but consistent bit allocation regardless of magnitude. By contrast, block-floating formats save storage by sharing the exponent among a block of values, reducing per-value overhead at the expense of per-value range control. The posit format differs fundamentally:

• No explicit encoding for NaN or subnormal values; only explicit $\pm\infty$ and zero.
• Tapered precision: maximum representational accuracy near $|x|=1$, reducing at extreme magnitudes.
• Unique regime field per value confers individual dynamic scaling.
• Adapts granularity on a per-value basis, in contrast to block-floating, where all block elements share a single exponent.

3. Reduced-Precision DNN Training: Challenges and Methodology

Reduced-precision DNN training with posits confronts several well-characterized obstacles:

• In early epochs, weight and activation distributions (notably in BatchNorm layers) can be highly non-stationary; low-precision regimes can impair convergence.
• n-bit representations do not guarantee alignment of peak precision with data distribution modes.
• Tensors (activations, gradients, errors) have disparate dynamic ranges; using a single $es$ parameter is insufficient.

To address these challenges, the following approach is taken (Lu et al., 2019):

1. Warm-Up Training: The first 1–5 epochs are conducted in FP32 to permit stabilization of layer statistics. Distribution information is collected post warm-up using log₂ domain centrality.
2. Layer-Wise Scaling: For each tensor $X$, compute a scale factor $S_f = 2^{\mathrm{center} + \sigma}$, where $$\mathrm{center} = \mathrm{round}\left(\mathrm{mean}( \log_2|X| )\right)$$ and $\sigma=2$. Data is normalized to leverage the highest fraction-bit density ($X / S_f$), quantized to posit, then rescaled: $X_p = P_{n,es}( X / S_f ) \cdot S_f$.
3. Adaptive $es$ per Tensor: $es=1$ for weights/activations (favoring fraction bits), $es=2$ for gradients/errors (favoring dynamic range).

The posit conversion operator $P_{n,es}(x)$ involves: clipping input to $[\mathrm{useed}^{2-n}, \mathrm{useed}^{n-2}]$, extracting sign and exponent, arithmetic for regime and local exponent, bit-allocation for exponent/fraction, followed by quantization and repacking procedures.

4. Empirical Results: DNN Training and Model Efficiency

On the ImageNet classification task with ResNet-18, Lu et al. implement the methodology as follows:

• Optimizer: SGD with momentum 0.9
• Learning Rate: Initial 0.1, divided by 10 every 30 epochs
• Epochs: Total 90, batch size 512
• Format Usage: Forward/weight updates in $(16,1)$; backward pass in $(16,2)$
• Warm-Up: First 5 epochs in FP32

The outcome is a Top-1 accuracy of 71.09%, matching the FP32 baseline of 71.02%. Model size is halved (16-bit vs 32-bit), off-chip memory bandwidth is also halved, and data movement/computation yields approximately 2× energy savings due to reduced precision. This demonstrates that, under the stated methodology, 16-bit posit arithmetic can support large-scale DNN training with no accuracy penalty (Lu et al., 2019).

5. Energy-Efficient Posit Hardware Design

A hardware multiply-and-accumulate (MAC) architecture leveraging posit offers the following block structure: [Posit Decode] → [FP Multiply-Accumulate] → [Posit Encode].

• Decoder: Converts $(n,es)$ posit into sign, regime (decoded via leading-zero/one detectors), and effective exponent+mantissa. An optimization eliminates explicit adders in the critical path by duplicating barrel shifters, with empirical speedup of 15–30%.
• Encoder: Reconstitutes a $2n$-bit representation post-MAC, fills regime bits, and right-shifts by $k$ or $k+1$. The same split-barrel shifter trick is used, with speedups of 25–35%.
• Resource Utilization and Power: In TSMC28nm at 750 MHz:
  • FP32 MAC: 2.52 mW, 4,322 $\mu$m²
  • Posit(8,1): 0.45 mW (↓82%), 1,208 $\mu$m² (↓72%)
  • Posit(16,1): 1.77 mW (↓30%), 4,079 $\mu$m² (↓6%)
  • Posit(16,2): 1.60 mW (↓37%), 3,897 $\mu$m² (↓10%)

Decode and encode steps together account for approximately 40% of pipeline latency, substantially reduced by listed optimizations.

6. Implementation Considerations and Limitations

Deploying posit arithmetic in DNN accelerators entails:

• Custom Encode/Decode Pipelines: These add some depth compared to IEEE-754, but pipeline/area overhead is minor post-optimization, with potential net gains.
• Necessary Remedies for Training Stability: Use of per-tensor scaling factors and an initial FP32 warm-up is essential.
• Empirical Scope: Only select $(n,es)$ configurations have been validated at scale; extreme low-bit precision (e.g., 8-bit) for large models remains an open area of research.
• System-Level Benefits: Memory and bandwidth requirements reduced by 2–4×; MAC power and area cut by 20–80%. Per-layer autotuning of scaling and $es$ can be compiler-integrated.

7. Broader Impact on DNN Accelerator Design

The dynamic regime field in the posit format enables a better precision–range compromise for DNN training than fixed-structure formats. With a combination of initial FP32 warm-up, per-layer log-domain scaling, and per-tensor $es$, full-scale ImageNet training at 16-bit posit is attainable without loss of accuracy, as established by Lu et al. The described hardware architecture—using optimized encode/decode logic and a conventional FP MAC—yields end-to-end units that are more area- and energy-efficient than IEEE-754 implementations, making posit arithmetic a viable candidate for next-generation low-power DNN training ASICs (Lu et al., 2019).