---
title: E6M2 Floating-Point Scale Overview
url: https://www.emergentmind.com/topics/e6m2-floating-point-scale
type: topic
---

# E6M2 Floating-Point Scale Overview

The E6M2 floating-point scale refers to a numerical format and associated quantization strategy tailored for efficient low-precision training and inference in large language models (LLMs). E6M2 denotes an 8- or 9-bit floating-point representation with 1 sign bit, 6 exponent bits, and 2 mantissa bits, offering a wide dynamic range with limited fractional resolution. E6M2 is employed in both standard floating-point quantization schemes and advanced block-wise quantization frameworks such as microscaling and $\mu$nit scaling, providing substantial savings in memory bandwidth and computational resources—particularly in contexts where traditional 16- or 32-bit formats are prohibitive.

## 1. Format Specification and Theoretical Properties

E6M2 allocates 1 bit for the sign, 6 bits for the exponent, and 2 bits for the mantissa. With a 6-bit exponent, the exponent bias is $b = 2^{6-1} - 1 = 31$. This yields the following representable ranges [2510.01863, 2502.05967]:

- **Smallest positive normal:** $2^{-30}$
- **Largest finite normal:** $(2 - 2^{-2}) \cdot 2^{31} = 1.75 \cdot 2^{31}$
- **Smallest positive subnormal:** $2^{-32}$
- **Unit roundoff (machine $\epsilon$):** $2^{-3} = 1/8$
- **Relative error (max):** $\lesssim 0.125$

The format's dynamic range is sufficient to accommodate activations, gradients, and weights encountered in LLM training under proper scaling.

## 2. Quantization Scaling Laws and Loss Analysis

The unified scaling law for floating-point quantization, particularly relevant for E6M2, predicts LLM validation loss after training subjected to FP quantization as a function of model size $N$, data size $D$, exponent bits $E$, mantissa bits $M$, and block size $B$ [2501.02423]:

\[
L(N,D,E,M,B) = \frac{n}{N^\alpha} + \frac{d}{D^\beta} + \epsilon + \frac{D^\beta}{N^\alpha} \frac{\log_2 B}{\gamma (E+0.5)^\delta (M+0.5)^\nu}
\]

For E6M2, $(E, M) = (6, 2)$ and typical block size $B=128$ yield a slightly elevated loss relative to the optimal E4M3 layout (which is closer to the empirical optimum for 8-9 bit formats), but they remain within cost-performance sweet spots, especially for regimes with stringent compute budgets. Exponent bits have approximately 8% more impact on validation loss per bit than mantissa bits ($\delta \approx 3.19$ vs. $\nu \approx 2.95$). For fixed precision $P = E + M + 1$, the bit allocation optimizing loss is $E_{\rm opt} = \frac{\delta}{\delta + \nu}P - 0.5$, $M_{\rm opt} = \frac{\nu}{\delta + \nu}P - 0.5$ [2501.02423].

## 3. Microscaling and Shared-Exponent Quantization

Microscaling introduces blockwise quantization with a shared scale $S$ for each block of $N$ values. For E6M2, a block's scale is determined by the maximum absolute value within the block, scaled so the largest value maps to the largest normal representable value $f_{\max}$:

\[
S = \frac{f_{\max}}{\max_i |x_i|}
\]
where $f_{\max}(6,2) = 1.75 \cdot 2^{31}$ [2510.01863, 2502.05967].

Each value $x_i$ in the block is then quantized as $q_i = \text{round}(x_i/S)$, and dequantization is performed as $\hat{x}_i = q_i \cdot S$. Handling of subnormals and overflow follows IEEE-inspired conventions: values below $2^{-32}$ round to zero; values above $f_{\max}$ saturate to $\pm\infty$.

This methodology allows E6M2 to drastically reduce memory footprint: for block size $B=32$, E6M2 microscaling requires $8.25$ bits per value (including scale overhead), yielding approximately $48\%$ memory reduction vs. bf16 [2510.01863].

## 4. Training Implementation and Scaling Strategies

$\mu$nit scaling enables static scaling based on a calibration or initialization scan per layer [2502.05967]. For an arbitrary tensor $X$ and desired E6M2 format, compute:
- $X_{\max,i} = \max |X|$ for relevant tensor $i$
- $S_i = 1.75 \cdot 2^{31} / X_{\max,i}$

The scaled and clipped values $\tilde{X} = \text{clip}(S_i X, -f_{\max}, +f_{\max})$ are then cast to E6M2, replaced in the forward/backward passes, and dequantized as needed. For LLMs using unit scaling in initialization (e.g., $1/\sqrt{\text{fan-in}}$), the distribution of $X$ is stationary enough that $S_i$ may require only infrequent updates.

Backpropagation commonly employs a high-precision master copy of weights, with gradients optionally stored in E6M2; forward and backward computation employ E6M2 or mixed-precision accumulators. Use of an exact or high-precision accumulator per block is necessary to minimize quantization error and avoid accumulation overflow [2510.01863].

## 5. Empirical Observations and Practical Trade-offs

Practical use of E6M2 in language models such as GPT-2 yields key trade-offs between range, memory, and accuracy [2510.01863]:
- For block size $B=32$, E6M2 microscaling achieves $\sim48\%$ reduction in memory usage vs. bf16 at a cost of a modest relative error increase (+0.112).
- Full use of the dynamic range requires careful tuning or modest amplification of $S$ when data distribution is tightly concentrated near zero.
- Configurations mixing Microscaling E6M2 with master-copy bf16 storage for sensitive parameters (e.g., probabilities) provide stable training with only marginal accuracy degradation.
- Lower mantissa bits induce coarser quantization, necessitating higher-precision accumulation or mixed-precision staging for dot-product computations to avoid overflow and excessive quantization loss.

Table: Example Empirical Results for GPT-2 (Tiny Shakespeare, 124M parameters, 100 steps, [2510.01863])

| Configuration                        | Memory Reduction vs. bf16 | Rel. Error Increase |
|---------------------------------------|--------------------------|--------------------|
| bf16 (all)                           | Baseline                 | +0.112             |
| E6M2 Microscaling (B=32) + bf16 master| 48%                      | +0.112             |
| E4M3 Microscaling (weights+grads)     | >50%                     | +0.995             |

## 6. Critical Data Size and Limitations

A key finding for low-precision training is the existence of a critical data size $D_{\rm crit}$ beyond which further increases in token count can degrade model performance due to excessive quantization-induced information loss. For E6M2, $D_{\rm crit}$ is parametrized by model size $N$, block size $B$, and precision allocation, and is computed as [2501.02423]:

\[
D_{\rm crit} = \left[\frac{d \gamma N^\alpha (E+0.5)^\delta (M+0.5)^\nu}{\log_2 B}\right]^{1/(2\beta)}
\]

For standard LLM regimes, E6M2 raises $D_{\rm crit}$ substantially above E4M3 or E2M1, making it better suited for longer training runs and larger datasets under fixed precision. Surpassing $D_{\rm crit}$ leads to increasing loss, limiting the utility of over-training under aggressively quantized settings.

## 7. Cost-Optimal Precision and Future Directions

For compute- and memory-bound regimes, the family of 4–8 bit floating-point formats remains cost-optimal according to floating-point scaling laws. E6M2 (8 or 9 bits) lies near the upper end of this tradeoff, offering increased range at the expense of reduced fractional accuracy compared to E4M3 or E3M4. Its principal application is in blockwise-quantized LLMs and transformers where large dynamic range is demanded by activation and gradient statistics, particularly in deeper models or those employing post-activation normalization.

A plausible implication is that further hardware support for E6M2-style blockwise quantization, exact per-block accumulation, and mixed-precision master copies would unlock additional efficiency gains while safeguarding model accuracy [2501.02423, 2510.01863, 2502.05967].

Source: https://www.emergentmind.com/topics/e6m2-floating-point-scale