---
title: Flexible FP8 Quantization
url: https://www.emergentmind.com/topics/flexible-fp8-quantization
type: topic
---

# Flexible FP8 Quantization

Flexible FP8 quantization encompasses a broad family of procedures and numerical formats that exploit the flexibility in assigning exponent and mantissa bits within 8-bit floating-point representations for neural network inference and training. By tuning the assignment of dynamic range versus precision, scaling strategy, and quantization granularity, practitioners can maximize algorithmic accuracy, hardware throughput, and memory efficiency for diverse deep learning architectures, all while maintaining <1% accuracy degradation in state-of-the-art large model deployments [2503.09975].

## 1. FP8 Format Definitions and Parameterization

The core of flexible FP8 quantization is the selection and parameterization of 8-bit floating-point formats. Let an FP8 datum $x_{\mathrm{fp8}}$ be specified as:

\[
x_{\mathrm{fp8}} = (−1)^{\text{sign}} \times 2^{\text{exponent} − \text{bias}} \times \left(1 + \frac{\text{mantissa}}{2^M}\right)
\]

Where
- 1-bit sign
- $E$ exponent bits
- $M$ mantissa bits
- exponent bias $= 2^{E - 1} - 1$

Standardized formats include E4M3 ($E=4$, $M=3$, bias$=7$, range $\approx \pm 240$ to $448$ on different hardware) and E5M2 ($E=5$, $M=2$, bias$=15$, range $\approx \pm 5.7\times10^4$) [2503.09975]. The allocation of $E$ and $M$ can be dynamically chosen per layer, channel, or tensor, responsive to each layer's dynamic range and precision requirements [2311.12359, 2208.09225, 2104.07329].

For further flexibility, formats such as E3M4 and E2M5 are supported in some frameworks, and even the bias can be varied as a tunable parameter—a crucial feature in FFP8 [2104.07329].

## 2. Quantization Pipeline and Scaling Strategies

Flexible FP8 quantization proceeds via a two-phase process: scaling and casting to 8-bit floating-point. For any tensor $x$,

1. **Compute the scale $s$** (chosen per-tensor, per-channel, per-block, or per-token):
   - Per-tensor: $s = \max_i|x_i| / Q_\text{max}$ with $Q_\text{max} = 2^{(M+E-1)}-1$
   - Per-channel: $s_k = \max_j|W_{jk}| / Q_\text{max}$
   - Block-wise: $s_b = \max_{i\in\text{block}}|x_i| / Q_\text{max}$

2. **Quantize**: $q = \mathrm{clip}(\mathrm{round}(x / s), q_\mathrm{min}, q_\mathrm{max})$, where $(q_\mathrm{min}, q_\mathrm{max}) = ( -Q_\text{max}, +Q_\text{max})$

3. **Cast to FP8 bit-pattern** using the target encoding (e.g., E4M3, E5M2)

4. **Dequantize** at inference or in higher-precision accumulators: $x_\mathrm{deq} = s \times q$ [2503.09975]

Two main scaling regimes are used:
- **Static scaling**: Calibration (collect max, percentiles, histograms) performed offline; scales are fixed during inference. Fast, but risks overflow/underflow on out-of-domain data.
- **Dynamic scaling**: Recompute scale per batch or per group at runtime. Ensures high quantization fidelity, especially for rare activation spikes, at the cost of minor overhead [2503.09975, 2312.05725].

Hardware acceleration often restricts $s$ to power-of-two values (e.g., Intel Gaudi2/3: $s\in\{2^{-8}, 2^{-4}, 2^0, 2^4\}$ or $2^k$ for $k\in[-32,31]$), enabling exponent manipulation instead of multiplications [2503.09975].

## 3. Granularity and Mixed-Precision Heuristics

Granularity of scaling and quantization dictates the trade-off between hardware efficiency, memory, and accuracy:
- **Per-tensor**: Cheapest in storage and fastest (best for large GEMMs), but may introduce up to ~0.3% loss (PPL, MMLU).
- **Per-channel**: Superior accuracy (<0.1% drop), at modest scale memory and latency cost [2503.09975, 2311.12359].
- **Block-wise**: Future hardware support expected; intermediate accuracy and efficiency [2503.09975, 2311.12359].
- **Per-token**: For activations in transformers, preserves small features/outliers at minimal accuracy loss [2509.22536].

Mixed-precision workflows frequently deploy different FP8 formats for weights, activations, and gradients. Common heuristics include:
- E4M3 for forward/inference, maximizing precision if dynamic range permits.
- E5M2 (or higher exponent-count) for gradients, moments, or activations with frequent outliers [2503.09975, 2409.12517].
- Per-layer or per-channel format adaptation, with grid or optimizations to minimize quantization MSE or task loss [2311.12359, 2104.07329].

## 4. Algorithmic Best Practices and Empirical Impact

Implementation should adhere to empirically substantiated best practices:
- **Offline quantization**: Quantize weights per-output-channel for best accuracy. Power-of-two rounded scaling is preferable for compatibility with fast hardware.
- **Online quantization**: Activations quantized per-tensor statically for throughput, per-channel or dynamic if accuracy is paramount.
- First and last layers (“embedding” and “lm-head” in transformers) may be left in higher precision or exempted if quantization induces >1% degradation [2503.09975, 2312.05725].
- Use a “backoff” factor $\beta\in[0.8,1.0]$ on scale to reduce overflow.
- For transformer-layer PoC, see pseudocode in [2503.09975].

The accuracy penalty is tightly bounded: in Llama2/3 and Mistral models (7B–70B parameters), per-tensor scaling costs <2% in PPL and <0.5% on MMLU, and per-channel scaling reduces this further to <0.3% [2503.09975]. In vision and language tasks, FP8 quantization outperforms INT8 in both accuracy and stability, e.g., recover within 0.1–0.3% of FP32 accuracy [2310.13513, 2309.14592, 2312.05725, 2208.09225].

Large-matrix matmuls (4096^3–8192^3) reach up to 865 TFLOPs at >92–98% MFU (machine floating-point unit utilization) using hardware-accelerated scaling, compared to ~430 TFLOPs in BF16—a near 2× speedup. Memory footprint is halved, enabling large models (e.g. 70B LLMs) on single devices without tensor-parallel sharding [2503.09975].

## 5. Design Trade-Offs: Dynamic Range vs. Precision

Increasing exponent bits ($E$) expands the dynamic range ($\sim2^E$), beneficial in layers with heavy-tailed or outlier-ridden distributions (e.g., transformer activations, normalized representations). Applications requiring tight MSE minimization on bell-shaped, light-tailed distributions (e.g., many CNNs), instantiate more mantissa bits ($M$) to enhance precision [2208.09225, 2311.12359, 2104.07329].

Quantization MSE decomposes into rounding and clipping:
\[
\mathbb{E}[(W-Q(W))^2] = \text{Rounding Error} + \text{Clipping Error}
\]
For heavy-tails, more exponent bits reduce clipping by accommodating outliers; for tightly bounded weights/activations, fewer exponents can be used to maximize local precision [2208.09225].

Selection of per-layer format may be grid-searched post hoc (PTQ) or even optimized “in the loop” via quantization-aware training (simulating different $(E,M)$ as differentiable variables) [2208.09225, 2311.12359, 2104.07329]. In quantization-aware training, flexible division of exponent and mantissa bits loses advantage as networks regularize the effect of outliers and self-adapt to hardware-imposed limits [2208.09225].

## 6. Hardware Implementation and Compatibility

Production-worthy implementations demand hardware with first-class FP8 arithmetic support, including decoder, mantissa/exponent arithmetic logic, and accumulation in BF16/FP32. Power-of-two scales can be implemented via exponent-bias adjustment in hardware, avoiding costly per-element multiplication [2503.09975].

Hardware-support for multiple FP8 formats (E4M3, E5M2) is common (Intel Gaudi, NVIDIA H100), and efficient MACs with pipeline depth of 2 stages and minimal LUT usage can be synthesized for custom minifloat formats [2311.12359, 2104.07329]. INT8×FP8 multiply is generally unsupported to avoid hybrid datapaths [2310.13513]. Integration with model deployment pipelines is further simplified by supporting direct FP32-to-FP8 casting, and 8→32 converters with negligible area/latency overhead [2104.07329].

## 7. Future Directions and Limitations

Flexible FP8 quantization continues to expand its design envelope:
- Block-wise scaling and scaling-aware transpose operators eradicate double quantization artifacts, enabling nearly cast-free dataflows (FP8-Flow-MoE) with only two Q/DQ boundaries for 21% throughput and 16.5 GB memory improvements in 671B MoE models [2511.02302].
- Layer-adaptive scaling, learned (shifted/squeezed) scaling, and quantization-aware training “learn” the optimal dynamic range/precision compromise in-situ [2001.05674].
- Hybrid-granularity or token-wise scaling strategies are applied in transformer LLM pretraining/fine-tuning for near-lossless reasoning accuracy and 14–22% speedups [2509.22536].
- Power-law “dynamic range expansion” and mixed per-tensor/per-group quantization in COAT enable full-parameter FP8 training with 1.5× memory and 1.4× speedup over BF16 [2410.19313].
- Efficient per-tensor FP8 quantization, unlocked by suppressing mechanical activation outliers (TWEO), enables W8A8 PTQ (static, symmetric) for state-of-the-art LLMs, fundamentally shifting quantization software/hardware co-design [2511.23225].

In summary, flexible FP8 quantization orchestrates format, scaling, and granularity choices across tensors and layers to attain near full-precision model accuracy with substantial memory and computational savings. These mechanisms are now robustly validated across vision, language, and generative architectures with both empirical and analytical support [2503.09975, 2208.09225, 2312.05725, 2104.07329].

Source: https://www.emergentmind.com/topics/flexible-fp8-quantization