---
title: Mixed FP8 Quantization
url: https://www.emergentmind.com/topics/mixed-fp8-quantization
type: topic
---

# Mixed FP8 Quantization

Mixed FP8 quantization refers to a class of quantization methodologies that judiciously combine distinct 8-bit floating-point (FP8) formats—usually E4M3 and E5M2, and sometimes other variants—at fine or coarse granularity across the tensors or even within the blocks of deep neural networks, with the objective of maximizing efficiency (memory, speed, energy) while nearly preserving full-precision model fidelity. Unlike uniform precision quantization, where a single number format (e.g., INT8, FP8-E4M3) is used globally, mixed FP8 quantization assigns formats adaptively, often guided by data-driven or analytic criteria (per-layer, per-group, or per-channel) and exploits unique trade-offs between dynamic range and precision in each format. This encyclopedic entry surveys the mathematical foundations, canonical approaches, software and hardware implications, empirical results, and practical deployment considerations of mixed FP8 quantization.

## 1. FP8 Numeric Formats and Representation

The mixed FP8 quantization paradigm relies primarily on two Open Compute Project (OCP)-standard FP8 encodings [E4M3, E5M2], but some frameworks employ additional variants such as E3M4 or E2M5. The FP8 binary layout consists of a sign bit, $e$ exponent bits, and $m$ mantissa bits, where dynamic range and precision vary inversely.

| Format   | Exponent Bits | Mantissa Bits | Exponent Bias | Normalized Range   | Machine Epsilon | Intended Use        |
|:---------|:-------------:|:-------------:|:-------------:|:-------------------|:---------------:|:-------------------|
| E4M3     |      4        |      3        |      7        | $\sim [2^{-6}, 480]$ | $2^{-3}$      | Weights, Opt. 1st Mom|
| E5M2     |      5        |      2        |     15        | $\sim [2^{-14}, 5.7\times 10^4]$ | $2^{-2}$    | Activations, Opt. 2nd Mom|
| E3M4     |      3        |      4        |      3        | $\sim [2^{-2}, 17]$  | $2^{-4}$       | Vision models        |

E4M3 provides finer quantization near zero and is robust for tensors with moderate dynamic range; E5M2 sacrifices mantissa accuracy for extended coverage of outliers, vital for quantities such as second-order optimizer states or activation spikes [2410.19313][2409.12517][2509.22536][2309.14592][2310.18313].

## 2. Rationale and Statistical Motivation

Low-bit floating-point quantization is favored over integer quantization where activation or weight distributions exhibit high kurtosis, wide dynamic range, or frequent outliers. The dynamic nature of neural activations and section-wise variation in tensor statistics across layers of large models—especially language models and vision transformers—renders a uniform quantization approach suboptimal [2305.12356][2309.14592][2310.13513]. Mixed FP8 quantization seeks to:

- **Exploit complementary strengths**: Assign E4M3 where higher resolution is critical, E5M2 (or further variants) where dynamic range is limiting, and in some frameworks, integrate INT8 or FP16 where needed.
- **Minimize quantization-induced loss**: Adaptive format selection can empirically halve the mean-squared quantization error compared to uniform FP8 and recover $0.5\%$–$1\%$ of model accuracy on challenging tasks [2309.14592][2308.03290].
- **Efficiently encode rare outliers**: Retain high dynamic range only in the portions of tensors where outlier amplitudes arise (activation “spikes” in LLM projections, second-moment optimizer states) [2410.19313][2504.21553][2511.23225].

## 3. Methodological Approaches to Mixed FP8 Quantization

Implementation strategies primarily fall into three classes:

### a. Per-Tensor/Per-Group Adaptive Assignments

Frameworks such as COAT [2410.19313], InfiR2 [2509.22536], and MoFQ [2305.12356] select the FP8 format per tensor (or even per group) based on the statistical properties (max, std, kurtosis) of the tensor, minimizing either simple MSE or more involved information-theoretic costs.

- **Dynamic Range Expansion (COAT)**: optimiser states are non-linearly transformed $f(x)=\mathrm{sign}(x)|x|^k$ before quantization to fit their empirical spread to the native range of the FP8 format, then inverted post-dequant [2410.19313].
- **Hybrid-Granularity Quantization (InfiR2)**: weights are quantized blockwise in E4M3, activations tokenwise in E5M2, with power-of-two rounding for scaling—yielding near-lossless training convergence [2509.22536].

### b. Fine-Grained/Blockwise Policies

FGMP [2504.14152] employs Fisher-weighted per-block assignment: 
- Blocks of weights and activations are allocated to FP8 or even lower-precision (e.g., NVFP4, FP4) based on the block’s estimated impact on model loss, leveraging the diagonal Fisher information matrix.
- Sensitivity-weighted clipping further reduces high-magnitude, low-importance errors.

### c. Outlier- and Architecture-Aware Schemes

- **Spike-Aware Mixed-Precision (SAMPQ)**: Detects and isolates rare spiking layers (e.g., initial/final projections) that require high-range FP8 or FP16, quantizing the bulk of the model in INT8/FP8 for large memory and compute savings [2504.21553].
- **TWEO**: Introduces a regularization loss that eliminates mechanically-induced extreme outliers, allowing for 100% FP8 coverage and enabling standard low-bit quantization schemes to perform at full-precision fidelity [2511.23225].

## 4. Mathematical Formulation and Quantization Workflow

The quantization workflow unrolls as follows (notation adheres to [2410.19313][2509.22536]):

1. **Collect Tensors**: For each quantizable tensor (weight, activation, optimizer state), calibrate its empirical maxima.
2. **Select Format**: Choose FP8 variant $F^*\in\{\text{E4M3}, \text{E5M2},... \}$ minimizing a metric:
   $$
   F^* = \arg\min_{F}\;\mathrm{MSE}_F \text{ or alternative cost}
   $$
3. **Compute Scale**: Per tensor/group/channel, set $S = \max|X| / V_{\max}^{F^*}$, possibly snap to nearest power-of-two [2509.22536].
4. **Quantization Mapping**:
   $$
   Q(x_i; S) = \mathrm{Clip}\left(\mathrm{round}(x_i/S),\,q_{\min},\,q_{\max}\right) \cdot S
   $$
5. **Dynamic Range Expansion** (when required, e.g., optimizer): apply $f(x)$ and its inverse as detailed under COAT [2410.19313].

6. **Assignment and Kernel Routing**: Pass the scale and format metadata forward for (re)quantization or dequantization within specialized GEMM kernels (e.g., NVIDIA FP8 tensor cores, Blackwell mxFP kernels).

Granularity is highly implementation-dependent: Per-tensor schemes simplify kernel dispatch and minimize metadata; per-group/channel arrangements optimize accuracy but increase memory for scale storage and control logic.

## 5. Empirical Benchmarks and Engineering Impact

Across architectures (LLMs, CNNs, VLMs, ViTs), mixed FP8 quantization demonstrates:

- **Memory Savings**: COAT achieves $1.54\times$ end-to-end training memory reduction versus BF16 and up to $1.65\times$ reduction in activation footprint [2410.19313]. InfiR2 and FP8-LM yield $10$–$39\%$ memory reduction in model/optimizer state [2509.22536][2310.18313].
- **Throughput Gains**: Reported $1.43$–$1.57\times$ speedup compared to BF16, with up to $36\%$ faster training under full FP8 (TWEO) [2410.19313][2511.23225].
- **Accuracy/Convergence**: Across Open LLM (Llama, OLMo), VLM (VILA), and reasoning (AIME24, GPQA) tasks, mixed FP8 matches BF16 within noise; uniform INT8 and even uniform FP8 can suffer large degradations in certain layers, which are avoided by mixed strategies [2410.19313][2308.03290][2305.12356].
- **Energy and TCO**: FGMP delivers $14\%$ end-to-end energy reduction in inference [2504.14152]; mixed FP8 on Gaudi 2 cuts inference TCO per token by reducing both power and model time-to-completion [2502.01070].

## 6. Hardware and Implementation Considerations

Mixed FP8 quantization is most effective when coupled with hardware supporting both exponent/mantissa parameterizations and dynamic routing of kernel operations:

- **Native Kernel Support**: Modern architectures (e.g., NVIDIA Hopper, Blackwell) provide FP8 tensor cores—supporting E4M3/E5M2—and custom operators (e.g., MicroMix, FP8-LM) for blockwise/channelwise mapping [2508.02343][2310.18313].
- **Control Logic and Overhead**: Adding per-block mixed-precision incurs minimal area overhead (<$5\%$) and negligible per-operation energy, provided kernel fusion and dataflow are optimized [2504.14152][2310.13513].
- **Scale Metadata**: The storage and update of scaling factors must be bandwidth- and memory-efficient; power-of-two rounding and quantized scale representations (e.g., E8M0) are frequently used [2509.22536][2508.02343].
- **Software Integration**: Model wrappers (PyTorch modules, custom quantize-dequantize operators), calibration routines, and format-selection heuristics are integrated into existing pipelines without hyperparameter tuning [2410.19313][2310.18313][2309.14592].

## 7. Best Practices and Deployment Guidelines

- For NLP and LLMs, default to E4M3 for weights and optimizer first moments; select E5M2 or higher range for activations/second moments exposed to outliers [2410.19313][2409.12517][2509.22536][2309.14592].
- Apply per-tensor quantization for linear layers; for non-linear or outlier-prone tensors, use per-group or per-channel granularity [2410.19313][2509.22536].
- Employ stateless (delayed) scaling for maximum hardware utilization and numerical stability [2409.12517][2511.23225].
- Integrate architecture-aware spike detection (SAMPQ, TWEO) where activation outliers create catastrophic quantization failures—use regularization if necessary to avoid collapse [2504.21553][2511.23225].
- In hardware-native settings (Hopper, Blackwell), exploit fused mixed-precision tensors and preferred scale storage for minimal code and memory overhead [2508.02343][2310.13513].
- For deployment, combine mixed FP8 quantization with hybrid INT/FP8 schemes at the layer level using MoFQ or similar algorithms, tailoring format to error minimization [2305.12356][2309.14592][2308.03290].

---

Mixed FP8 quantization is a principled, empirically validated approach for maximally efficient memory, compute, and energy usage in modern large-scale neural networks, yielding near-lossless accuracy and throughput on par or better than uniform quantization on current accelerator hardware [2410.19313][2509.22536][2504.14152][2310.18313][2511.23225][2308.03290][2309.14592][2305.12356].

Source: https://www.emergentmind.com/topics/mixed-fp8-quantization