---
title: Mixed FP8 Formats
url: https://www.emergentmind.com/topics/mixed-fp8-formats
type: topic
---

# Mixed FP8 Formats

Mixed FP8 formats refer to the systematic use and deployment of multiple floating-point 8-bit (FP8) number systems within a single machine learning or signal processing workflow, where the specific exponent and mantissa allocation may differ between tensors, channels, or even operations. Whereas traditional quantization approaches favored uniform formats (e.g., INT8 or a single FP8 variant), mixed FP8 quantization dynamically selects or assigns, per layer or per tensor, the most suitable FP8 format among a menu of alternatives—trading off dynamic range and resolution to minimize quantization error subject to hardware and bandwidth constraints. This paradigm exploits the observation that neural network weights, activations, and gradients exhibit diverse value distributions and dynamic range requirements that no single format can optimally accommodate. Mixed FP8 quantization thus aims to approach full-precision (FP16/FP32) fidelity while maintaining an 8-bit data path and minimizing area/power overheads [2310.13513][2104.07329][2208.09225].

## 1. FP8 Families and Representational Trade-Offs

FP8 number systems are parameterized by exponent bits ($e$) and mantissa (fraction) bits ($m$), with the constraint $1 + e + m = 8$ (one sign bit, $e$ exponent, $m$ mantissa). The most commonly used variants are:

| Name   | Exponent ($e$) | Mantissa ($m$) | Bias    | Dynamic Range                 | Min Subnormal        | Step near zero ($2^{-\mathrm{bias}-m}$) |
|--------|----------------|---------------|---------|-------------------------------|----------------------|-----------------------------------------|
| E5M2   | 5              | 2             | 15      | $6.1 \times 10^{-5}$ to $5.74 \times 10^4$ | $1.5 \times 10^{-5}$ | $1.5 \times 10^{-5}$                     |
| E4M3   | 4              | 3             | 7       | $1.6 \times 10^{-2}$ to $240$             | $4 \times 10^{-3}$   | $4 \times 10^{-3}$                       |
| E3M4   | 3              | 4             | 3       | $0.125$ to $15.5$                         | $8 \times 10^{-3}$   | $8 \times 10^{-3}$                       |
| E2M5   | 2              | 5             | 1       | $0.5$ to $3.94$                            | $1.6 \times 10^{-2}$ | $1.6 \times 10^{-2}$                     |

E5M2 maximizes range at the cost of granularity; E4M3 and E3M4 provide finer steps near zero but reduced range [2310.13513].

Mixed FP8 frameworks, such as those described in [2310.13513][2208.09225], allow per-layer or per-tensor selection among these variants. E.g., weights and activations can be encoded with higher precision (more $m$), whereas gradients requiring wider dynamic range use higher $e$ (E5M2).

Flexible FP8 quantization can extend beyond IEEE-recommended layouts; some methodologies select $(e,m)$ along with exponent bias $b$ and even presence of the sign bit, as in the FFP8 model [2104.07329].

## 2. Mixed-Precision Quantization Methodologies

Modern mixed FP8 quantization algorithms deploy empirical and analytic search procedures to assign optimal $(e,m)$ per tensor or channel, leveraging a calibration set to estimate the expected error for each candidate format. The main stages are:

- **Unified quantization and resolution analysis**: Both integer and floating FP8 quantizers are formalized in terms of effective resolution $r(x)$ and error metrics (mean-squared error, upper-bounded by $\frac{1}{4I} \sum_{i=1}^{I} r_i^2$) [2310.13513].
- **Format selection process**: For each tensor, error statistics (MSE, clip fraction) are measured for the candidate FP8 formats, and the configuration minimizing error (while respecting clipping constraints) is chosen [2310.13513][2104.07329][2208.09225].
- **Joint optimization for weight–activation pairs**: For matrix multiplications, the pair $(\alpha_\mathrm{weights}, \alpha_\mathrm{act})$ is chosen to minimize $||WX - Q^{\alpha_\mathrm{w}}(W)Q^{\alpha_\mathrm{x}}(X)||_F^2$ on a small calibration set [2310.13513].
- **Hierarchies of search**: Modes range from *All Mixed* (arbitrary mixture of INT8 and all FP8 variants, per layer), to *Mixed FP8* (restriction to FP8 only), to *Limited Mix* (weights and activations matched) [2310.13513][2104.07329].

The overall quantization workflow is strictly post-training, requiring no retraining or forward/backward modifications unless quantization-aware training is specifically invoked [2104.07329][2208.09225].

## 3. Hardware Architectures for Mixed FP8 Execution

Mixed-precision FP8 workloads impose stringent requirements on hardware, especially with respect to support for multiple $(e,m)$ formats and seamless format switching. Architectures typically feature:

- **Parametrized decoding and arithmetic paths**: Decoders map 8-bit FP8 words into internal representations (e.g., FP11, FP19, INT9), feeding into multiplier trees sized for each variant [2310.13513][2512.00053].
- **Parallel multiplier paths**: INT8 and FP8 paths operate side-by-side (e.g., INT9 8×8, FP8 5×5), with minimal area overhead (<5%) relative to pure INT8 engines, enabling runtime selection of format without pipeline stalling or resource underutilization [2310.13513][2512.00053][2411.18065].
- **Accumulator design**: Mixed-dot engines accumulate products in high-precision accumulators (FP32), maintaining numerical fidelity even as multipliers operate on low-precision FP8 [2512.00053].
- **Microcoded control for on-the-fly reconfiguration**: Emerging designs like FlexiBit use per-layer microcode control to instantaneously reformat PE datapaths for their assigned $(e,m)$ [2411.18065]. This enables full area and bandwidth efficiency across arbitrary FP8 variants without hardware idling or padding.
- **Scalable casting and conversion**: Lightweight on-chip cast units unpack/repack between FP8 variants and higher-precision internal formats as inputs/outputs traverse the hardware pipeline, making per-layer mixed deployment feasible [2104.07329][2301.03904].

## 4. Algorithmic Strategies and Adaptive Assignments

Mixed FP8 quantization leverages the statistics of each tensor, using both data-driven calibration and analytic modeling:

- **Clipping minimization**: Each tensor’s $(e,m)$ is chosen to ensure the vast majority of data fits inside representable dynamic range, discarding candidates exceeding a user-set clipping threshold (typically a few percent) [2104.07329][2208.09225].
- **Precision maximization**: Among candidates that cover the needed range, the format with largest $m$ is picked to minimize rounding error unless specific application constraints (e.g., resilience to extreme outliers) dictate otherwise [2104.07329][2208.09225].
- **Empirical search vs. analytic profile**: Networks with near-Gaussian distributions can be assigned lower $e$ (e.g., E5M2), but layers with heavy-tailed or outlier-prone statistics (e.g., transformers, attention mechanisms, early convolutions in segmentation) benefit from higher $e$ and lower $m$ (E4M3, E3M4, or even E2M5) [2208.09225][2310.13513].
- **Calibration-driven layer-wise mapping**: Small calibration datasets ($\sim$200–300 samples) suffice for stable format assignment across typical vision and language models [2310.13513][2104.07329].

State-of-the-art systems such as MicroMix further allow mixed FP8/FP6/FP4 “channel-wise” assignments with block scaling, dynamically grouping activations into sub-tensors with higher or lower-precision MXFPx for each linear layer, based on analytically-derived error upper bounds [2508.02343].

## 5. Practical Impact, Empirical Results, and Limits

The effects of mixed FP8 formats are quantitatively demonstrated across architectures and workloads:

- On ImageNet classification (ResNet, MobileNet, ViT), mixed FP8 recovers within 0.2 points of FP32 top-1 accuracy; *All Mixed* approaches close to 0.1 points [2310.13513]. INT8 alone typically loses 0.3–5 points, especially in models with high activation variance.
- On COCO detection and Cityscapes segmentation, mixed FP8 closes nearly all of the gap to full-precision: e.g., Retina-FPN mAP 36.85 (Mixed FP8) vs. 37.00 (FP32) [2310.13513].
- Natural language tasks (e.g., BERT on GLUE): FP8-based schemes lose $<0.3$\% relative to FP32 baselines; INT8 quantization yields ≈4.3% performance drop [2310.13513].
- Ablation demonstrates subnormal handling is critical; disabling subnormals can catastrophically degrade accuracy (ResNet-50 drops to $<60\%$ Top-1 when subnormals are disabled) [2310.13513].
- Mixed FP8 remains effective at bit-widths down to 6, outperforming INT6 which yields catastrophic loss ($<50\%$ retained) [2310.13513].
- In inference and training, mixed FP8 reduces memory footprint and bandwidth by 2× compared to FP16 [2209.05433]. In advanced LLM training pipelines, end-to-end FP8 schemes reduce HBM usage by up to 39% and accelerate model throughput by 21–75% versus BF16 [2310.18313].

## 6. Software, Integration, and Best Practices

- No retraining is necessary to benefit from mixed FP8 quantization in post-training quantization (PTQ); profiling and calibration suffice [2104.07329][2208.09225].
- Per-tensor/per-layer format records or control registers are updated at runtime to instruct hardware on the correct FP8 decoding/encoding, with minimal overhead [2104.07329][2411.18065].
- In training scenarios, scaling factors per tensor and global auto-scaling procedures are deployed to mitigate overflow and underflow, with delayed scaling for distributed gradient communication [2310.18313].
- For fine control, joint learning of scaling and exponent bits can be incorporated into quantization-aware training (QAT) via straight-through estimators, enabling the layerwise format distribution to evolve adaptively [2208.09225].
- Empirical studies recommend weights and activations in higher-precision FP8 (E4M3, E3M4), gradients in wide-range FP8 (E5M2), and always performing accumulation in at least FP16 to preserve numerical stability [2104.07329][2310.13513][2310.18313].
- Hardware targets (H100, Blackwell, custom GPGPU, ASIC) should support both mainline and “non-standard” FP8 variants (E3M4, E2M5) to extract maximum benefit from mixed-precision mapping [2411.18065][2508.02343][2310.13513].

## 7. Limitations, Future Directions, and Architectural Evolution

Despite their clear empirical benefits, the adoption of mixed FP8 formats presents architectural and design challenges:

- Static hardware can limit the effective range of supported FP8 formats to pre-defined sets (e.g., E4M3/E5M2 in NVIDIA products), constraining the full potential of adaptive quantization. Fully bit-parallel parametric designs (FlexiBit) remove this limitation at minimal area/power penalty, enabling truly arbitrary $($E, M$)$ assignments [2411.18065].
- Mixed-precision deployment requires fine-grained software–hardware co-design, including runtime control of format selection and on-the-fly format-specific computation graphs or microcode streams [2411.18065][2104.07329].
- As networks diversify (dense, sparse, LLMs, diffusers, Mixture-of-Experts), workload-specific optimal format signatures will require adaptive calibration and possibly in-training format learning [2208.09225].
- The transparency and correctness of numerical results in extreme quantization regimes depend critically on correct handling of subnormals, rounding, and exceptional values—failure to implement these can result in silent loss of accuracy or instability [2310.13513].

Further research aims to unify automatic quantization, hardware autotuning, and theoretical characterization of FP8 error propagation to enable robust deployment of mixed-precision FP8 across all advanced deep learning workloads.

Source: https://www.emergentmind.com/topics/mixed-fp8-formats