---
title: 'FP8 Calculations: Formats, Quantization & Architectures'
url: https://www.emergentmind.com/topics/fp8-calculations
type: topic
---

# FP8 Calculations: Formats, Quantization & Architectures

An 8-bit floating-point (FP8) format refers to a family of low-precision, IEEE-inspired number representations in which a single byte is divided among a sign bit, exponent field, and significand (mantissa) field. Recent advances in both hardware and software have made FP8 arithmetic highly relevant for efficient deep learning training and inference, high-performance computing, and edge deployment. The FP8 calculation ecosystem now encompasses multiple formats, quantization and scaling strategies, hardware-accelerated arithmetic, mixed-precision kernels, and complete end-to-end workflows.

## 1. FP8 Format Definitions and Numerical Properties

The canonical FP8 format is defined as follows: a single 8-bit word comprises 1 sign bit, E exponent bits, and M = 7 – E mantissa (fraction) bits, with an exponent bias $B = 2^{E-1} - 1$ [2209.05433, 2502.01070, 2303.17951].

The real value encoded by an FP8 bit pattern $[s\,|\,e_{E-1}\ldots e_0\,|\,m_{M-1}\ldots m_0]$ is:
- For normal values ($1 \leq e \leq 2^E - 2$):
  $$
  f = (-1)^s \cdot 2^{e - B} \cdot \left(1 + \frac{m}{2^M}\right)
  $$
- For subnormals ($e=0$):
  $$
  f = (-1)^s \cdot 2^{1-B} \cdot \frac{m}{2^M}
  $$
- $e = 2^E-1$ is used for special values (NaN, $\infty$).

Two widely adopted FP8 standards are E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits) [2209.05433, 2208.09225, 2502.01070]:

| Format   | Exponent Bits (E) | Mantissa Bits (M) | Bias | Min Subnormal     | Min Normal       | Max Normal  |
|----------|-------------------|-------------------|------|-------------------|------------------|-------------|
| E4M3     | 4                 | 3                 | 7    | $2^{-9}$          | $2^{-6}$         | 448         |
| E5M2     | 5                 | 2                 | 15   | $2^{-16}$         | $2^{-14}$        | 57344       |

Machine epsilon (relative rounding error) is $2^{-M}$. Thus, E4M3: $0.125$; E5M2: $0.25$ [2209.05433, 2309.14592]. Some hardware implements full IEEE compliance for E5M2; E4M3 often omits separate $\infty$ encoding, using extra bit patterns for extended normals [2209.05433].

## 2. Quantization, Scaling, and Conversion Pipelines

FP8 quantization relies on matching tensor dynamic range to the representable FP8 range and, when necessary, locally adjusting scale factors [2502.01070, 2309.14592, 2511.02302]. The basic quantization (applied per-tensor, per-row, per-group, or per-channel) is:

- Compute a scaling factor $s = \max_i |x_i| / x_{\mathrm{max}}^{\mathrm{FP8}}$, where $x_{\mathrm{max}}^{\mathrm{FP8}}$ is the largest normal FP8 value for the format in use.
- Quantize: $q_i = \operatorname{clamp}\left(\mathrm{round}(x_i / s), q_{\min}, q_{\max}\right)$.
- Dequantize: $\hat{x}_i = s \cdot q_i$.

Post-training quantization (PTQ) computes these scales using a small calibration set and experimentally observed maxima; quantization-aware training (QAT) may allow the scale (and, in some cases, the effective mantissa bits) to be learned during optimization, leveraging straight-through estimators to enable gradient flow [2208.09225, 2309.14592].

Dynamic or group-wise scaling, as employed in frameworks like COAT, further matches FP8's dynamic range to the tensor [2410.19313]. In optimal cases, “unit scaling” exploits architectural invariance to select fixed scales (e.g., $s_{layer} = 1/\sqrt{fan\_in}$ per layer) [2502.05967].

## 3. FP8 Arithmetic and Kernel Implementations

FP8 multiply-accumulate (MAC) and matrix-matrix multiply (GEMM) implementations typically cast operands to FP8, execute in higher-precision accumulators (FP16, BF16, or FP32), and, if required, cast the result back to FP8 [2602.10262, 2505.20524, 2303.17951]. The conversion to FP8 utilizes rounding-to-nearest-even, with saturation to special values at overflow. Intensive workflows (e.g., in LLMs or MoE models) employ blockwise or tilewise quantization to maximize hardware occupancy and minimize double-quantization error [2511.02302].

FP8 arithmetic can also be implemented directly with pure integer logic (integer-based add, shift, multiply), significantly reducing silicon area and critical path on FPGAs or ASICs [2406.18441]. Some neuromorphic approaches achieve bit-exact FP8 arithmetic by mapping arithmetic and rounding to threshold logic circuits in spatial combinational pipelines [2512.07724].

In high-performance computing, FP64 computations can be emulated using FP8 Tensor Cores via the Ozaki scheme—splitting operands into precisely re-scaled components to realize error-free transformations, followed by FP8 GEMMs and reconstructing the result with higher-precision accumulation [2508.00441, 2603.10634].

## 4. Error Analysis, Precision/Range Tradeoffs, and Suitability

The primary mathematical tradeoff for FP8 is between dynamic range (exponent bits) and precision (mantissa bits) [2208.09225, 2209.05433, 2309.14592]. E4M3 provides finer quantization near zero, more suitable for weights and activations with low variance, while E5M2 (and even higher-exponent formats) cover wider dynamic range, better for gradients, optimizer states, or outlier-plagued activations. For distributions with heavy tails (such as those in transformer activations), increasing exponent bits lowers MSE—network architecture and data distribution should govern format selection [2208.09225].

Empirical studies across 75 architectures show FP8 PTQ outperforms INT8 in quantization error and end-to-end accuracy, with E4M3 working best for NLP, E3M4 for CV [2309.14592].

## 5. Hardware Acceleration and Execution Characteristics

Recent accelerators (NVIDIA Hopper/H100, Intel Gaudi 2, AMD MI300A) natively support both E4M3 and E5M2 FP8 kernels [2602.10262, 2502.01070]. These devices achieve up to 2× throughput/TFLOPS and 1.8× power efficiency compared to FP16, but actual gains are limited by occupancy, memory bandwidth, and kernel tiling strategies. On AMD MI300A, FP8 MFMA instructions with FP32 accumulation are available; maximum throughput is reached when large numbers (≥256) of active wavefronts are sustained [2602.10262]. For small batch sizes or "thin" GEMMs (common in decode-stage LLM inference), achievable FP8 performance is often less than hardware peak.

FP8 hardware, however, can be 50–180% less efficient than INT8 in terms of pure compute throughput, especially for inference; thus, INT8 remains preferable for edge-centric inference deployments [2303.17951]. On the other hand, FP8 offers critical advantages for training and high-dynamic-range workloads.

## 6. Post-training Quantization, Outlier Handling, and Hybrid Kernels

FP8 quantization workflows, as validated in FireQ, integrate outlier smoothing, channel-wise scaling, and RoPE-aware normalization to maintain accuracy under aggressive quantization—crucial for LLMs with rotary positional embeddings [2505.20839]. Mixed-precision kernels, e.g., INT4 weights with FP8 activations, can further optimize bandwidth and performance.

Advanced methods, such as dynamic range expansion via companding (COAT) or mixed-precision quantization by per-tensor/activation regime, substantially reduce quantization-induced error while enabling end-to-end FP8 computation (including optimizer states and large layer activations) [2410.19313].

## 7. Applications, Conversion, and Limitations

FP8 arithmetic is now widespread in LLM training (matching or exceeding BF16 in speed and downstream accuracy at scale), federated learning (offering 2.9× communication savings over FP32), and scientific computing (enabling 8–53× acceleration for FP64 emulation) [2410.19313, 2407.02610, 2603.10634]. When converting FP8-trained networks to INT8 for inference, post-training quantization without retraining is feasible, but can incur 50–180% compute efficiency loss unless the workload is specifically optimized for integer ops [2303.17951].

FP8 formats are not universally superior: for latency-sensitive inference, low-occupancy kernels, or edge inference with INT-only hardware, INT8 remains preferred. Moreover, relative errors (machine epsilon) are substantially higher than for FP16/BF16, which places inherent accuracy limits on FP8 for outlier-prone or numerically unstable models [2303.17951, 2209.05433].

---

**References**:
- [2209.05433] "FP8 Formats for Deep Learning"
- [2303.17951] "FP8 versus INT8 for efficient deep learning inference"
- [2208.09225] "FP8 Quantization: The Power of the Exponent"
- [2508.00441] "DGEMM without FP64 Arithmetic -- using FP64 Emulation and FP8 Tensor Cores with Ozaki Scheme"
- [2410.19313] "COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training"
- [2511.02302] "FP8-Flow-MoE: A Casting-Free FP8 Recipe without Double Quantization Error"
- [2602.10262] "Execution-Centric Characterization of FP8 Matrix Cores, Asynchronous Execution, and Structured Sparsity on AMD MI300A"
- [2309.14592] "Efficient Post-training Quantization with FP8 Formats"
- [2502.01070] "An Inquiry into Datacenter TCO for LLM Inference with FP8"
- [2502.05967] "$\mu$nit Scaling: Simple and Scalable FP8 LLM Training"
- [2406.18441] "On Approximate 8-bit Floating-Point Operations Using Integer Operations"
- [2512.07724] "The Native Spiking Microarchitecture: From Iontronic Primitives to Bit-Exact FP8 Arithmetic"
- [2505.20524] "Towards Fully FP8 GEMM LLM Training at Scale"
- [2505.20839] "FireQ: Fast INT4-FP8 Kernel and RoPE-aware Quantization for LLM Inference Acceleration"
- [2407.02610] "Towards Federated Learning with On-device Training and Communication in 8-bit Floating Point"

Source: https://www.emergentmind.com/topics/fp8-calculations