--- title: FP8 Low-Precision Computation url: https://www.emergentmind.com/topics/fp8-low-precision-computation type: topic --- # FP8 Low-Precision Computation FP8 low-precision computation refers to the use of 8-bit floating-point number formats, typically E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 exponent bits, 2 mantissa bits), in neural network training and inference—supplanting traditional 16- or 32-bit floating-point representations to achieve substantial improvements in memory utilization, bandwidth, and computational throughput. Modern accelerators (e.g., NVIDIA Hopper/Ada, Intel Gaudi2/3, recent RISC-V cores) implement optimized FP8 pipelines, enabling full-stack precision reduction and new algorithmic interventions that maintain training stability and accuracy at scale. This article surveys the formats, hardware architectures, quantization algorithms, architectural modifications, stability considerations, and empirical outcomes underpinning contemporary FP8 research and deployment. ## 1. FP8 Number Formats and Representational Properties FP8 arithmetic centers on two IEEE-inspired formats standardized for deep learning, E4M3 and E5M2 [2209.05433], [2409.12517]: - **E4M3**: 1 sign bit, 4 exponent bits (bias=7), 3 mantissa bits; dynamic range for normals: $2^{-6}$ to $448$; unit roundoff (½ ulp) for normals ≤ 6.25%. - **E5M2**: 1 sign, 5 exponent bits (bias=15), 2 mantissa bits; range: $2^{-14}$ to $57344$; unit roundoff ≤ 12.5%. E4M3 omits infinities in the normal range, reserving one NaN codepoint, and maximizes precision near zero, making it suitable for forward activations and weights. E5M2 retains full IEEE-754 coverage (NaN/Inf/subnormals), offering extended dynamic range for backward pass and gradient storage [2209.05433]. Arithmetic units (e.g., Tensor Cores) typically upcast inputs to higher precision (FP16/BF16) for accumulation, with inputs/outputs quantized to FP8, and scale factors (per-tensor or per-block) encode the dynamic range mapping [2407.08608], [2503.09975]. ## 2. Quantization, Scaling Strategies, and Algorithmic Pipelines ### Quantization/Dequantization For a real-valued tensor $X$, quantization uses a per-tensor or per-block scale $s$, mapping as follows: - **Quantization:** $q_{ij} = \mathrm{round}(x_{ij} / s)$, clamped to the FP8 representable range. - **Dequantization:** $\hat{x}_{ij} = q_{ij} \times s$. Selection of $s$ (scale) is vital: common policies include per-tensor (max-abs), per-channel (row or column max), and hybrid per-group [2210.05433], [2409.12517]. Block-quantization (128-element tiles) is widely adopted for compute alignment and hardware efficiency [2407.08608], [2511.02302]. ### Dynamic Scaling and Stability Delayed scaling schemes preserve stability by maintaining a history of absolute maxima and adaptively tuning $s$ to avoid late-stage underflow/overflow. For FP8 GEMM in LLM training, this yields a minimal-overhead, tensor-wise pipeline, where all quantization and dequantization is handled in the forward and backward passes with a single scale tracked per tensor [2505.20524]. Smooth quantization, dynamic range expansion (via nonlinear mappings, e.g., $f(x)=\mathrm{sign}(x)|x|^k$), and "unit scaling" (initializing all tensors to unit variance) further mitigate stability risks from limited dynamic range, especially for optimizer state or in non-linear contexts [2410.19313], [2303.11257]. ### Numeric Error and Consistency FP8 inherently increases quantization and rounding error, but judicious accumulation in FP16 or FP32, careful scale selection, and distribution-aware designs keep final training loss and accuracy within 0.1–1% of BF16/FP16 baselines across major models [2209.05433], [2410.19313], [2310.18313]. ## 3. Architectural and Dataflow Innovations for FP8 Training ### GEMM and Grouped GEMM State-of-the-art approaches achieve end-to-end FP8 computation for all GEMMs in transformer blocks (including attention), using architectural modifications to suppress outliers and "post-norm" residuals with low initial gain [2505.20524]. This circumvents prior reliance on fallbacks to higher precision in attention or activation bottlenecks. Motivated by memory/computation constraints in MoE models or variable input sizes, TMA-Adaptive FP8 Grouped GEMM eliminates inefficient padding by dynamically selecting pre-defined TMA descriptors and employing dual-phase load/store strategies to guarantee memory alignment without wasting bandwidth [2508.16584], [2511.02302]. ### Fused and Blockwise Kernels FlashAttention-3 and similar kernels maximize Hopper/Gaudi utilization by interleaving asynchronous data movement (TMA) and GEMM, block quantization, block softmax, and hardware-fused per-block rescale. This enables FP8 attention to approach 1.2 PFLOPs/s, $>80\%$ hardware utilization, and a 1.6–2.0$\times$ speedup over FP16 [2407.08608]. Fused operator suites (e.g., fused SwiGLU+Quantize, fused permute+pad) further reduce kernel launches and HBM traffic, essential in massive MoE training [2511.02302]. ### Specialized Hardware Dedicated FP8 MAC units with stochastic rounding and precision-tunable accumulators (e.g., FP8→FP12 MAC with eager SR and no subnormals) are prototyped to reduce hardware delay, area, and energy—up to 50% compared with single-precision—while maintaining accuracy [2404.14010]. Open RISC-V extensions (MiniFloat-NN, ExSdotp) implement SIMD ISA instructions, fused expanding sum-of-dot-products, and support both 4e3m and 5e2m FP8 at throughput up to 575 GFLOPS/W [2207.03192]. ## 4. Stability, Outliers, and Numeric Pathologies Prolonged FP8 training uncovers unique instabilities, most notably "outlier amplification" in SwiGLU activations during trillion-token scale training. Early FOG (Fast Outlier-Guarded) architectures employ post-norm residuals, frozen QK normalization gain, and input scaling to maintain low activation kurtosis, which empirically suppresses divergence and loss-blowup [2505.20524], [2409.12517]. "Smooth-SwiGLU" is developed to guard against alignment-induced spikes by enveloping the quadratic nonlinearity in a per-channel scaling block: $\mathrm{Smooth\text{-}SwiGLU}_i(\mathbf{x}) = s_i^{-1} Q(s_i g_i(\mathbf{x}) \sigma(f_i(\mathbf{x})))$. This preserves output numerics while capping activation spikes within representable FP8 range. With this, trillion-token, 7B-parameter decoders achieve stable convergence on par with BF16 [2409.12517]. General stability recommendations include tracking activation kurtosis as a health metric, monitoring loss landscape sharpness, using dynamic precision scheduling, and limiting FP8 to non-sensitive layers where needed [2405.18710]. ## 5. System-Level Impact and Empirical Outcomes Across hardware and scale, FP8 computation yields substantial efficiency gains: | Scenario | Baseline | FP8 Throughput | Speedup | Memory Savings | Accuracy Gap | |---------------------------|----------|---------------------|--------------|---------------|-------------------| | LLM training (8B param) | 9.1k tok/s (BF16) | 12.8k tok/s | +40% | | Match/slightly win vs BF16 [2505.20524] | | Llama-7B pretraining | 12.7 samples/s (BF16, Gaudi2) | 16.9 samples/s | +34% | Adam moments:–30% | ≤0.3 pt [2409.12517] | | Low-rank GEMM (N=20k) | 49 TFLOPS (PyTorch FP32) | 378 TFLOPS | $7.8\times$ | –75% | 1–2% rel. err. [2511.18674] | | CNNs (FPGA, M4E3) | 0.60–1.42 GOPS/DSP | 4$\times$ best prior | ∼50% | – | <0.5% top-1 [2003.03852] | In large MoE and LLM pretraining, end-to-end FP8 (with optimizer and activations quantized) enables not only dramatic speedups (1.4–2$\times$ vs. BF16), but also >1.5$\times$ reduction in overall GPU memory consumption, facilitating batch-size scaling and model scaling on fixed hardware budgets [2410.19313], [2310.18313], [2511.02302]. COAT and FP8-LM further integrate quantized optimizer moment and mixed-granularity activation solutions that minimize quantization error via dynamic range expansion or per-group scaling [2410.19313], [2310.18313]. In the context of reinforcement learning, low-precision rollout stacks with blockwise FP8 quantization for both weights and KV cache, and token-level importance sampling corrections, provide up to 44% end-to-end speedups with negligible reward and policy learning drift [2601.18150]. ## 6. Deployment, Best Practices, and Hardware Guidelines - **Scaling granularity:** Prefer per-tensor or (for maximal accuracy) per-channel scales, especially for weights with heterogeneous statistics [2409.12517], [2503.09975]. - **Activation norm and residuals:** Use post-norm (LayerScale/RMSNorm) on residual branches and frozen QK normalization for attention to avoid activation kurtosis spikes [2505.20524]. - **GEMM and grouped GEMM:** Use padding-elimination (e.g., TMA-adaptive) kernels for grouped computation; batch size and layout restrictions (e.g., block_N mult of 64) apply [2508.16584]. - **Optimizer states:** Quantize first moment with E4M3, second moment with E5M2 or higher range; dynamic range expansion or similar nonlinearity reduces bias [2410.19313], [2409.12517]. - **FP8 is not universally robust:** Without architectural and algorithmic modification (e.g., precision scheduling, smoothing, per-layer higher-precision), LLM training may experience early divergence under FP8. Monitoring activation sharpness and loss spike frequency is essential as a stability diagnostic [2405.18710]. - **Hardware/ISA design:** Accumulating in FP16/FP32, stochastic rounding in MACs, and supporting variable mantissa widths or per-group alignment maximizes efficiency [2404.14010], [2602.05743], [2207.03192]. ## 7. Current Limitations and Future Directions FP8’s efficacy in deep learning arises from workload-inherent statistical resilience and algorithmic interventions, but caution remains: - **Dynamic range and mantissa resolution trade-off**: Insufficient exponent width sharply degrades stability; E4M3/E5M2 provide empirically sufficient coverage [2209.05433], [2405.18710]. - **Long-horizon stability**: Subtle pathologies (e.g., in SwiGLU) emerge only at trillions-of-token scale, requiring specialized remedies (e.g., Smooth-SwiGLU) [2409.12517]. - **Double quantization error**: End-to-end casting-free FP8 dataflows in MoE must include scaling-consistent transposes and fused quantization operators to prevent compounding grid mismatch errors [2511.02302]. - **Hardware co-design**: Continued development of custom accelerators, flexible mantissa-width predictors, and optimized storage/access patterns is essential [2404.14010], [2602.05743]. - **Numerics-consistent inference:** FP8-trained models enable simple W8A8 deployments, but must coordinate quantization parameters to avoid train–inference mismatch [2502.05967]. Future research will address automated per-layer adaptive precision, maintainability in extreme-scale distributed systems, and further integration of FP8 formats in scientific workloads via techniques such as Ozaki-scheme decomposition [2508.00441]. --- **References**: [2209.05433], [2409.12517], [2505.20524], [2511.02302], [2410.19313], [2310.18313], [2407.08608], [2310.18313], [2503.09975], [2508.16584], [2508.00441], [2602.05743], [2511.18674], [2003.03852], [2404.14010], [2601.18150], [2303.11257], [2405.18710], [2207.03192], [2502.05967]. Source: https://www.emergentmind.com/topics/fp8-low-precision-computation