--- title: Float8 Training for Efficient Deep Learning url: https://www.emergentmind.com/topics/float8-training type: topic --- # Float8 Training for Efficient Deep Learning Float8 training refers to the use of 8-bit floating-point numerical formats in the training of deep neural networks. These formats, typically denoted by abbreviations like FP8 or Float8, offer a trade-off between reduced compute/memory resource consumption and potential numerical instability or accuracy loss relative to standard 16- or 32-bit floating-point representations. Recent research demonstrates that with appropriate algorithmic adaptations, Float8 training achieves substantial efficiency gains without significant degradation in convergence or final model quality across a wide array of domains, including large-scale vision, language, federated learning, and extreme classification tasks. ## 1. Definitions and Float8 Numerical Formats Modern Float8 formats are IEEE-inspired binary interchange representations, the most canonical being E4M3 (1 sign, 4 exponent, 3 mantissa bits) and E5M2 (1 sign, 5 exponent, 2 mantissa bits) [2209.05433]. These formats encode real numbers as: \[ x = (-1)^s \cdot (1 + M/2^m) \cdot 2^{E-b} \] for "normal" numbers (where $s$ is the sign bit, $M$ is the mantissa, $E$ is the exponent, $b$ is the exponent bias), and as \[ x = (-1)^s \cdot (M/2^m) \cdot 2^{1-b} \] for subnormals. E4M3 extends the dynamic range by repurposing the exponent-all-ones field, lacking infinities, while E5M2 follows full IEEE-754 semantics, including special encodings for $\infty$ and NaN. A selection of Float8 formats used in the literature: | Name | Exponent | Mantissa | Bias | Dynamic Range | Use | |---------|----------|----------|------|---------------------|--------------------| | E4M3 | 4 | 3 | 7 | ±$1.18\!\times\!10^{-3}..±143$ | Activations/weights [2209.05433, 2410.06511] | | E5M2 | 5 | 2 | 15 | ±$6.10\!\times\!10^{-5}..±6.55\!\times\!10^4$ | Gradients [2209.05433, 2410.06511] | | HiFloat8 | Tapered-precision | (1–3) | variable | 38 binades | Unified forward/backward [2409.16626] | Formats are chosen per tensor type, with wider exponent (E5M2) for gradients and narrower (E4M3) for forward-pass activations and weights. Some approaches (HiFloat8) use a single tapered-precision Float8 format, assigning more mantissa bits to lower exponents for a superior trade-off between range and precision [2409.16626]. ## 2. Core Algorithmic Techniques Float8 training requires multiple modifications to conventional mixed-precision training pipelines [1905.12334, 2301.13464, 2209.05433]: - **Precision Assignment**: Tensors in the computational graph are grouped (e.g., between GEMM operators), sorted by size, and demoted from high (FP16/FP32) to low (FP8) precision to meet a target memory ratio. During training, if the overflow rate in any group exceeds a threshold (typically 1%), all tensors in that group are promoted back to high precision [2301.13464]. - **Quantization/Dequantization**: Input data is quantized to FP8 via scaling and rounding (nearest or stochastic), optionally using per-tensor or per-row scaling factors. After computation, results are dequantized for further processing. In large-scale systems (e.g., TorchTitan), dynamic scale estimation occurs every forward/backward pass; in μnit Scaling, a static, theoretically-derived 1/$\sqrt{\mathrm{fan\_in}}$ scale is used for all hidden linear layers [2502.05967]. - **Loss/Gradient Scaling**: To address the small dynamic range/subnormal coverage of FP8, the backward loss or gradients are scaled to prevent underflow. Static or dynamic strategies are adopted, often with an overflow-detection mechanism that halves the loss scale and skips the step on overflow [1905.12334, 2409.16626, 2206.02915]. - **Rounding Mode**: Stochastic rounding has been found critical to avoid systemic quantization bias, particularly for deeper networks or massive classification heads [1905.12334, 2510.11168]. Round-to-nearest is used if quantization noise or bias can otherwise be tolerated. - **Dynamic Promotion**: To guarantee convergence and avoid numerical instability/divergence, tensors exhibiting high overflow rates in FP8 are dynamically promoted to higher precision for the rest of training [2301.13464]. ## 3. System Integration and Distributed Support From practical deployment in frameworks to large-scale distributed systems: - **Framework Support**: Float8 is being integrated as a first-class dtype (e.g., `torchao.float8` in PyTorch), with hardware-level support on NVIDIA H100 GPUs (FP8 Tensor Cores) and Graphcore IPUs [2410.06511, 2407.17353]. - **Scaling Policies**: Both static (fixed scale per tensor) and dynamic (estimated per pass as $\max(|x|)/q_{\max}$) scaling policies are implemented [2410.06511]. - **Hardware-Software Co-design**: Kernel fusion, pipeline demarcation (e.g., via `torch.compile`), and extension of memory subsystems (SymmetricMemory) are used to maximize bandwidth and overlap FP8 computation/communication [2410.06511]. - **Distributed (Federated) Learning**: Float8 enables low-overhead on-device training and communication. The FP8FedAvg-UQ approach combines unbiased stochastic quantization, per-tensor clipping, and a global FP32 server model. Empirical results show up to 6x communication savings with negligible accuracy loss [2407.02610]. ## 4. Empirical Results and Model Performance Float8 training, when combined with the above algorithmic and system strategies, achieves high-fidelity convergence across vision, language, and federated learning workloads: - **Image Classification (CNNs)**: On ImageNet, ResNet-50 trained with E5M2-based FP8 plus loss scaling matches or slightly exceeds FP32 accuracy (75.47% FP32 vs 75.70% FP8/e5m2) [1905.12334]. With S2FP8 (dynamic shift+squeeze), accuracy degradation remains below 1% even without tuned loss scaling [2001.05674]. - **Large Language Models (LLMs)**: Llama 3.1 models (8–405B parameters) trained using Float8 via TorchTitan retain perplexity within 0.1 versus BF16 across 15T tokens, with 50–65% throughput speedups [2410.06511]. μnit Scaling eliminates the need for hyperparameter rescaling/tuning and achieves convergence and accuracy on par with mixed-precision BF16, up to model sizes of 13B [2502.05967]. - **Extreme Multilabel and Output-Space Models**: Classifier heads with 3M–18M labels trained entirely in Float8, with BF16-based Kahan summation and stochastic rounding, yield >6× memory reduction and precision matching or closely following FP32 baselines [2510.11168]. - **Federated Learning**: FP8FedAvg-UQ reduces client-server communication by ≥2.9×, with test accuracies within 1% of full-precision baselines across i.i.d. and non-i.i.d. splits [2407.02610]. ## 5. Advanced Techniques and Alternatives - **Scale Propagation (Scalify)**: Scale-propagation approaches introduce explicit scale tracking and propagation through the computation graph, minimizing the need for frequent rescaling and reducing overhead. Only a handful of strategic dynamic rescale points are used (e.g., after LayerNorm backward) [2407.17353]. - **Tapered and Adaptive Formats**: HiFloat8 implements a single format with mantissa count tapered to the exponent binade, bridging the gap between precision and dynamic range and achieving near FP16 performance in both forward and backward passes [2409.16626]. - **Hardware Support**: Open RISC-V clusters and ExSdotp units enable efficient, associative 8/16-bit operations for training, achieving up to 575 GFLOPS/W for FP8→FP16 GEMMs [2207.03192]. Custom hardware architectures further reduce energy and area per MAC. - **Full Low-Precision Heads**: In the extreme classification regime, classification-head memory and compute can dwarf the encoder. Pure FP8 heads (plus Kahan and stochastic rounding) match accuracy at a fraction of the memory overhead, with custom kernels fusing backward gradient calculation and updates in SRAM [2510.11168]. ## 6. Limitations, Best Practices, and Open Problems While Float8 training is now viable for a broad array of architectures, several issues remain under investigation: - **Task/Model Sensitivity**: Uniform all-FP8 precision often results in divergence, especially in small or "hard" models. Layer-specific or groupwise adaptation with live fallback to higher precision is necessary for robust training [2301.13464]. - **Scaling/Overflow Management**: Per-tensor scale selection is critical. Too broad or infrequent scaling leads to saturation or underutilization; frequent per-matmul scaling incurs computational overhead. Approaches like μnit Scaling (static) and Scale Propagation (minimal dynamic correction) attempt to balance this tradeoff [2502.05967, 2407.17353]. - **Hyperparameter Tuning and Transfer**: Static scaling protocols such as μnit Scaling remove the need for bespoke tuning across model sizes—hyperparameters like learning rate and weight decay can be transferred directly with simple formulas involving model width [2502.05967]. - **Accumulation and Optimizer State**: Most schemes accumulate in FP32 or BF16 and retain master weights in at least FP16. Gradient and optimizer state compression to FP8 is possible but requires explicit management of scale and, often, the use of higher exponent ranges [2407.17353]. - **Rounding and Regularization**: Stochastic rounding is empirically required for unbiased, stable training in deep or wide models, as round-to-nearest introduces subtle bias [1905.12334, 2510.11168]. - **Hardware Ecosystem**: Broad adoption depends on first-class support in major ML frameworks, continued hardware development (widespread FP8 Tensor Core deployment), and continued software-hardware co-optimization [2410.06511, 2207.03192]. ## 7. Comparative Summary of Major Float8 Training Algorithms | Method/Framework | Scaling Strategy | Promotion/Overflow | Empirical Highlights | Reference | |------------------------|-------------------------|-------------------------|-----------------------------------------|-------------------| | Mixed-Precision Assignment | Largest-first demotion, overflow-triggered promotion | Tensor-wise live FP8→FP16 fallback | >2× mem savings, full convergence on CIFAR/ImageNet | [2301.13464] | | Static Per-Layer μS | Static unit-variance, no dynamic rescale | None needed | Full LLM FP8 training (up to 13B), matches BF16 | [2502.05967] | | Scale Propagation | Persistent scale over graph, rare dynamic rescale | Only at LayerNorm/critical points | FP8 loss curves overlay FP16; 30% memory/35% speedup | [2407.17353] | | S2FP8/Shifted-Squeezed | Per-tensor shift/squeeze via statistics | No loss scaling required | <1% accuracy drop without tuning/residual high-prec | [2001.05674] | | Stochastic Rounding | Optional on all quantizations | Required for very deep/wide nets | Unbiased, eliminates quantization noise bias | [1905.12334, 2510.11168] | | ELMO FP8 XMC Head | Pure FP8 head, BF16+Kahan encoder | Stochastic rounding in kernel | 6×–13× memory reduction, equivalent precision | [2510.11168] | | TorchTitan | Dynamic scale, overflow clamping | Clamped, optional reduction | 50–65% speedup, 0.1 perplexity loss on Llama-3.1 | [2410.06511] | Taken collectively, the modern Float8 training ecosystem incorporates robust numerical schemes, precise tensorwise adaptation, and direct hardware and software integration, enabling full-precision-level accuracy at a fraction of the computational and bandwidth cost previously required. These advances have made Float8 a credible and practical datatype for neural network training at scale. Source: https://www.emergentmind.com/topics/float8-training