FP16 & FP32 Mixed-Precision Arithmetic

• FP16 arithmetic with FP32 accumulation is a mixed-precision approach that uses 16-bit multiplications combined with 32-bit summation to balance speed and accuracy.
• The methodology employs split mantissa techniques and fused multiply-add operations to overcome FP16 precision limits, ensuring near-FP32 performance.
• Optimization through hardware-software co-design, kernel fusion, and auto-tuning results in significant speedups and power efficiencies in deep learning and HPC workloads.

FP16 arithmetic with FP32 accumulation is a mixed-precision computing methodology in which floating-point operations are executed using 16-bit (half-precision) data formats for multiplications, while the accumulation (summing of products) is performed in 32-bit (single-precision) formats. This approach is widely adopted in contemporary scientific computing, deep learning training and inference, and GPU-accelerated workloads, as it combines the memory and performance advantages of FP16 arithmetic with the dynamic range and numerical stability of FP32 accumulation.

1. Data Formats and Mixed-Precision Paradigm

The IEEE-754 FP16 (“binary16”) format comprises 1 sign bit, 5 exponent bits (bias 15), and 10 mantissa bits (implicit leading 1), supporting a 6.55 × 10⁴ dynamic range and an epsilon ≈ 2⁻¹⁰. FP32 (“binary32”) contains 1 sign bit, 8 exponent bits (bias 127), and 23 mantissa bits (implicit leading 1), offering extensive precision and a dynamic range of ≈ 10³⁸. In mixed-precision units, the basic fused multiply-accumulate (FMA) operation is:

$$s \leftarrow s + a_{\mathrm{FP16}} \cdot b_{\mathrm{FP16}}$$

with $s$ in FP32 (Khattak et al., 7 Dec 2025, Abdel-Aziz et al., 2021). In hardware implementations such as NVIDIA Tensor Cores and RISC-V-based TP-FPUs, the multipliers operate on two 16-bit values, but the products are cast and summed in a full FP32 accumulator (Rout et al., 19 Nov 2025, Mach et al., 2020).

2. Algorithmic Techniques for FP32-Equivalent Precision

To compensate for the limited mantissa length of FP16 in high-precision tasks, FP32 operands are typically decomposed into two FP16 summands:

$$x_{\mathrm{hi}} \leftarrow \mathrm{toFP16}(x) \ x_{\mathrm{lo}} \leftarrow \mathrm{toFP16}((x - \mathrm{toFP32}(x_{\mathrm{hi}})) \cdot 2^{l_\mathrm{F16}+1})$$

where $l_{\mathrm{F16}}=10$ is the FP16 mantissa length (Ootomo et al., 2022, Xue et al., 31 Jul 2025). This “split mantissa” or “decomposition” enables the sum $$x \approx x_{\mathrm{hi}} + x_{\mathrm{lo}}/2^{l_\mathrm{F16}+1}$$ to retain nearly all FP32 bits in a mixed-precision computation.

For matrix-matrix multiplication (GEMM), this yields the “halfhalf” algorithm:

$$\hat{C}_{\mathrm{F32}} = A_{\mathrm{F16}} B_{\mathrm{F16}} + \frac{\Delta A_{\mathrm{F16}} B_{\mathrm{F16}} + A_{\mathrm{F16}} \Delta B_{\mathrm{F16}}}{2^{l_\mathrm{F16}+1}}$$

Negligible fourth-order error terms can be omitted as they are below the LSB of FP32 after scaling (Ootomo et al., 2022). Analogous methods are described for the BF16×BF16→FP32 path, where a three-term decomposition can reconstruct all FP32 mantissa bits (Henry et al., 2019).

3. Hardware Microarchitecture and Accumulation Pathways

FP16×FP16→FP32 arithmetic is realized as a multi-stage pipeline. The fundamental stages in open-source, FPGA-borne and ASIC architectures are:

• Multiplication: FP16 operands are multiplied (11×11-bit significands yielding up to 22 bits) (Rout et al., 19 Nov 2025, Mach et al., 2020).
• Exponent Alignment: The product’s exponent is aligned to FP32 bias (e.g., $E^\mathrm{raw} = E_A + E_B + 127 - 2\times15 + 1$) (Rout et al., 19 Nov 2025).
• Accumulator Width: Intermediate significands are held in a wide register, e.g., 27/29/31 bits (incl. guard/round/sticky) corresponding to block FMA size and GPU generation (Khattak et al., 7 Dec 2025).
• Carry-save Addition and Final Rounding: Multiple products are reduced via a carry-save adder (CSA) tree, then normalized and rounded—typically using round-to-nearest-even for FP32 output (Rout et al., 19 Nov 2025).
• Pipeline Latency and Throughput: For example, Vortex GPGPU’s FEDP achieves four-stage latency and one FMA/cycle throughput, with Tensor Core competitive area/energy efficiency (Rout et al., 19 Nov 2025).

Different generations of NVIDIA Tensor Core (Volta, Ampere, Blackwell) increase both accumulator width and fused FMA block size for improved rounding and dynamic range (Khattak et al., 7 Dec 2025).

4. Error Propagation and Numerical Stability

In mixed-precision computation, errors propagate from FP16 (quantization, underflow, rounding), FP32 accumulation, and partial sum cancellation. Mantissa analysis shows expected mantissa bits preserved to 23.75/24 in sophisticated “halfhalf” decompositions, compared to 22.25/24 for naive splits (Ootomo et al., 2022). The relative error in the mixed-precision scheme is:

$$|\widehat{z} - z| \leq f_{n+2} \cdot \sum |x_{\ell} y_{\ell}|, \quad f_m = \frac{m\, \epsilon_{\mathrm{FP32}}}{1-m\, \epsilon_{\mathrm{FP32}}}$$

Summation ordering (term-wise or grouped) plays a significant role in stability: ordering by magnitude reduces catastrophic cancellation. Scaling of the residual in decomposition (e.g., $S=2^{12}$) mitigates underflow in the low part and preserves up to 22 mantissa bits even for very small exponents (Xue et al., 31 Jul 2025).

5. Optimization Strategies: Software and Hardware

Contemporary implementations exploit hierarchical blocking, pipelined kernel fusion, and auto-tuning:

• CUTLASS and equivalents: Decomposition, residue scaling, and fragment-wide accumulation avoid internal Tensor Core round-to-zero at each step. Primary accumulation can be moved outside the Tensor Core, using FP32 round-to-nearest (Ootomo et al., 2022).
• Hardware-Software Co-design: Empirical studies show only 26 mantissa bits and 8-bit exponent alignment are required in practice for inference—enabling narrower data paths and smaller shifters while preserving accuracy (<0.05% top-1 drop on ImageNet) (Abdel-Aziz et al., 2021).
• Kernel Fusion/Epilogue: Fused GEMM kernels integrate activation, bias, and layout transformations, reducing memory traffic (Benoit, 23 Oct 2025).
• Auto-tuning and Parameter Search: Parameter sweeps over block sizes, warp sizes, and pipeline stages, filtered to maximize hardware occupancy and minimize error (Ootomo et al., 2022).

6. Application Domains and Benchmarks

FP16 arithmetic with FP32 accumulation is foundational in:

• Deep Learning: Mixed-precision training/inference yields 2–4× speedup and memory reduction; loss-scaling remedies gradient underflow. Inference and backprop are safe in FP16 with FP32 accumulation, but master weights and optimizer states must remain in FP32 (Micikevicius et al., 2017, Benoit, 23 Oct 2025).
• Scientific Computing: Lattice Boltzmann solvers, iterative linear solvers, and force-field MD models leverage mixed precision for energy savings and bandwidth (Lehmann et al., 2021, Henry et al., 2019). Near–FP32 accuracy is achievable for shifts of lattice equilibrium functions, provided DDF-shifting and summation-by-magnitude are employed (Lehmann et al., 2021).
• AI Accelerators/NPUs: Emulation of FP32 GEMM on FP16-only hardware (e.g., H2SGEMM on Huawei Ascend 910A) delivers up to 77% of the FP32-equivalent peak and, in some cases, improved numerical stability due to controlled error propagation (Xue et al., 31 Jul 2025).

Benchmark results show 3–10× speedups on Ampere/Hopper GPUs, with power efficiencies exceeding 120 GFLOPS/W, well beyond FP32-only paths (Ootomo et al., 2022, Benoit, 23 Oct 2025).

7. Limitations, Best Practices, and Future Directions

While mixed-precision arithmetic is now standard, several caveats apply:

• Accumulation in FP16: Performing GEMM or reductions in pure FP16 can result in unacceptably large rounding errors, especially for large $N$ (Khattak et al., 7 Dec 2025, Micikevicius et al., 2017).
• Reductions and Sensitive Operations: Softmax, layer normalization, or large summations should be performed in FP32, even when input data is FP16 (Benoit, 23 Oct 2025).
• Exponent Range, Underflow, and Rounding: The choice of scaling factor in decomposition is critical; improper scaling can induce severe underflow or overflow for low/high-exponent inputs (Xue et al., 31 Jul 2025).
• Reproducibility: Non-IEEE rounding in hardware (round-to-zero inside TC, absence of sticky bits) can introduce behavior varying with architecture and kernel blocking (Khattak et al., 7 Dec 2025).
• Advanced Slicing: Adaptive decomposition into multiple slices (up to three for BF16) can push accuracy near round-to-nearest FP32 (Henry et al., 2019).

Emerging directions include domain-specialized lossless conversion routines, hardware support for dynamic low-precision slicing, and cross-layer auto-tuning to further optimize mixed-precision policy (Xue et al., 31 Jul 2025, Khattak et al., 7 Dec 2025). The principle extends to other formats (FP8, BF8) as hardware and software stacks evolve toward even lower energy per operation.

---

References

• (Ootomo et al., 2022) Recovering single precision accuracy from Tensor Cores while surpassing the FP32 theoretical peak performance
• (Khattak et al., 7 Dec 2025) Accurate Models of NVIDIA Tensor Cores
• (Benoit, 23 Oct 2025) Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields
• (Henry et al., 2019) Leveraging the bfloat16 Artificial Intelligence Datatype For Higher-Precision Computations
• (Rout et al., 19 Nov 2025) A Configurable Mixed-Precision Fused Dot Product Unit for GPGPU Tensor Computation
• (Lehmann et al., 2021) On the accuracy and performance of the lattice Boltzmann method with 64-bit, 32-bit and novel 16-bit number formats
• (Mach et al., 2020) FPnew: An Open-Source Multi-Format Floating-Point Unit Architecture for Energy-Proportional Transprecision Computing
• (Micikevicius et al., 2017) Mixed Precision Training
• (Xue et al., 31 Jul 2025) H2SGEMM: Emulating FP32 GEMM on Ascend NPUs using FP16 Units with Precision Recovery and Cache-Aware Optimization
• (Abdel-Aziz et al., 2021) Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators