Double-Precision Matrix Multiplication Emulation via Ozaki-II Scheme with FP8 Quantization

Abstract: In high-performance computing (HPC) applications, FP64 arithmetic remains indispensable for ensuring numerical accuracy and stability. However, in recent hardware generations, improvements in FP64 arithmetic performance have been relatively modest. Consequently, achieving sustained performance gains for FP64 computations necessitates the effective utilization of high-throughput low-precision arithmetic, such as INT8 and FP8. In several recent architectures, such as NVIDIA Blackwell Ultra and NVIDIA Rubin, INT8 performance has been significantly reduced, making reliance on INT8 alone insufficient. The use of FP8 arithmetic is thus increasingly important. In this paper, we propose a method for emulating double-precision (FP64) general matrix--matrix multiplication (DGEMM), a fundamental and performance-critical kernel in many HPC applications, using FP8 matrix multiply-accumulate (MMA) units. The Ozaki-I and Ozaki-II schemes are well established as foundational approaches for emulating DGEMM via low-precision arithmetic. For DGEMM emulation via the Ozaki-I scheme, implementations using INT8, FP8, and FP16 MMA units have been proposed, all of which can be realized based on the same underlying algorithmic structure. In contrast, although implementations of DGEMM emulation via the Ozaki-II scheme using INT8 MMA units have been reported, the original algorithm cannot be directly adapted to exploit FP8 MMA units. In this work, we introduce a novel technique to overcome this limitation and demonstrate FP64 matrix multiplication emulation based on the Ozaki-II scheme that operates on FP8 MMA units. Compared to FP8-based emulation via the Ozaki-I scheme, our method significantly reduces the number of required FP8 matrix multiplications and enables efficient FP64 emulation on emerging GPU architectures.

• The paper introduces an FP8-based Ozaki-II emulation method that employs Karatsuba decomposition and modular reduction to accurately emulate double-precision DGEMM.
• It demonstrates improved precision with over 53 effective bits using fewer multiplications compared to FP8-based Ozaki-I schemes, though it incurs higher memory overhead.
• Empirical benchmarks on platforms like RTX 5080 and B200 validate the method’s competitiveness, especially on INT8-constrained hardware environments.

Double-Precision Matrix Multiplication Emulation via Ozaki-II Scheme with FP8 Quantization

Overview and Motivation

This study introduces a practical method for emulating double-precision matrix multiplication (DGEMM) using FP8 quantization, specifically based on the Ozaki-II scheme (2603.10634). Recent hardware evolution in HPC has led to dramatic improvements in throughput for low-precision arithmetic (FP8, INT8), while FP64 performance has lagged. Architectural shifts, especially in NVIDIA Blackwell Ultra and Rubin GPUs, have seen a significant reduction in INT8 computational resources, further underscoring the need for FP8-based approaches in DGEMM emulation. The Ozaki-II scheme is a modular fixed-point decomposition technique previously implemented with INT8 units, but direct extension to FP8 is non-trivial due to underlying algorithmic constraints. This paper details an FP8-based Ozaki-II emulation method, combining Karatsuba-based matrix decompositions and modular reduction for improved efficiency and precision, and rigorously compares it with INT8-based Ozaki-II and FP8-based Ozaki-I approaches.

Ozaki-II Scheme Adaptation for FP8

The classic Ozaki-II scheme is rooted in CRT-based long integer multiplication, decomposing FP64 matrices into fixed-point representations and modular residues for accurate reconstruction. The direct application of FP8 quantization is structurally limited—the range of representable moduli in FP8_E4M3 is insufficient, constraining the dynamic range and effective precision below FP64 or even FP32 requirements.

To mitigate this, the method adopts Karatsuba decomposition, expressing integer matrices as unevaluated sums of two FP8 matrices scaled by a base $s=16$. The resulting matrix products exploit Karatsuba’s three-multiplication expansion, and careful selection of coprime moduli—up to $p_\ell \leq 513$—permits CRT reconstruction with $N \geq 13$ moduli yielding $2^{115}$ dynamic range, sufficient for FP64 emulation.

Complementarily, for certain moduli (perfect squares), modular reduction techniques obviate the need for Karatsuba recombination, reducing computational overhead further. The hybridized approach thus minimizes the required number of FP8 matrix multiplications for target precision, outperforming prior FP8-based Ozaki-I schemes in efficiency.

Precision and Hardware Throughput Analysis

A necessary requirement for high-precision emulation is error-free FP32 accumulation during FP8 matrix multiplies, which is satisfied for $k \leq 2^{16}$ due to FP8’s 4-bit mantissa. Use of FP16/BF16/FP4 is less suitable, as either their accumulation precision or representational limitations constrain applicability or require inefficient blocking.

The FP8-based Ozaki-II approach achieves over 53 effective bits with $N \geq 12$ moduli (36 multiplications), whereas FP8-based Ozaki-I needs at least 121 multiplications for 11 slices. INT8-based Ozaki-II remains more efficient, requiring only 14 multiplications for comparable precision.

Figure 1: Predicted throughput heatmaps for DGEMM emulation using INT8-based Ozaki-II scheme, as functions of BLAS GEMM throughput and memory bandwidth.

Figure 2: Predicted throughput heatmaps for DGEMM emulation using proposed FP8-based Ozaki-II scheme, as functions of BLAS GEMM throughput and memory bandwidth.

Figure 3: FP8/INT8 matrix-multiplication speedup heatmaps needed for FP8-based Ozaki-II scheme to outperform INT8-based Ozaki-II in accurate mode, conditional on memory bandwidth.

Performance models and empirical benchmarks on RTX 5080 and B200 platforms show that the INT8-based scheme remains superior where INT8 throughput is comparable to FP8. On INT8-depleted hardware, FP8-based emulation is competitive, with analytic models exceeding native FP64 throughput in bandwidth-compute favorable regions. However, FP8-based emulation incurs larger working memory overhead (e.g., 55 GB versus 27 GB for $m=n=k=16384$), mainly due to the redundant exponent storage and the need for multi-matrix decomposition per modulus.

Figure 4: Accuracy comparison on RTX 5080 in fast and accurate modes for FP8- and INT8-based Ozaki-II emulation and FP8 Ozaki-I baseline.

Empirical Evaluation

The accuracy of the FP8-based emulation matches or slightly exceeds that of the INT8-based Ozaki-II and FP8 Ozaki-I baselines when $N$ is chosen sufficiently large. Fast and accurate modes differ, with accurate mode using direct multiplication for tighter bounds, especially for matrices with large dynamic range.

Figure 5: DGEMM throughput comparison on RTX 5080 across emulation and native modes.

Figure 6: DGEMM throughput comparison on B200, highlighting INT8- and FP8-based emulation performance.

For throughput, INT8-based Ozaki-II is consistently faster than FP8-based on RTX 5080 and B200 (up to $2.9\times$), except when INT8 hardware is constrained or for very large $k$ where FP8 peak dominates. On smaller matrices or low $k$, memory-bound effects and overheads reduce the advantage.

Time breakdown analyses show that gemm phases dominate as matrix size increases, with FP8-based emulation spending proportionally more time on matrix multiplications due to the three-multiply Karatsuba expansion per modulus.

Figure 7: Time breakdown for RTX 5080, partitioned into quantization, gemm, modular reduction, dequantization, and miscellaneous overhead.

Figure 8: Time breakdown for B200, exposing efficiency loss in gemm for small matrices and scaling effects.

Implications and Future Directions

The findings have strong implications for mixed-precision HPC on modern and emerging GPU platforms. INT8 capacity remains critical for efficient high-precision emulation, enabling lower computational and memory demands. As hardware designs emphasize FP8 over INT8, the FP8-based Ozaki-II scheme provides a vital path for maintaining accuracy and throughput in DGEMM-dominated applications, such as scientific computing, simulation, and large-scale AI/ML workloads. Practical considerations, especially working memory and decomposition overhead, motivate continued algorithmic refinement and hardware-aware optimizations.

The authors also release an open-source GPU library supporting both INT8- and FP8-based Ozaki-II emulation, fostering reproducibility and adoption in diverse environments.

Conclusion

This work advances Ozaki-II DGEMM emulation by enabling FP8 quantization, combining Karatsuba decompositions and modular reduction to maximize precision and efficiency with fewer matrix multiplications. INT8-based emulation remains preferable where hardware permits, but FP8-based approaches extend applicability to INT8-constrained platforms, aligning with hardware trends in HPC and AI. Memory footprint and decomposition overheads warrant ongoing optimization, and the performance models developed offer accurate, predictive tools for deployment and algorithmic strategy selection in future HPC systems.