---
title: 'EmuGEMM: High-Precision GEMM Emulation'
url: https://www.emergentmind.com/topics/emugemm
type: topic
---

# EmuGEMM: High-Precision GEMM Emulation

EmuGEMM is a precision-emulation framework for general matrix–matrix multiplication (GEMM) that exploits the throughput and power efficiency of low-precision (INT8) matrix engines—such as NVIDIA/AMD Tensor Cores—to achieve floating-point (FP32, FP64) or arbitrary-precision results. EmuGEMM systematically implements Ozaki Schemes I and II, fusing mathematical reconstruction strategies with hardware-level pipeline optimizations, and demonstrates substantial performance and energy efficiency improvements over existing software and hardware approaches on modern GPUs [2508.03984][2504.08009][2606.25453].

## 1. Mathematical Foundations: Ozaki Schemes I and II

EmuGEMM builds on two modular emulation paradigms: mantissa splitting (Ozaki Scheme I) and integer modular arithmetic with the Chinese Remainder Theorem (Ozaki Scheme II).

**Scheme I (Mantissa Splitting):**  
Matrices \(A \in \mathbb{R}^{M \times K}\), \(B \in \mathbb{R}^{K \times N}\) are decomposed into $p$ INT8 slices per operand, quantized with power-of-two row/column scalings $\mu$, $\nu$. The approximation
\[
A \approx \operatorname{diag}(\mu)\sum_{i=0}^{p-1}2^{-\beta i}A'_i,\qquad B \approx \sum_{j=0}^{p-1}2^{-\beta j}B'_j\operatorname{diag}(\nu)
\]
yields a blockwise multiplication
\[
C \approx \operatorname{diag}(\mu)\left(\sum_{s=0}^{p-1}2^{-\beta s}C_s\right)\operatorname{diag}(\nu),\qquad C_s = \sum_{i=0}^{s}A'_i B'_{s-i}
\]
requiring $p(p+1)/2$ exact INT8→INT32 GEMMs.

**Scheme II (CRT-Based Modular Arithmetic):**  
The input matrices are scaled/truncated to $A', B' \in \mathbb{Z}$, then, for each of $N$ pairwise-coprime moduli $p_1,\ldots,p_N$, residue matrices $(A'_i, B'_i)$ are formed and multiplied:
\[
C'_i = A'_i B'_i \bmod p_i
\]
The outputs $C'_i$ are then reconstructed with the Chinese Remainder Theorem:
\[
C' = \sum_{i=1}^N \frac{P}{p_i}q_i(C'_i \bmod p_i) \bmod P
\]
where $q_i$ is the inverse of $P/p_i$ modulo $p_i$, $P = \prod_{i=1}^N p_i$, and scaling is inverted to recover high-precision $C$.

The theoretical precision of Scheme II is set by $\log_2P$, controllable via $N$ and the sizes of the $p_i$ [2504.08009][2606.25453].

## 2. End-to-End Algorithmic Pipeline

The EmuGEMM workflow for high-precision emulation proceeds as follows [2508.03984][2504.08009]:

1. **Moduli and Preprocessing:**  
   - Select $N$ coprime moduli $p_1,\dots,p_N$ (for Scheme II), or $p$ slices and shift width $\beta$ (Scheme I).
   - Compute necessary pre-factors and scaling vectors for diagonal alignment (row $\mu$, column $\nu$).

2. **Integerization and Scaling:**
   - Scale and truncate $A$, $B$ to integer/INT8 range using powers of two for exponents, so as to guarantee uniqueness: $2\sum_h |a'_{ih}||b'_{hj}| < P$.
   - For each modulus (Scheme II), compute $A'_i = \operatorname{rmod}(A',p_i)$, $B'_i = \operatorname{rmod}(B',p_i)$ as INT8 matrices.

3. **Low-Precision GEMMs:**
   - Execute $N$ independent INT8$\times$INT8$\to$INT32 GEMMs per modulus (Scheme II) or $p(p+1)/2$ for mantissa slices (Scheme I).

4. **Modular Reduction and Accumulation:**
   - Post-process INT32 accumulators to residues (e.g., modulo $p_i$ for CRT).
   - For Scheme II, accumulate partial results in higher-precision (FP64, double-double if necessary), handling rounding/carry as required for FP64 emulation.

5. **CRT Reconstruction and Scaling:**
   - Apply CRT to combine the mod-$p_i$ results.
   - Re-invert the scaling: $C = \operatorname{diag}(\mu)^{-1} C'' \operatorname{diag}(\nu)^{-1}$ (both schemes).

6. **Result:**  
   - Output matches floating-point accuracy (FP32/FP64) with errors determined only by the number of moduli (Scheme II) or slices (Scheme I), and the accuracy of truncation/scaling.

## 3. Hardware and Dataflow Optimizations

EmuGEMM achieves high efficiency on NVIDIA Hopper and Blackwell by fusing integer GEMM operations with on-chip accumulation, optimized tiling, and shared-memory pipelining [2606.25453].

- **Elimination of Global Memory Round-Trips:**  
   - INT32 accumulators for all slice/modulus products are held on-chip (in RF or TMEM) throughout the $K$-loop. No intermediate partial sums are written/read from global memory, relieving bandwidth bottlenecks.

- **Interleaved Data Layout (Scheme I):**  
   - $p$ INT8 slices are interleaved at the Tensor Core tile granularity along $K$; all slices are loaded together using a single TMA descriptor, maximizing memory efficiency.

- **Persistent Kernel Fusion:**  
   - Fusion of all $p(p+1)/2$ MMA evaluations ensures occupancy and maximizes throughput.

- **On-Chip Modular Reduction (Scheme II):**  
   - Modulo reductions are performed on the accumulator registers or TMEM epilogue, and only final INT8 tiles are written back.

- **Pipeline Overlap:**  
   - Data movement (TMA load), MMA compute, and epilogue (shift-reduce or modulo) are overlapped to hide memory and pipeline latencies, saturating INT8 hardware.

- **Blackwell Architecture Features:**  
   - Dedicated TMEM expands the effective pipeline depth, enabling higher $p$ and larger tile sizes.

## 4. Performance, Scalability, and Power Efficiency

The EmuGEMM implementations realize substantial speedups and energy improvements over native and legacy emulation methods [2508.03984][2606.25453]:

| Architecture  | Method          | Precision | Matrix Size $N$ | Throughput (TFLOP/s) | Speedup vs Native | Power Efficiency Gain |
|---------------|-----------------|-----------|-----------------|----------------------|-------------------|----------------------|
| GH200 Hopper  | EmuGEMM-II      | FP64      | 16,384          | 81.6                 | 1.4×              | 43%                  |
| GH200 Hopper  | EmuGEMM-II      | FP32      | 16,384          | 160                  | 3.0×              | 154%                 |
| Hopper        | EmuGEMM-I, $p=4$| FP32      | 16,384          | 400                  | 1.7× (vs FP32)    | –                    |
| Hopper        | EmuGEMM-II, $p=15$| FP64    | 16,384          | 94                   | 1.6× (vs FP64)    | –                    |
| Blackwell B200| EmuGEMM-II, $p=15$| FP64    | 16,384          | 165                  | 4.6× (vs FP64)    | –                    |

Key highlights:

- EmuGEMM maintains 1.4–3.0× throughput advantage over native DGEMM/SGEMM at large $N\ge 16,384$.
- Power efficiency improvements of 43–154% over FP64/FP32 are reported for large problems.
- On Blackwell, EmuGEMM-II outperforms cuBLAS ZGEMM by up to 5.5× at FP64-level accuracy.

Overhead from scaling, residue conversion, and CRT reconstruction falls below $10\%$ for $N\ge 16,384$, making EmuGEMM optimal for large-scale GEMM.

## 5. Precision–Throughput Trade-Offs and Parameterization

The emulation accuracy and cost are governed by the number of moduli (Scheme II) or split slices (Scheme I):

- Scheme I precision is nearly linear in $p$, with each slice width $\beta$ yielding about $\beta p$ bits. For example, $p=4$ slices and $\beta=8$ offer $\sim$32 bits.
- Scheme II achieves $\log_2 P$ bits, with $N\approx 14\text{–}17$ moduli sufficing for FP64, $N\approx 6\text{–}9$ for FP32 accuracy. Number and size of moduli adapt to application needs.

A smaller $N$ reduces overhead and matches, or exceeds, intermediate precisions (e.g., between FP32 and TF32).

## 6. Limitations and Application Scope

EmuGEMM is most beneficial for large GEMM problems ($n, k, m \geq 8,000$), where the compute dominates the overheads of scaling, conversion, and reconstruction [2508.03984][2606.25453].

- For small matrices ($n \leq 4,096$), overheads may reach $20$–$50$% of runtime.
- Accuracy for highly ill-conditioned matrices may require “accurate” mode scaling (an extra INT8 GEMM) to properly bound dynamic range.
- EmuGEMM can be tuned for various hardware (any with INT8$\times$INT8 Tensor Core analogues), using standard CUDA WMMA/AMX APIs and on-chip optimized kernels; legacy hardware or those lacking sufficient on-chip storage may see reduced benefit.

## 7. Comparative Frameworks and Benchmarks

EmuGEMM substantially exceeds the performance of prior emulation schemes and vendor-mixed-precision libraries:

- Compared to legacy Ozaki (Scheme I) emulation on CPU, CRT-based EmuGEMM (Scheme II) reduces GEMM calls by up to $2.3\times$ and more than doubles throughput for quadruple-precision emulation [2504.08009].
- In single-precision, EmuGEMM achieves >2× higher performance and lower power than cuMpSGEMM and BF16×9 on large matrices [2508.03984][2606.25453].
- Elimination of global-memory round-trips for INT32 partial sums is a primary source of speedup, raising arithmetic intensity by up to $8\times$ [2606.25453].

## Summary

EmuGEMM synthesizes Ozaki Scheme I (mantissa splitting) and Scheme II (CRT-based modular arithmetic) into an optimized set of fused-kernel implementations targeting the highest-throughput, lowest-power matrix engines. Mathematical rigor is preserved up to and beyond FP64, with parameterizable accuracy–throughput trade-offs. Its principal innovations—strict on-chip execution, specialized data layouts, and pipeline fusion—allow modern architectures like Hopper and Blackwell to approach the intrinsic INT8 roofline even for full-precision GEMM. This positions EmuGEMM as a practical and high-performance tool for scalable mixed-precision and scientific compute workloads [2508.03984][2504.08009][2606.25453].

Source: https://www.emergentmind.com/topics/emugemm