Low-Precision Matrix Engines

• Low-precision matrix engines are specialized systems that perform matrix multiplication using reduced (≤16-bit) arithmetic to boost efficiency and throughput.
• They employ advanced numerical algorithms like iterative refinement and mixed-precision emulation to maintain numerical accuracy despite lower precision.
• Integration in modern GPUs, CPUs, and edge devices supports diverse applications, from deep learning inference to scientific simulation and quantitative analysis.

Low-precision matrix engines are hardware and algorithmic systems dedicated to high-throughput matrix multiplication and related linear algebra operations using reduced-numerical-precision arithmetic (typically ≤16 bits). Originating in response to the computational and bandwidth demands of deep learning, such engines now propagate to scientific, embedded, and high-performance computing where performance, efficiency, and adaptive numerical fidelity are critical. Their design leverages specialized data paths, mixed-precision error management, and innovative software stacks, combining advances from modern processor architectures, numerical methods, and emerging number systems.

1. Hardware Architectures and Precision Formats

Matrix engines (ME) are now fundamental blocks in modern processors—GPUs, AI accelerators, and general-purpose CPUs—owing to peak throughput and energy efficiency unattainable by classic general-purpose units (Abdelfattah et al., 2020, Domke et al., 2020). Key forms include:

• Systolic Arrays and Tensor Cores: Evolved from 2D systolic arrays, these execute tens or hundreds of multiply–accumulate (MAC) operations per cycle. For example, NVIDIA Tensor Cores (Volta–Blackwell) and Google TPU systolic arrays process FP16, bfloat16, FP8, INT8 inputs, often using mixed-precision accumulation (e.g., FP16→FP32) (Domke et al., 2020, Mukunoki, 1 Aug 2025).
• CPU Matrix Engines: Intel AMX, IBM MMA, and emerging RISC-V/ARM ME instructions support low-precision GEMM through wide dot-product instructions with on-chip accumulator tiling (Kuzma et al., 2023, Martínez et al., 13 Jun 2025, Cammarata et al., 10 Apr 2025).
• Edge and TinyML Engines: RedMulE, Quadrilatero, and similar architectures integrate compact systolic arrays achieving up to 688 GFLOPS/W at sub-100 mW power budgets suitable for on-device adaptive neural network compute (Tortorella et al., 2022, Tortorella et al., 2023, Cammarata et al., 10 Apr 2025).
• Number System Innovations: FP16, bfloat16, FP8 (E4M3, E5M2), and posit<16,2> formats are widely used. Hardware designs may provide custom logic for on-the-fly conversion, casting, and even approximate normalization to reduce area/power (area savings ~16%) (Alexandridis et al., 21 Aug 2024, Quinlan et al., 23 Aug 2024).

A representative operation is low-precision GEMM:

$D = A \times B + C$

where $A, B$ are typically in FP16, INT8, or even FP8, and accumulation is performed in FP32 or INT32.

2. Numerical Algorithms for Mixed- and Low-Precision Engines

The shift to low/narrower precisions is accompanied by developed algorithms that permit aggressive downscaling of arithmetic accuracy without sacrificing final numerical fidelity:

• Iterative Refinement (IR): The principal technique; compute the expensive $O(n^3)$ arithmetic (e.g., LU or Cholesky) in low precision, then iteratively correct the solution in higher precision:

$x_{n+1} = x_n - f(x_n)/f'(x_n)$

In matrix terms, after a low-precision factorization and solve, the high-precision residual is used to generate a correction, which is applied after a high-precision update (Abdelfattah et al., 2020, Dmytryshyn et al., 5 Mar 2025).

• GMRES-based IR (GMRES-IR): Uses low-precision preconditioning for a GMRES solver in the correction step, relaxing condition number and precision requirements (Abdelfattah et al., 2020).
• Ozaki Scheme / Emulation: Achieves high-precision matrix multiplication (e.g., DGEMM) via a decomposition into multiple low-precision slices. For floating-point or integer MEs, operands are split, low-precision GEMMs are accumulated and appropriately scaled:

$$x^T y = \sum_{i,j} 2^{-(i+j)\alpha} x^{(i)T} y^{(j)}, \quad \alpha = \left\lfloor \frac{-\log_2 u + \log_2 k}{2} \right\rfloor$$

where $u$ is the unit roundoff, $k$ is the inner-product length (Ootomo et al., 2023, Mukunoki, 1 Aug 2025, Uchino et al., 6 Aug 2025, Dawson et al., 18 Jul 2024).

• Sparse Residual Correction: Fast low-precision GEMM is combined with residual error correction applied through sparse matrix multiplication, exploiting the fact that significant residuals occur rarely. This balances accuracy and speed, especially effective for int8 quantized arithmetic (Gu, 11 Mar 2024).
• Low-Rank Factorization and Compression: Randomized, quantized, and blockwise low-rank matrix factorizations (e.g., LPLR, RSVD-aided LRAMM) reduce the expensive GEMM step’s complexity by projecting onto low-precision, low-dimension subspaces while bounding the overall error:

$A \approx L R, \quad L=Q(AS),\; R=Q'(L^\dagger A)$

with quantizer $Q$ chosen per data norm and desired compression ratio (Saha et al., 2023, Gu, 27 May 2024).

3. Software Ecosystem and Compiler Support

Widespread deployment of low-precision MEs has pressured the software stack toward increased flexibility and hardware awareness:

• High-Performance Libraries: Vendor libraries (cuBLAS, cublasLt), LAPACK (including routines for IEEE format conversion), MAGMA (mixed-precision factorization/refinement), PETSc, Trilinos, hypre, Ginkgo, SuperLU, STRUMPACK, and heFFTe now support low- and mixed-precision routines, on-the-fly conversion, adaptive preconditioners, and decoupling of storage vs. compute precision (Abdelfattah et al., 2020).
• Compiler-Intrinsic Matrix Optimization: Automatic embedding of data packing, tiling, and micro-kernel construction in compilers (LLVM) enables performance close to hand-crafted BLAS, and easy retargeting to new matrix engines (e.g., POWER10 MMA, Intel AMX, ARM ME), supporting low-precision and mixed-precision operand lowering (Kuzma et al., 2023).
• Emulators and Experimental Tools: Tools for low-precision emulation (“chop”, MPFR-based posit arithmetic) facilitate prototyping for hardware lacking native support (Abdelfattah et al., 2020, Quinlan et al., 23 Aug 2024).

4. Performance, Power, and Trade-offs

Performance improvements from low-precision matrix engines are architecture- and workload-dependent, but several robust trends are established:

Architecture/Method	Speedup vs. FP64	Energy Efficiency	Use Cases
Tensor Cores (FP16/FP32 acc)	4–8×	Up to 4–5× (MAGMA, cuBLAS)	DNN training, dense GEMM
INT8 TC with Ozaki scheme	Up to 6×	50–75% less memory	DGEMM via fixed-point, quantum sim
FP8 TC Ozaki / Emulation	2× FP16TC	Work-memory constrained	Large batch DGEMM, Blackwell GPU
INT8 CRT-based emulation	1.4–3×	43–154% more power eff.	GH200 Superchip, SGEMM/DGEMM
Systolic array on edge SoC	22× over SW	4.65–77% area/energy gain	Extreme edge, IoT TinyML
Sparse-residual (SPMM)	1.46× (CPU)	+15% accuracy	Low-precision quantized GEMM

Significant caveats arise:

• HPC Kernel Coverage: Most speedup is realized in workloads dominated by dense Level-3 BLAS (e.g., HPL, certain DNNs), but many scientific/HPC workloads are bandwidth-limited or rely on Level-1/2 BLAS, with only modest global gains (Domke et al., 2020).
• Programmability and Mapping: Non-trivial porting is needed for legacy codes to optimally leverage matrix engines. Level-1/2/elementwise code gains little; overzealous “GEMMization” may degrade performance for some applications (Domke et al., 2020).
• Accuracy Control: Precision emulation (Ozaki/CRT) incurs overhead scaling with the input exponent range and bit-slice count; wide dynamic range or large splitting increases compute/memory demand (Ootomo et al., 2023, Mukunoki, 1 Aug 2025).
• Efficiency Boundaries: For small matrices/control-limited SoCs, control/pipeline/memory overhead dominates; for sparse/dense hybrid workloads, SPMM (sparse matrix mult) win is function of residual matrix density (Tortorella et al., 2022, Gu, 11 Mar 2024).

5. Application Domains and Methodological Innovations

Low-precision matrix engines underpin a variety of advanced methods:

• Quantized Deep Learning: MEs enable high-throughput INT8/FP16 inference/training, exploiting mixed-precision micro-kernels and carefully tuned data layout (blocking, micro-panelization, alignment) (Martínez et al., 13 Jun 2025, Cammarata et al., 10 Apr 2025).
• Scientific Simulation: Error-tolerant kernels (e.g., quantum chemistry density matrix purification, quantum circuit simulation) benefit from mixed- or low-precision GEMM with iterative refinement or error-free emulation schemes ensuring scientific fidelity (Dawson et al., 18 Jul 2024, Ootomo et al., 2023).
• Hierarchical Matrices and Compression: In hierarchical off-diagonal low-rank (HODLR) matrices, off-diagonal factor precision can be adaptively lowered without substantially influencing global matrix–vector or LU backward error, amortizing memory and compute cost (Carson et al., 31 Jul 2024).
• Polynomial Evaluation: Mixed-precision Paterson–Stockmeyer evaluation of polynomials (matrix exponentials, cosines) executes blocks corresponding to small-magnitude coefficients in low precision, yielding 20–40% complexity reductions while preserving error tolerances in the working precision (Liu, 2023).
• New Number Systems: Posit arithmetic (\posit<16,2>) used in low-precision LU factorization and iterative refinement provides enhanced dynamic range and resilience to overflow/underflow relative to fp16, supporting robust sparse linear system solves (Quinlan et al., 23 Aug 2024).

6. Emerging Directions and Research Outlook

Low-precision matrix engines represent a broad, rapidly evolving field intersecting hardware, software, and applied mathematics:

• Hardware–Algorithm Co-design: Continuing trends include the introduction of new number formats (e.g., FP8, hybrid fixed-floating, posits), matrix ISA extensions (e.g., Quadrilatero/Kinara’s KAPU), approximate functional units, adaptive clock/power gating, and further coupling of memory precision to bandwidth needs (Cammarata et al., 10 Apr 2025, Alexandridis et al., 21 Aug 2024).
• Software–Hardware Portability: Compiler infrastructures (LLVM intrinsics, modular micro-kernel lowering) and algorithmic frameworks are increasingly accommodating emerging matrix engines and their varying semantics, bitwidths, and data movement models (Kuzma et al., 2023).
• Energy and Power: Design for power- and area-limited deployment (edge, IoT, smart-sensor, TinyML) prioritizes energy per MAC, exploiting new architectures and aggressive clock/power gating (Tortorella et al., 2022, Tortorella et al., 2023).
• Scientific Computing Integration: Wider adoption of Ozaki-inspired error-free emulation, low-rank approximate multiplication, and mixed-precision error analysis is transitioning precision-tolerant scientific kernels onto AI-oriented hardware (Dawson et al., 18 Jul 2024, Mukunoki, 1 Aug 2025, Uchino et al., 6 Aug 2025).
• Numerical Stability Analysis: Research continues into roundoff error, convergence thresholds (especially for iterative refinement or stationary methods), and the dependence of attainable accuracy on both the input matrix spectrum (dynamic range) and the format-specific parameters (unit roundoff, bits/slice) (Abdelfattah et al., 2020, Liu, 2023, Dmytryshyn et al., 5 Mar 2025).

In summary, the current paradigm for low-precision matrix engines centers on exploiting hardware data paths optimized for low- or mixed-precision multiplications, in combination with sophisticated algorithmic error management and adaptive software, to maximize performance, energy efficiency, and fidelity. The field is defined by a synergy between hardware capability, mathematical insight, and software systemization, all underpinning next-generation high-performance and embedded computing.