High Performance GMRES Mixed Precision

• The paper presents HPG-MxP as a benchmark quantifying useful mixed-precision speedups in GMRES-based solvers by comparing iteration counts to a double-precision reference.
• It employs right-preconditioned restarted GMRES-IR with CGS2 orthogonalization and geometric multigrid preconditioning to solve sparse Poisson-like systems.
• Exascale implementations demonstrated a 1.6× speedup and significant energy savings, highlighting the trade-offs in achieving practical solver acceleration.

High Performance GMRES Mixed Precision (HPG-MxP) is a benchmark for useful mixed-precision performance on sparse, memory-bandwidth-limited scientific workloads. It is designed to measure how effectively a high-performance computing system converts reduced precision into speedup for a GMRES-based sparse solver while preserving convergence behavior close to a double-precision reference, rather than merely counting low-precision arithmetic throughput (Kashi et al., 15 Jul 2025). In the broader mixed-precision benchmark ecosystem, it is positioned alongside HPL-AI and HPL-MxP, but it emphasizes sparse linear algebra / GMRES-like iterative behavior rather than dense LU factorization (Dongarra et al., 23 Sep 2025).

1. Definition and benchmark objective

HPG-MxP was proposed because many scientific simulation workloads are memory-bandwidth limited, not compute limited, so the large low-precision throughput gains seen in dense AI-style kernels do not automatically translate into practical solver acceleration. The benchmark therefore asks a narrower and more application-relevant question: what useful speedup can an HPC machine deliver for sparse mixed-precision scientific computing, under a requirement of “essentially the same usefulness as double-precision computation” (Kashi et al., 15 Jul 2025).

The benchmark targets sparse matrix-based simulation motifs typical of PDE solvers. Like HPCG, it uses a structured Poisson-like problem on a uniform mesh, but it replaces Conjugate Gradient with GMRES, which extends relevance to nonsymmetric problems and changes the dominant cost structure through Krylov-basis storage, orthogonalization, and restart behavior. This makes HPG-MxP a memory-wall benchmark for sparse iterative solvers rather than a dense-factorization benchmark (Kashi et al., 15 Jul 2025).

A central feature is that HPG-MxP does not reward mixed precision simply for being faster. The mixed-precision score is reduced if mixed precision requires more iterations than the double-precision reference to reach the target residual reduction. This makes HPG-MxP a benchmark of useful performance, not raw mixed-precision flop rate (Kashi et al., 15 Jul 2025).

2. Linear system, solver, and prescribed numerical structure

HPG-MxP solves a sparse linear system

$\mathbf{A}\mathbf{x}=\mathbf{b}.$

The preferred benchmark matrix is the same symmetric weakly diagonally dominant matrix as in HPCG, with diagonal entries $a_{ii}=26$ and off-diagonal stencil entries $a_{ij}=-1$. The benchmark allows an optional nonsymmetric matrix, but the symmetric version is noted to be at least as hard for GMRES (Kashi et al., 15 Jul 2025).

The prescribed algorithm is right-preconditioned restarted GMRES with iterative refinement, specifically a GMRES-IR formulation with CGS2 orthogonalization. The right-preconditioned system is

$$\mathbf{A}\mathbf{M}^{-1}\mathbf{y}=\mathbf{b}, \qquad \mathbf{x}=\mathbf{M}^{-1}\mathbf{y}.$$

From the residual

$$\mathbf{r}_0=\mathbf{b}-\mathbf{A}\mathbf{x}_0,\qquad \beta=\|\mathbf{r}_0\|_2,\qquad \mathbf{q}_1=\mathbf{r}_0/\beta,$$

GMRES builds the Krylov space

$$\mathcal{K}_m(\mathbf{A}\mathbf{M}^{-1},\mathbf{r}_0) =\text{span}\{\mathbf{r}_0,(\mathbf{A}\mathbf{M}^{-1})\mathbf{r}_0,\dots,(\mathbf{A}\mathbf{M}^{-1})^{m-1}\mathbf{r}_0\}.$$

Within a restart cycle, HPG-MxP applies the preconditioner,

$\mathbf{z}\gets \mathbf{M}^{-1}\mathbf{q}_k,$

forms

$\mathbf{q}_{k+1}\gets \mathbf{A}\mathbf{z},$

and orthogonalizes with CGS2. At the end of the cycle, it solves the projected upper-triangular system and forms the correction

$$\mathbf{r}\gets \mathbf{Q}\mathbf{t},\qquad \mathbf{x}_m \gets \mathbf{x}_0 + \mathbf{M}^{-1}\mathbf{r}.$$

The benchmark requires the residual computation

$\mathbf{r}\gets \mathbf{b}-\mathbf{A}\mathbf{x}$

and the solution update to be in FP64, while much of the inner GMRES-IR work may run in lower precision. In the exascale implementation reported in (Kashi et al., 15 Jul 2025), the practical focus is on double/single mixed precision, not FP16, BF16, or FP8 (Kashi et al., 15 Jul 2025).

The preconditioner is one cycle of geometric multigrid with a forward Gauss-Seidel smoother. Restriction is simple injection,

$a_{ii}=26$0

and prolongation is $a_{ii}=26$1. This keeps the benchmark centered on sparse, stencil-based kernels—SpMV, smoothing, restriction, prolongation-related work, vector updates, and reductions—whose arithmetic intensity is low and whose performance is dominated by data movement (Kashi et al., 15 Jul 2025).

3. Validation, scoring, and benchmark methodology

HPG-MxP has three benchmark phases: validation, mixed-precision benchmark, and double-precision reference benchmark. Validation measures how much mixed precision changes convergence. By default, double-precision GMRES is run from a zero initial guess until the residual norm is reduced by $a_{ii}=26$2, recording $a_{ii}=26$3, and mixed-precision GMRES-IR is run to the same tolerance, recording $a_{ii}=26$4. On one Frontier node, the reported validation counts were 2305 GMRES iterations for double precision and 2382 for mixed-precision GMRES-IR (Kashi et al., 15 Jul 2025).

The benchmark score is then penalized by the validation ratio: if mixed precision requires more iterations than double precision, the score is reduced; if mixed precision requires fewer iterations, no bonus is given. This operationalizes the benchmark’s emphasis on scientific usefulness rather than raw low-precision rate (Kashi et al., 15 Jul 2025).

The published benchmark parameters are: restart length 30, local mesh size $a_{ii}=26$5, maximum GMRES iterations per solve 300, validation on 8 GCDs, and validation relative tolerance $a_{ii}=26$6. The official runtime is typically 1800 s, reduced to 900 s for runs on at least 1024 nodes (Kashi et al., 15 Jul 2025).

A notable methodological point is that HPG-MxP counts floating-point operations of different precisions equally in the benchmark FLOP model. The reported performance is therefore a mixed-precision benchmark number, not an FP64-only throughput number in the classical LINPACK sense (Kashi et al., 15 Jul 2025).

The benchmark also includes a full-scale validation mode. Reported iteration ratios $a_{ii}=26$7 show that the standard one-node validation is “more or less as stringent” as full-scale validation. For example, the standard ratio was 0.968, while full-scale ratios ranged from 0.958 to 1.067 over runs from 2 to 4096 nodes (Kashi et al., 15 Jul 2025). This supports the benchmark’s claim that its validation mechanism is strong enough to penalize mixed precision reasonably without requiring full-machine convergence testing for every score.

4. Exascale implementation and reported performance

A highly optimized implementation of HPG-MxP was reported for Frontier. Per node, the system provides 1 AMD Milan 64-core CPU and 4 AMD MI250x GPUs, each with 2 GCDs, giving 8 GPUs/GCDs per node and 64 GB HBM per GCD. The implementation scaled to 9408 nodes and 75,264 GPUs/GCDs (Kashi et al., 15 Jul 2025).

The reported implementation introduced several algorithmic enhancements beyond a reference version. These included multicolored Gauss-Seidel using Jones-Plassmann-Luby (JPL) coloring, ELLPACK (ELL) storage instead of CSR for regular stencil matrices, overlap of halo exchange with interior compute using asynchronous GPU streams and MPI, a fused SpMV-restriction kernel enabled by injection restriction, and a redesign that kept mixed-precision operations on the GPU instead of the CPU (Kashi et al., 15 Jul 2025).

The headline result was the first reported 1.6× speedup for HPG-MxP using double/single mixed precision on an exascale GPU system. At full scale, the mixed-precision benchmark delivered 17.23 PFLOP/s on 9408 nodes and 75,264 GPUs/GCDs, with 78% weak-scaling-like efficiency from 1 node to 9408 nodes (Kashi et al., 15 Jul 2025).

The performance analysis showed that the principal kernels sit at the HBM bandwidth limit. Four motifs consumed almost all time: the Gauss-Seidel smoother, CGS2 orthogonalization, SpMV, and multigrid restriction. Mixed precision helped most in orthogonalization, which behaves more like dense BLAS-2 and benefits strongly from reduced floating-point data width. It helped less in SpMV and multigrid / Gauss-Seidel, because those kernels still move index arrays and other sparse-metadata overheads (Kashi et al., 15 Jul 2025).

This is the benchmark’s defining performance lesson. A full transition from FP64 to FP32 in a purely bandwidth-limited dense model might suggest a 2× ceiling, but HPG-MxP measured about 1.6× because sparse kernels also move integer indices, metadata, halo data, and synchronization-related payloads. The benchmark therefore quantifies the gap between ideal low-precision bandwidth scaling and the actually realizable gain on sparse scientific workloads (Kashi et al., 15 Jul 2025).

5. Algorithmic lineage in mixed-precision GMRES research

HPG-MxP sits within a broader research line on mixed-precision GMRES and GMRES-based iterative refinement. A key result for sparse GMRES is that double precision is needed chiefly for residual computation and updating the approximate solution, while the inner correction solve can be performed in single precision, provided restart occurs after each improvement of single-precision accuracy. On a single shared-memory node, that strategy yielded average speedups of 19% for MGS and 24% for CGSR relative to double-precision GMRES while retaining double-precision accuracy (Lindquist et al., 2020). A closely related GPU study likewise identified GMRES-IR as the most effective multiprecision strategy among those tested, with reported speedups up to 1.58× when combined with an fp32 polynomial preconditioner on an NVIDIA V100 (Loe et al., 2021).

Preconditioning is central in this literature. Five-precision GMRES-based iterative refinement with SPAI preconditioning showed that an explicit sparse approximate inverse can be a viable high-performance alternative to LU-based preconditioning, because its application is a sparse matrix-vector product rather than a sparse triangular solve (Carson et al., 2022). The later BSPAI-GMRES-IR method extended this by partitioning SPAI entries by magnitude into multiple precision buckets. Its central analytical conclusion is that when

$a_{ii}=26$8

the backward and forward errors of GMRES with bucketed SPAI application are approximately the same as in the uniform-precision case, while storage and application bandwidth can be reduced; the tradeoff is that aggressive demotion of preconditioner entries can sharply increase GMRES iterations (Khan et al., 2023).

Another strand addressed the repeated inner GMRES solves inside iterative refinement. Mixed Precision GMRES-based Iterative Refinement with Recycling incorporated Krylov subspace recycling into mixed-precision GMRES-IR, exploiting the fact that each refinement step solves a system with the same coefficient matrix but a different right-hand side. The reported effect was often a substantial reduction in total Krylov iterations and, in some cases, improved robustness of restarted GMRES-based refinement (Oktay et al., 2022). Taken together, these studies show that HPG-MxP is not an isolated benchmark design: it operationalizes a family of ideas about where low precision is safe, where higher precision remains essential, and how Krylov and preconditioning costs interact in sparse, bandwidth-bound regimes.

6. Power, energy, and interpretive limits

HPG-MxP has also been used as a case study for fine-grained power and energy attribution on AMD exascale systems. In that context it is described as a benchmark that “stresses the memory hierarchy with sparse linear-algebra phases across precisions,” provides “stable, repeatable memory bandwidth-bound segments,” and has clear phase boundaries such as plan/allocate, warm-up, steady iterative solver loops, and finalize (McDaniel et al., 7 Apr 2026). This phase structure makes it suitable for time-resolved power analysis in a way that dense HPL-like plateaus are not.

Using energy-counter-derived instantaneous power and time-aligned attribution, mixed precision was found to reduce node energy on Frontier by 31% for HPG-MxP. On Portage, across 128 independent runs on 128 nodes, average node energy decreased from 55.2 kJ in full precision to 40.3 kJ in mixed precision, a 27% reduction (McDaniel et al., 7 Apr 2026). The reported interpretation is that, unlike rocHPL-MxP where energy savings are dominated by a shorter runtime, HPG-MxP savings arise from both runtime compression and modestly lower average APU power (McDaniel et al., 7 Apr 2026).

Several limitations shape how HPG-MxP results should be interpreted. The benchmark matrix is synthetic and structured, so application speedups depend in practice on matrix conditioning and pseudospectra (Kashi et al., 15 Jul 2025). The benchmark requires duplicate precision contexts, so memory-footprint comparisons with pure FP64 are not entirely trivial (Kashi et al., 15 Jul 2025). The power-analysis work also emphasizes that filtered vendor power counters are too coarse for HPG-MxP’s short phases; phase-level attribution should instead use power reconstructed from cumulative energy counters and validated against slower off-chip sensors (McDaniel et al., 7 Apr 2026).

A common misconception is therefore to treat HPG-MxP as a sparse analogue of dense mixed-precision LINPACK that simply rewards lower precision. Its actual purpose is narrower and more stringent: it measures whether reduced precision can be converted into real sparse-solver speedups under a convergence constraint. Another misconception is to equate its score with direct application performance. The benchmark is a controlled sparse GMRES-IR workload, not a substitute for application-specific analysis. What it contributes is a reproducible measure of how a machine’s memory system, sparse-kernel implementation quality, reduction performance, and mixed-precision software stack interact on a scientifically constrained GMRES-based solver (Kashi et al., 15 Jul 2025).