---
title: High Performance GMRES Mixed Precision
url: https://www.emergentmind.com/topics/high-performance-gmres-mixed-precision-hpg-mxp
type: topic
---

# High Performance GMRES Mixed Precision

High Performance GMRES Mixed Precision (HPG-MxP) is a benchmark for **useful mixed-precision performance** on **sparse, memory-bandwidth-limited scientific workloads**. It is designed to measure how effectively a high-performance computing system converts reduced precision into speedup for a GMRES-based sparse solver while preserving convergence behavior close to a double-precision reference, rather than merely counting low-precision arithmetic throughput [2507.11512]. In the broader mixed-precision benchmark ecosystem, it is positioned alongside HPL-AI and HPL-MxP, but it emphasizes **sparse linear algebra / GMRES-like iterative behavior** rather than dense LU factorization [2509.19618].

## 1. Definition and benchmark objective

HPG-MxP was proposed because many scientific simulation workloads are **memory-bandwidth limited**, not compute limited, so the large low-precision throughput gains seen in dense AI-style kernels do not automatically translate into practical solver acceleration. The benchmark therefore asks a narrower and more application-relevant question: what useful speedup can an HPC machine deliver for sparse mixed-precision scientific computing, under a requirement of “essentially the same usefulness as double-precision computation” [2507.11512].

The benchmark targets sparse matrix-based simulation motifs typical of PDE solvers. Like HPCG, it uses a structured Poisson-like problem on a uniform mesh, but it replaces Conjugate Gradient with **GMRES**, which extends relevance to nonsymmetric problems and changes the dominant cost structure through Krylov-basis storage, orthogonalization, and restart behavior. This makes HPG-MxP a memory-wall benchmark for sparse iterative solvers rather than a dense-factorization benchmark [2507.11512].

A central feature is that HPG-MxP does not reward mixed precision simply for being faster. The mixed-precision score is reduced if mixed precision requires more iterations than the double-precision reference to reach the target residual reduction. This makes HPG-MxP a benchmark of **useful performance**, not raw mixed-precision flop rate [2507.11512].

## 2. Linear system, solver, and prescribed numerical structure

HPG-MxP solves a sparse linear system
\[
\mathbf{A}\mathbf{x}=\mathbf{b}.
\]
The preferred benchmark matrix is the same symmetric weakly diagonally dominant matrix as in HPCG, with diagonal entries \(a_{ii}=26\) and off-diagonal stencil entries \(a_{ij}=-1\). The benchmark allows an optional nonsymmetric matrix, but the symmetric version is noted to be at least as hard for GMRES [2507.11512].

The prescribed algorithm is **right-preconditioned restarted GMRES with iterative refinement**, specifically a **GMRES-IR** formulation with **CGS2 orthogonalization**. The right-preconditioned system is
\[
\mathbf{A}\mathbf{M}^{-1}\mathbf{y}=\mathbf{b}, \qquad \mathbf{x}=\mathbf{M}^{-1}\mathbf{y}.
\]
From the residual
\[
\mathbf{r}_0=\mathbf{b}-\mathbf{A}\mathbf{x}_0,\qquad \beta=\|\mathbf{r}_0\|_2,\qquad \mathbf{q}_1=\mathbf{r}_0/\beta,
\]
GMRES builds the Krylov space
\[
\mathcal{K}_m(\mathbf{A}\mathbf{M}^{-1},\mathbf{r}_0)
=\text{span}\{\mathbf{r}_0,(\mathbf{A}\mathbf{M}^{-1})\mathbf{r}_0,\dots,(\mathbf{A}\mathbf{M}^{-1})^{m-1}\mathbf{r}_0\}.
\]
Within a restart cycle, HPG-MxP applies the preconditioner,
\[
\mathbf{z}\gets \mathbf{M}^{-1}\mathbf{q}_k,
\]
forms
\[
\mathbf{q}_{k+1}\gets \mathbf{A}\mathbf{z},
\]
and orthogonalizes with **CGS2**. At the end of the cycle, it solves the projected upper-triangular system and forms the correction
\[
\mathbf{r}\gets \mathbf{Q}\mathbf{t},\qquad
\mathbf{x}_m \gets \mathbf{x}_0 + \mathbf{M}^{-1}\mathbf{r}.
\]
The benchmark requires the residual computation
\[
\mathbf{r}\gets \mathbf{b}-\mathbf{A}\mathbf{x}
\]
and the solution update to be in **FP64**, while much of the inner GMRES-IR work may run in lower precision. In the exascale implementation reported in [2507.11512], the practical focus is on **double/single mixed precision**, not FP16, BF16, or FP8 [2507.11512].

The preconditioner is one cycle of **geometric multigrid** with a **forward Gauss-Seidel smoother**. Restriction is simple injection,
\[
(\mathbf{R}\mathbf{v})_i := v_{c_f(i)},
\]
and prolongation is \(\mathbf{P}=\mathbf{R}^T\). This keeps the benchmark centered on sparse, stencil-based kernels—SpMV, smoothing, restriction, prolongation-related work, vector updates, and reductions—whose arithmetic intensity is low and whose performance is dominated by data movement [2507.11512].

## 3. Validation, scoring, and benchmark methodology

HPG-MxP has three benchmark phases: **validation**, **mixed-precision benchmark**, and **double-precision reference benchmark**. Validation measures how much mixed precision changes convergence. By default, double-precision GMRES is run from a zero initial guess until the residual norm is reduced by \(10^{-9}\), recording \(n_d\), and mixed-precision GMRES-IR is run to the same tolerance, recording \(n_{ir}\). On one Frontier node, the reported validation counts were **2305** GMRES iterations for double precision and **2382** for mixed-precision GMRES-IR [2507.11512].

The benchmark score is then penalized by the validation ratio: if mixed precision requires more iterations than double precision, the score is reduced; if mixed precision requires fewer iterations, no bonus is given. This operationalizes the benchmark’s emphasis on scientific usefulness rather than raw low-precision rate [2507.11512].

The published benchmark parameters are: restart length **30**, local mesh size **\(320^3\)**, maximum GMRES iterations per solve **300**, validation on **8 GCDs**, and validation relative tolerance **\(10^{-9}\)**. The official runtime is typically **1800 s**, reduced to **900 s** for runs on at least **1024 nodes** [2507.11512].

A notable methodological point is that HPG-MxP counts floating-point operations of different precisions equally in the benchmark FLOP model. The reported performance is therefore a mixed-precision benchmark number, not an FP64-only throughput number in the classical LINPACK sense [2507.11512].

The benchmark also includes a full-scale validation mode. Reported iteration ratios \(n_d/n_{ir}\) show that the standard one-node validation is “more or less as stringent” as full-scale validation. For example, the standard ratio was **0.968**, while full-scale ratios ranged from **0.958** to **1.067** over runs from **2** to **4096** nodes [2507.11512]. This supports the benchmark’s claim that its validation mechanism is strong enough to penalize mixed precision reasonably without requiring full-machine convergence testing for every score.

## 4. Exascale implementation and reported performance

A highly optimized implementation of HPG-MxP was reported for **Frontier**. Per node, the system provides **1 AMD Milan 64-core CPU** and **4 AMD MI250x GPUs**, each with **2 GCDs**, giving **8 GPUs/GCDs per node** and **64 GB HBM per GCD**. The implementation scaled to **9408 nodes** and **75,264 GPUs/GCDs** [2507.11512].

The reported implementation introduced several algorithmic enhancements beyond a reference version. These included **multicolored Gauss-Seidel** using **Jones-Plassmann-Luby (JPL)** coloring, **ELLPACK (ELL)** storage instead of CSR for regular stencil matrices, overlap of halo exchange with interior compute using asynchronous GPU streams and MPI, a **fused SpMV-restriction** kernel enabled by injection restriction, and a redesign that kept mixed-precision operations on the GPU instead of the CPU [2507.11512].

The headline result was the first reported **1.6× speedup** for HPG-MxP using **double/single mixed precision** on an exascale GPU system. At full scale, the mixed-precision benchmark delivered **17.23 PFLOP/s** on **9408 nodes** and **75,264 GPUs/GCDs**, with **78%** weak-scaling-like efficiency from **1 node** to **9408 nodes** [2507.11512].

The performance analysis showed that the principal kernels sit at the **HBM bandwidth limit**. Four motifs consumed almost all time: the **Gauss-Seidel smoother**, **CGS2 orthogonalization**, **SpMV**, and **multigrid restriction**. Mixed precision helped most in **orthogonalization**, which behaves more like dense BLAS-2 and benefits strongly from reduced floating-point data width. It helped less in **SpMV** and **multigrid / Gauss-Seidel**, because those kernels still move index arrays and other sparse-metadata overheads [2507.11512].

This is the benchmark’s defining performance lesson. A full transition from FP64 to FP32 in a purely bandwidth-limited dense model might suggest a **2×** ceiling, but HPG-MxP measured about **1.6×** because sparse kernels also move integer indices, metadata, halo data, and synchronization-related payloads. The benchmark therefore quantifies the gap between ideal low-precision bandwidth scaling and the actually realizable gain on sparse scientific workloads [2507.11512].

## 5. Algorithmic lineage in mixed-precision GMRES research

HPG-MxP sits within a broader research line on mixed-precision GMRES and GMRES-based iterative refinement. A key result for sparse GMRES is that double precision is needed chiefly for **residual computation** and **updating the approximate solution**, while the inner correction solve can be performed in single precision, provided restart occurs after each improvement of single-precision accuracy. On a single shared-memory node, that strategy yielded average speedups of **19%** for MGS and **24%** for CGSR relative to double-precision GMRES while retaining double-precision accuracy [2011.01850]. A closely related GPU study likewise identified **GMRES-IR** as the most effective multiprecision strategy among those tested, with reported speedups up to **1.58×** when combined with an fp32 polynomial preconditioner on an NVIDIA V100 [2109.01232].

Preconditioning is central in this literature. Five-precision GMRES-based iterative refinement with **SPAI** preconditioning showed that an explicit sparse approximate inverse can be a viable high-performance alternative to LU-based preconditioning, because its application is a sparse matrix-vector product rather than a sparse triangular solve [2202.10204]. The later **BSPAI-GMRES-IR** method extended this by partitioning SPAI entries by magnitude into multiple precision buckets. Its central analytical conclusion is that when
\[
u_1 \approx \epsilon \approx u_p,
\]
the backward and forward errors of GMRES with bucketed SPAI application are approximately the same as in the uniform-precision case, while storage and application bandwidth can be reduced; the tradeoff is that aggressive demotion of preconditioner entries can sharply increase GMRES iterations [2307.03914].

Another strand addressed the repeated inner GMRES solves inside iterative refinement. **Mixed Precision GMRES-based Iterative Refinement with Recycling** incorporated Krylov subspace recycling into mixed-precision GMRES-IR, exploiting the fact that each refinement step solves a system with the same coefficient matrix but a different right-hand side. The reported effect was often a substantial reduction in total Krylov iterations and, in some cases, improved robustness of restarted GMRES-based refinement [2201.09827]. Taken together, these studies show that HPG-MxP is not an isolated benchmark design: it operationalizes a family of ideas about where low precision is safe, where higher precision remains essential, and how Krylov and preconditioning costs interact in sparse, bandwidth-bound regimes.

## 6. Power, energy, and interpretive limits

HPG-MxP has also been used as a case study for fine-grained power and energy attribution on AMD exascale systems. In that context it is described as a benchmark that “**stresses the memory hierarchy with sparse linear-algebra phases across precisions**,” provides “**stable, repeatable memory bandwidth-bound segments**,” and has clear phase boundaries such as plan/allocate, warm-up, steady iterative solver loops, and finalize [2604.06056]. This phase structure makes it suitable for time-resolved power analysis in a way that dense HPL-like plateaus are not.

Using energy-counter-derived instantaneous power and time-aligned attribution, mixed precision was found to reduce node energy on **Frontier** by **31%** for HPG-MxP. On **Portage**, across **128 independent runs on 128 nodes**, average node energy decreased from **55.2 kJ** in full precision to **40.3 kJ** in mixed precision, a **27%** reduction [2604.06056]. The reported interpretation is that, unlike rocHPL-MxP where energy savings are dominated by a shorter runtime, HPG-MxP savings arise from both **runtime compression** and **modestly lower average APU power** [2604.06056].

Several limitations shape how HPG-MxP results should be interpreted. The benchmark matrix is synthetic and structured, so application speedups depend in practice on matrix conditioning and pseudospectra [2507.11512]. The benchmark requires duplicate precision contexts, so memory-footprint comparisons with pure FP64 are not entirely trivial [2507.11512]. The power-analysis work also emphasizes that filtered vendor power counters are too coarse for HPG-MxP’s short phases; phase-level attribution should instead use power reconstructed from cumulative energy counters and validated against slower off-chip sensors [2604.06056].

A common misconception is therefore to treat HPG-MxP as a sparse analogue of dense mixed-precision LINPACK that simply rewards lower precision. Its actual purpose is narrower and more stringent: it measures whether reduced precision can be converted into **real sparse-solver speedups** under a convergence constraint. Another misconception is to equate its score with direct application performance. The benchmark is a controlled sparse GMRES-IR workload, not a substitute for application-specific analysis. What it contributes is a reproducible measure of how a machine’s memory system, sparse-kernel implementation quality, reduction performance, and mixed-precision software stack interact on a scientifically constrained GMRES-based solver [2507.11512].

Source: https://www.emergentmind.com/topics/high-performance-gmres-mixed-precision-hpg-mxp