GMRES Algorithm for Solving Linear Systems

• GMRES is an iterative Krylov subspace method that minimizes the ℓ2-norm of the residual for solving large, non-symmetric linear systems.
• It uses the Arnoldi process to construct an orthonormal basis and solves a reduced least-squares problem, enabling robust preconditioning and convergence analysis.
• Restarted, block, and s-step variants optimize memory usage and parallel performance, making GMRES suitable for high-performance scientific computing applications.

The Generalized Minimum RESidual (GMRES) algorithm is an iterative Krylov subspace method for solving square (often large, non-symmetric, and possibly indefinite) linear systems of the form $A x = b$ with $A \in \mathbb{R}^{n \times n}$ and $b \in \mathbb{R}^n$. GMRES is characterized by selecting at each iteration the approximate solution that minimizes the $\ell_2$-norm of the residual over an affine Krylov subspace. It is notable for its broad applicability across scientific computing, signal processing, inverse problems, computational imaging, and large-scale engineering simulation domains.

1. Core Algorithmic Structure

GMRES seeks the vector $x_m \in x_0 + \mathcal{K}_m(A, r_0)$, where $r_0 = b - A x_0$ and $$\mathcal{K}_m(A, r_0) = \operatorname{span}\{r_0, A r_0, A^2 r_0, \ldots, A^{m-1} r_0\}$$, such that $\|b - A x_m\|_2$ is minimized. The methodology leverages the Arnoldi process to construct an orthonormal basis $V_{m+1} = [v_1, \ldots, v_{m+1}]$ of $\mathcal{K}_{m+1}(A, r_0)$, resulting in the Arnoldi relation

$A \in \mathbb{R}^{n \times n}$0

where $A \in \mathbb{R}^{n \times n}$1 is upper Hessenberg and $A \in \mathbb{R}^{n \times n}$2.

The approximate solution is represented as $A \in \mathbb{R}^{n \times n}$3, where $A \in \mathbb{R}^{n \times n}$4 solves the small least-squares problem:

$A \in \mathbb{R}^{n \times n}$5

with $A \in \mathbb{R}^{n \times n}$6 and $A \in \mathbb{R}^{n \times n}$7. The true residual norm is exactly given by $A \in \mathbb{R}^{n \times n}$8 due to the orthonormality of $A \in \mathbb{R}^{n \times n}$9 (Sidky et al., 2022, Zou, 2021).

2. Preconditioning and Variants

GMRES accommodates several preconditioning strategies:

• Left Preconditioning: Replace $b \in \mathbb{R}^n$0 with $b \in \mathbb{R}^n$1, solve $b \in \mathbb{R}^n$2.
• Right Preconditioning: Solve $b \in \mathbb{R}^n$3; the approximate solution is $b \in \mathbb{R}^n$4.
• Split (Two-sided) Preconditioning: With preconditioners $b \in \mathbb{R}^n$5 and $b \in \mathbb{R}^n$6, solve $b \in \mathbb{R}^n$7, and recover $b \in \mathbb{R}^n$8.

At each step, only matvecs with $b \in \mathbb{R}^n$9 and $\ell_2$0 are required; no transposes or normal equations are formed (Sidky et al., 2022, Gazzola et al., 2018). In computed tomography and inverse problems, GMRES is uniquely robust against "unmatched" pairs ($\ell_2$1) through variants such as AB-GMRES and BA-GMRES, which remain monotonically convergent in the relevant seminorms, in contrast to CGLS which may diverge (Sidky et al., 2022).

Polynomial preconditioning can be implemented using the GMRES residual polynomial itself. The polynomial is determined during a preliminary GMRES cycle, and its roots (harmonic Ritz values) are used to factor and efficiently apply the preconditioner. Stability measures such as root-adding mitigate adverse effects from outlying eigenvalues (Loe et al., 2019).

3. Restarting, Block, and Advanced Implementations

Given the prohibitive memory and orthogonalization costs for large $\ell_2$2, restarted GMRES($\ell_2$3) discards the Krylov basis after $\ell_2$4 steps and resumes with the latest iterate. This may deteriorate global optimality and induce stagnation, which is mitigated by subspace recycling (deflation/augmentation). GMRES-DR, GCRO-DR, and flexible GMRES (FGMRES) retain additional spectral information, enabling superior convergence in challenging regimes (Zou, 2021).

Extensions include:

• Block GMRES: Construction of block Krylov subspaces for multiple right-hand sides. Block Arnoldi and block Hessenberg structures accelerate convergence and enable recycling across multiple systems (Sukmanyuk et al., 2024, Zou, 2021).
• s-step (Communication Avoiding) GMRES: Bundling $\ell_2$5 Arnoldi steps, forming basis blocks by polynomial powers or matrix function applications to exploit memory hierarchy and minimize global synchronizations (Chronopoulos et al., 2020, Zou, 2021).
• Parallel and Low-Synchronization Variants: Iterated Gauss-Seidel GMRES (IGS-GMRES) achieves $\ell_2$6 synchronizations/iteration, preserving orthogonality to $\ell_2$7, essential for exascale architectures (Thomas et al., 2022).

4. Convergence Analysis

Convergence of GMRES is governed by the spectral properties and field-of-values (numerical range) of $\ell_2$8:

• If $\ell_2$9 is diagonalizable, the residual satisfies

$x_m \in x_0 + \mathcal{K}_m(A, r_0)$0

where $x_m \in x_0 + \mathcal{K}_m(A, r_0)$1 (Zou, 2021).

• For normal $x_m \in x_0 + \mathcal{K}_m(A, r_0)$2, Chebyshev bounds are sharp. For non-normal systems, field-of-values bounds are used.
• If the numerical range $x_m \in x_0 + \mathcal{K}_m(A, r_0)$3 avoids the origin, Elman’s bound provides geometric decay:

$x_m \in x_0 + \mathcal{K}_m(A, r_0)$4

with $x_m \in x_0 + \mathcal{K}_m(A, r_0)$5 (Embree, 2023).

• If $x_m \in x_0 + \mathcal{K}_m(A, r_0)$6, no monotonicity is guaranteed. Lyapunov-based inner products can restore geometric convergence at the cost of norm distortion (Embree, 2023).

For singular $x_m \in x_0 + \mathcal{K}_m(A, r_0)$7, if the range-symmetry condition $x_m \in x_0 + \mathcal{K}_m(A, r_0)$8 holds, GMRES provides a least-squares solution in at most rank$x_m \in x_0 + \mathcal{K}_m(A, r_0)$9 steps (Hayami et al., 2020).

5. Numerical Stability, Orthogonality, and Parallel Performance

The standard implementation of GMRES uses modified Gram-Schmidt orthogonalization, with backward stability established up to $r_0 = b - A x_0$0 (Thomas et al., 2022). IGS-GMRES, using two Gauss-Seidel sweeps, maintains $r_0 = b - A x_0$1 orthogonality, eliminates stagnation for highly non-normal matrices, preserves the minimal singular value of the Krylov basis near unity, and is backward-stable (Thomas et al., 2022).

Low synchronization and block/grouped dot-products enable deployment on large-scale parallel hardware and GPU clusters. For instance, IGS-GMRES achieves up to $r_0 = b - A x_0$2 speedup relative to vanilla MGS-GMRES due to reduced communications. s-step and pipelined GMRES approaches exploit cache locality and overlap communications, further improving performance (Thomas et al., 2022, Chronopoulos et al., 2020, Zou, 2021).

6. Applications and Specialized Contexts

GMRES is widely used in scientific computing (CFD, MIMO detection, CT reconstruction), especially in settings with non-self-adjoint or ill-conditioned operators:

• Computed Tomography (CT): BA-GMRES and AB-GMRES enable reconstruction with unmatched projector/back-projector pairs and regularization by early stopping (semi-convergence phenomenon) (Sidky et al., 2022).
• Massive MIMO Detection: GMRES (and the related conjugate residual method) achieves near-Cholesky MMSE performance at a fraction of the computational cost, without preconditioning (Zhang et al., 2018).
• Linear Discrete Ill-posed Problems: Preconditioned and Arnoldi-based regularized versions (Arnoldi-Tikhonov, Arnoldi-TSVD) mitigate semi-convergence and stabilize ill-posed inversions (Gazzola et al., 2018).
• Global/Block/Randomized GMRES: RGl-GMRES enables efficient Krylov subspace construction for very large numbers of right-hand sides using sketching techniques, providing nearly identical convergence with significant computational cost reduction (Badahmane et al., 16 Feb 2026).

7. Multiright-Hand Side and Flexible Extensions

Recent variants address the case of dynamically arriving or multiple right-hand sides. Modern algorithms maintain a single orthonormal basis encapsulating the union of Krylov spaces across solutions, thus achieving improved storage efficiency and robustly supporting flexible and deflated restart strategies. Such extensions subsume GCR-type recurrences, enabling both flexible preconditioning and spectral augmentation within a streamlined computational framework (Sukmanyuk et al., 2024).

---

References:

• (Sidky et al., 2022) Iterative image reconstruction for CT with unmatched projection matrices using the generalized minimal residual algorithm
• (Thomas et al., 2022) Iterated Gauss-Seidel GMRES
• (Zhang et al., 2018) Residual-Based Detections and Unified Architecture for Massive MIMO Uplink
• (Embree, 2023) Extending Elman's Bound for GMRES
• (Loe et al., 2019) Toward Efficient Polynomial Preconditioning for GMRES
• (Chronopoulos et al., 2020) s-Step Orthomin and GMRES implemented on parallel computers
• (Zou, 2021) GMRES algorithms over 35 years
• (Hayami et al., 2020) GMRES on singular systems revisited
• (Badahmane et al., 16 Feb 2026) New Randomized Global Generalized Minimum Residual (RGl-GMRES) method
• (Sukmanyuk et al., 2024) Generalized minimal residual method for systems with multiple right-hand sides
• (Gazzola et al., 2018) Arnoldi decomposition, GMRES, and preconditioning for linear discrete ill-posed problems
• (He, 1 Jun 2025) Convergence Analysis of An Alternating Nonlinear GMRES on Linear Systems