---
title: GMRES Algorithm for Solving Linear Systems
url: https://www.emergentmind.com/topics/generalized-minimum-residual-gmres-algorithm
type: topic
---

# GMRES Algorithm for Solving Linear Systems

The Generalized Minimum RESidual (GMRES) algorithm is an iterative Krylov subspace method for solving square (often large, non-symmetric, and possibly indefinite) linear systems of the form $A x = b$ with $A \in \mathbb{R}^{n \times n}$ and $b \in \mathbb{R}^n$. GMRES is characterized by selecting at each iteration the approximate solution that minimizes the $\ell_2$-norm of the residual over an affine Krylov subspace. It is notable for its broad applicability across scientific computing, signal processing, inverse problems, computational imaging, and large-scale engineering simulation domains.

## 1. Core Algorithmic Structure

GMRES seeks the vector $x_m \in x_0 + \mathcal{K}_m(A, r_0)$, where $r_0 = b - A x_0$ and $\mathcal{K}_m(A, r_0) = \operatorname{span}\{r_0, A r_0, A^2 r_0, \ldots, A^{m-1} r_0\}$, such that $\|b - A x_m\|_2$ is minimized. The methodology leverages the Arnoldi process to construct an orthonormal basis $V_{m+1} = [v_1, \ldots, v_{m+1}]$ of $\mathcal{K}_{m+1}(A, r_0)$, resulting in the Arnoldi relation
$$
A V_m = V_{m+1} H_m,
$$
where $H_m \in \mathbb{R}^{(m+1) \times m}$ is upper Hessenberg and $V_m = [v_1, \ldots, v_m]$.

The approximate solution is represented as $x_m = x_0 + V_m y_m$, where $y_m$ solves the small least-squares problem:
$$
y_m = \arg\min_{y \in \mathbb{R}^m} \| \beta e_1 - H_m y \|_2,
$$
with $\beta = \|r_0\|_2$ and $e_1=(1,0,\ldots,0)^T$. The true residual norm is exactly given by $\| b - A x_m \|_2 = \| \beta e_1 - H_m y_m \|_2$ due to the orthonormality of $V_{m+1}$ [2201.07408, 2110.04017].

## 2. Preconditioning and Variants

GMRES accommodates several preconditioning strategies:

- **Left Preconditioning:** Replace $A$ with $M^{-1}A$, solve $M^{-1}A x = M^{-1}b$.
- **Right Preconditioning:** Solve $A M^{-1} y = b$; the approximate solution is $x = M^{-1} y$.
- **Split (Two-sided) Preconditioning:** With preconditioners $M_L$ and $M_R$, solve $M_L^{-1} A M_R^{-1} y = M_L^{-1} b$, and recover $x = M_R^{-1} y$.

At each step, only matvecs with $A$ and $M^{-1}$ are required; no transposes or normal equations are formed [2201.07408, 1806.06599]. In computed tomography and inverse problems, GMRES is uniquely robust against "unmatched" pairs ($B \neq A^T$) through variants such as AB-GMRES and BA-GMRES, which remain monotonically convergent in the relevant seminorms, in contrast to CGLS which may diverge [2201.07408].

Polynomial preconditioning can be implemented using the GMRES residual polynomial itself. The polynomial is determined during a preliminary GMRES cycle, and its roots (harmonic Ritz values) are used to factor and efficiently apply the preconditioner. Stability measures such as root-adding mitigate adverse effects from outlying eigenvalues [1911.07065].

## 3. Restarting, Block, and Advanced Implementations

Given the prohibitive memory and orthogonalization costs for large $m$, restarted GMRES($m$) discards the Krylov basis after $m$ steps and resumes with the latest iterate. This may deteriorate global optimality and induce stagnation, which is mitigated by subspace recycling (deflation/augmentation). GMRES-DR, GCRO-DR, and flexible GMRES (FGMRES) retain additional spectral information, enabling superior convergence in challenging regimes [2110.04017].

Extensions include:

- **Block GMRES:** Construction of block Krylov subspaces for multiple right-hand sides. Block Arnoldi and block Hessenberg structures accelerate convergence and enable recycling across multiple systems [2408.05513, 2110.04017].
- **s-step (Communication Avoiding) GMRES:** Bundling $s$ Arnoldi steps, forming basis blocks by polynomial powers or matrix function applications to exploit memory hierarchy and minimize global synchronizations [2001.04886, 2110.04017].
- **Parallel and Low-Synchronization Variants:** Iterated Gauss-Seidel GMRES (IGS-GMRES) achieves $O(1)$ synchronizations/iteration, preserving orthogonality to $O(\epsilon)$, essential for exascale architectures [2205.07805].

## 4. Convergence Analysis

Convergence of GMRES is governed by the spectral properties and field-of-values (numerical range) of $A$:

- If $A$ is diagonalizable, the residual satisfies
$$
\|r_m\|_2 \leq \kappa(X) \min_{p \in \mathcal{P}_m,\, p(0)=1} \max_{i} |p(\lambda_i)| \|r_0\|_2,
$$
where $A = X \Lambda X^{-1}$ [2110.04017].
- For normal $A$, Chebyshev bounds are sharp. For non-normal systems, field-of-values bounds are used.
- If the numerical range $W(A)$ avoids the origin, Elman’s bound provides geometric decay:
$$
\frac{\|r_k\|_2}{\|b\|_2} \leq \left(1 - \frac{\mu(A)^2}{\|A\|_2^2}\right)^{k/2},
$$
with $\mu(A) = \lambda_{\min} ((A+A^*)/2)$ [2312.15022].
- If $0 \in W(A)$, no monotonicity is guaranteed. Lyapunov-based inner products can restore geometric convergence at the cost of norm distortion [2312.15022].

For singular $A$, if the range-symmetry condition $R(A) = R(A^T)$ holds, GMRES provides a least-squares solution in at most rank$(A)$ steps [2009.00371].

## 5. Numerical Stability, Orthogonality, and Parallel Performance

The standard implementation of GMRES uses modified Gram-Schmidt orthogonalization, with backward stability established up to $O(\epsilon)\kappa(B_k)$ [2205.07805]. IGS-GMRES, using two Gauss-Seidel sweeps, maintains $O(\epsilon)$ orthogonality, eliminates stagnation for highly non-normal matrices, preserves the minimal singular value of the Krylov basis near unity, and is backward-stable [2205.07805].

Low synchronization and block/grouped dot-products enable deployment on large-scale parallel hardware and GPU clusters. For instance, IGS-GMRES achieves up to $2 \times$ speedup relative to vanilla MGS-GMRES due to reduced communications. s-step and pipelined GMRES approaches exploit cache locality and overlap communications, further improving performance [2205.07805, 2001.04886, 2110.04017].

## 6. Applications and Specialized Contexts

GMRES is widely used in scientific computing (CFD, MIMO detection, CT reconstruction), especially in settings with non-self-adjoint or ill-conditioned operators:

- **Computed Tomography (CT):** BA-GMRES and AB-GMRES enable reconstruction with unmatched projector/back-projector pairs and regularization by early stopping (semi-convergence phenomenon) [2201.07408].
- **Massive MIMO Detection:** GMRES (and the related conjugate residual method) achieves near-Cholesky MMSE performance at a fraction of the computational cost, without preconditioning [1802.05982].
- **Linear Discrete Ill-posed Problems:** Preconditioned and Arnoldi-based regularized versions (Arnoldi-Tikhonov, Arnoldi-TSVD) mitigate semi-convergence and stabilize ill-posed inversions [1806.06599].
- **Global/Block/Randomized GMRES:** RGl-GMRES enables efficient Krylov subspace construction for very large numbers of right-hand sides using sketching techniques, providing nearly identical convergence with significant computational cost reduction [2602.14786].

## 7. Multiright-Hand Side and Flexible Extensions

Recent variants address the case of dynamically arriving or multiple right-hand sides. Modern algorithms maintain a single orthonormal basis encapsulating the union of Krylov spaces across solutions, thus achieving improved storage efficiency and robustly supporting flexible and deflated restart strategies. Such extensions subsume GCR-type recurrences, enabling both flexible preconditioning and spectral augmentation within a streamlined computational framework [2408.05513].

---

**References:**

- [2201.07408] Iterative image reconstruction for CT with unmatched projection matrices using the generalized minimal residual algorithm
- [2205.07805] Iterated Gauss-Seidel GMRES
- [1802.05982] Residual-Based Detections and Unified Architecture for Massive MIMO Uplink
- [2312.15022] Extending Elman's Bound for GMRES
- [1911.07065] Toward Efficient Polynomial Preconditioning for GMRES
- [2001.04886] s-Step Orthomin and GMRES implemented on parallel computers
- [2110.04017] GMRES algorithms over 35 years
- [2009.00371] GMRES on singular systems revisited
- [2602.14786] New Randomized Global Generalized Minimum Residual (RGl-GMRES) method
- [2408.05513] Generalized minimal residual method for systems with multiple right-hand sides
- [1806.06599] Arnoldi decomposition, GMRES, and preconditioning for linear discrete ill-posed problems
- [2506.01081] Convergence Analysis of An Alternating Nonlinear GMRES on Linear Systems

Source: https://www.emergentmind.com/topics/generalized-minimum-residual-gmres-algorithm