Flexible GMRES Method

• Flexible GMRES is a Krylov subspace method that generalizes GMRES by allowing varying, nonlinear, or inexact preconditioners to adapt iteratively.
• It incorporates a modified Arnoldi process and dynamic preconditioning to enable tailored convergence curves in complex scientific and engineering computations.
• The method’s flexibility facilitates robust and scalable solvers for applications such as multi-physics simulations and high-performance computing environments.

Flexible GMRES is a class of Krylov subspace methods generalizing the Generalized Minimal Residual (GMRES) algorithm to allow for varying, nonlinear, or inexact preconditioners that can adapt at every iteration. Unlike standard GMRES—which presumes a fixed (often linear) preconditioner—Flexible GMRES (FGMRES) is structured to accommodate the iterative, algorithmic, or nonlinear nature of state-of-the-art preconditioning strategies encountered in complex scientific and engineering computations. FGMRES finds wide application in scenarios where dynamic, multilevel, or block-based preconditioners are needed, as well as in high-performance computing environments demanding extreme scalability and robustness.

1. Conceptual Framework of Flexible GMRES

Flexible GMRES maintains the fundamental structure of GMRES but removes the requirement that the preconditioner is stationary across iterations. Denote the linear system as $Ax = b$, and let $\{M_j^{-1}\}$ denote a preconditioner (possibly nonlinear or changing) applied at the $j$th iteration. The method updates the Krylov basis with search directions subject to $u_j = M_j^{-1} v_j$, where $v_j$ is the $j$th basis vector from the (modified) Arnoldi process. This process allows the preconditioner to reflect arbitrary changes, including inexact inner solves, evolving multilevel decompositions, or data-driven procedure updates (1511.07226).

The general update may be summarized as:

• $r_0 = b - Ax_0$,
• At each step, given $v_j$, compute $u_j = M_j^{-1} v_j$ (where $M_j$ may be nonlinear/inexact),
• Set $z_j = Au_j$,
• Orthogonalize $z_j$ against earlier vectors to build the next $v_{j+1}$,
• Construct the approximate solution in the basis $\{u_0, u_1, ..., u_j\}$ by minimizing the residual.

This formulation enables powerful preconditioning not attainable with fixed preconditioners, such as adaptively-tuned inner solvers, randomization/sketching, or data-driven strategies.

2. Theoretical Flexibility and Convergence Properties

A haLLMark of FGMRES and, more generally, of the GMRES paradigm is the remarkable flexibility in achievable convergence behavior. A key theoretical result is that any prescribed nonincreasing sequence of residual norms (i.e., any "convergence curve") can be realized by GMRES for a suitably constructed system, irrespective of the eigenvalue distribution (2506.17193). This property further extends to Weighted GMRES, where the residual norm can be defined with respect to any Hermitian positive-definite (hpd) matrix $M$ (i.e., in the $M$-induced inner product).

The main results include:

• For any linear system $(A, b)$ and any nonincreasing sequence $\{r_k\}$, there exists a weight matrix $M$ such that GMRES in the $M$-inner product produces exactly that residual history.
• A simultaneous prescription of two convergence curves is possible for standard and weighted GMRES: there exists $M$ such that GMRES and $M$-GMRES will achieve two user-specified nonincreasing sequences, contingent on explicit factorization constraints (governed by an upper triangular matrix $T$ satisfying singular value conditions).
• Any two convergence curves can be realized simultaneously for left and right preconditioned GMRES—even with arbitrarily chosen spectra for the preconditioned matrices (2506.17193).

These results underscore the inherent malleability of FGMRES: the convergence trajectory is determined not only by spectral properties, but also by the choice of inner product, preconditioner, and their dynamic tuning throughout iterations.

3. Weighted, Preconditioned, and Split-Preconditioned GMRES

A weight matrix $M$ defines a new inner product $\langle x, y \rangle_M = y^* M x$, leading to the weighted GMRES method ("$M$-GMRES") that minimizes the residual in the $M$-norm. The flexibility to select $M$ allows for a direct translation between GMRES with variable preconditioning and weighted/minimal residual solvers in alternative geometries.

Crucially:

• $M$-GMRES is mathematically equivalent to GMRES applied to a split-preconditioned system $(H_L^{-1}AH_R^{-1})$, where $M = H_L H_L^*$.
• This equivalence provides a direct pathway to interpret advanced preconditioning—left, right, or split—as instantiations of the flexible, weighted minimal residual framework.
• The freedom to select $M$ (or equivalently, $H$) enables the realization of prescribed convergence behavior in preconditioned (including dynamically preconditioned) GMRES.

This theoretical versatility is foundational for splitting preconditioners, block preconditioning in multiphysics applications, and for constructing non-Euclidean inner products suited to particular operator structures (2506.17193).

4. Algorithmic Mechanisms and Mathematical Formulations

Flexible GMRES variants share a backbone architecture defined by four critical elements:

a) Residual Norm Decomposition

Let $g_i = \sqrt{\|r_{i-1}\|^2 - \|r_i\|^2}$ denote the norm reduction at step $i$. For any desired curve $\{g_i\}$, there exists a (nested, Euclidean) orthonormal basis $W$ such that:

$b = \sum_{i=1}^m g_i w_i.$

A transformation $T$ exists so that in the basis $\widetilde{W} = WT$, the coefficients match a target $\tilde{g}$, and $M$ can be constructed as $M^{-1} = (WT)(WT)^*$ (2506.17193).

b) Flexible Preconditioning

At iteration $j$, the solver may select a preconditioner $M_j$, possibly varying, nonlinear, or inexact. The computed update direction is $u_j = M_j^{-1} v_j$. FGMRES generates an approximation by minimizing the residual in the subspace spanned by $\{u_0, ..., u_j\}$.

c) Simultaneous Convergence Realization

Given two residual decrease vectors $g$ (for GMRES) and $\tilde{g}$ (for $M$-GMRES), there exists an upper-triangular $T$ (with prescribed singular values) satisfying $g = T \tilde{g}$ and:

$b = W T \tilde{g}, \qquad M^{-1} = (W T)(W T)^*.$

d) Preconditioner Placement and Curves

For a nonsingular preconditioner $H$, right and left preconditioned GMRES systems $(AH, b)$ and $(HA, Hb)$ can be constructed to achieve any pair of convergence curves, with arbitrary spectra for preconditioned matrices.

A summary of these relationships is:

Variant	Residual Norm	Key Matrix Formulation
GMRES	$\|r_k\|_2$	$A, b$
Weighted GMRES	$\|r_k\|_M$	$M = (WT)(WT)^*$
Preconditioned GMRES	$\|r_k\|_2$ on $(AH, b)$	$H$ chosen to set desired curve

5. Practical Implications and Applications

Understanding this deep flexibility has direct impact on algorithm design and solver deployment:

• For practitioners, the convergence of GMRES is not strictly determined by eigenvalues or spectrum; operator geometry and preconditioning strategy (including their dynamic or nonlinear features) fundamentally influence the rate and pattern of convergence.
• The choice of inner product (designed via weighting, or dynamically through preconditioners) can be used to "steer" the convergence curve: algorithm designers can effectively prescribe rates of decrease or target robustness against stagnation.
• A choice between left, right, and split preconditioning is not innocuous: experiments can and do yield vastly different convergence curves, even holding the spectrum or core operator fixed.
• The theoretical machinery opens clear routes to developing more advanced flexible GMRES variants, including those for block systems, indefinite formulations, or under stochastic/randomized preconditioning.

6. Theoretical and Methodological Significance

The extension of GMRES to allow for any prescribed nonincreasing residual curve—via flexible and weighted frameworks—substantially broadens the theoretical foundation for iterative Krylov methods. It highlights the role of the basis construction, transformation matrices, and induced inner products in modulating convergence. The explicit formula for matching convergence curves via the singular values of $T$ and the eigenvalues of $M$ provides a detailed algebraic toolkit for both analysis and practical construction of adaptive and robust iterative solvers.

In summary, Flexible GMRES is not merely a practical enhancement of GMRES, but a mathematical framework in which convergence is fundamentally decoupled from spectrum and instead made fully prescriptive via dynamic selection of preconditioners and inner products. Theoretical results on the generality of convergence curves (2506.17193) and concrete mechanisms for imposing them highlight the adaptability and design latitude of this class of solvers, with broad implications for real-world linear algebra, PDEs, and scientific computing.