Harmonic Ritz Values in Krylov Methods

• Harmonic Ritz values are spectral approximations from Petrov–Galerkin projections in Krylov subspace methods, crucial for approximating interior or poorly separated eigenvalues.
• They provide insights into convergence, stagnation, and deflation in iterative solvers like GMRES by linking residual norms with eigenvalue patterns.
• Their computation involves solving small generalized eigenproblems and leveraging preconditioning strategies to tackle large-scale numerical linear algebra challenges.

A harmonic Ritz value is a spectral approximation arising from a Petrov–Galerkin projection, particularly central within Krylov subspace methods such as GMRES and harmonic Rayleigh–Ritz algorithms. Harmonic Ritz values are indispensable in the practical computation of interior or poorly separated eigenvalues and play a crucial role in understanding and diagnosing convergence, stagnation, and deflation strategies in iterative solvers. This article expounds on the formal definitions, algebraic characterizations, convergence theory, stagnation and admissibility phenomena, and practical implications for algorithms in numerical linear algebra.

1. Formal Definition and Algebraic Characterization

Let $A\in\mathbb{C}^{n\times n}$ be a general (possibly non-Hermitian) matrix. For a given $m$-dimensional Krylov subspace $K_m(A, b) = \mathrm{span}\{b,Ab,\ldots,A^{m-1}b\}$ and its orthonormal basis $V_m$, the $m$-step Arnoldi relation is

$A V_m = V_{m+1} \bar{H}_m,$

where $\bar{H}_m \in \mathbb{C}^{(m+1)\times m}$ is upper Hessenberg and $H_m$ denotes its leading $m \times m$ part.

The harmonic Ritz value $\lambda_H$ and vector $u\in\mathbb{C}^m$ at step $m$ satisfy: $$(A-\lambda_H I)V_m u \perp A K_m,\qquad \text{equivalently }(V_m^* A^*A V_m)\,u = \lambda_H (V_m^*A^*V_m)\,u.$$ In Arnoldi coordinates, using $A V_m = V_{m+1} \bar{H}_m$, this is

$\bar{H}_m^* \bar{H}_m u = \lambda_H H_m^* u,$

or (with $$\bar{H}_m = \begin{bmatrix} H_m \ h_{m+1,m} e_m^T \end{bmatrix}$$),

$$(H_m^* H_m + |h_{m+1,m}|^2 e_m e_m^T) u = \lambda_H H_m^* u.$$

The spectrum of this pencil yields the harmonic Ritz values $\{\lambda_{H,i}\}_{i=1}^m$.

For the harmonic Rayleigh–Ritz approach with shift $\tau\in\mathbb{C}$, the Petrov–Galerkin condition becomes: For $u\in K$,

$A u - \lambda u \perp (A-\tau I)K.$

Solving for $q\in\mathbb{C}^m$ with a basis $V_m$ for $K$ gives

$Cq = (\widetilde{\lambda} - \tau) Bq,$

where $B = V_m^*(A-\tau I)^*V_m$ and $C = V_m^*(A-\tau I)^*(A-\tau I)V_m$.

2. Stagnation, Admissibility, and Residual Correlation

A central theme in GMRES and related solvers is the relationship between harmonic Ritz values and residual norms. If GMRES does not stall, the harmonic Ritz spectrum at each iteration is unconstrained except by algebraic multiplicity; otherwise, specific structure emerges:

• If GMRES first stagnates at step $k$ (i.e., $\|r_k\| = \|r_{k-1}\|$), for all $\ell\geq k$, each $\Theta^{(\ell)}$ contains exactly $\ell-k$ infinite values and the $k$ finite values $\Theta^{(k)}$—i.e., the harmonic Ritz spectrum loses exactly one eigenvalue (typically $0$) and retains the remainder unchanged.
• Proposition 2.4 of (Du, 2016): At stagnation,

$$\theta_1^{(k+1)} = \theta_1^{(k)},\dots,\theta_k^{(k+1)} = \theta_k^{(k)},\quad \theta_{k+1}^{(k+1)} = \infty,$$

and this pattern extends for subsequent iterations.

• The harmonic residual $r_H=AV_m u-\lambda_H V_m u$ coincides with the GMRES residual $r_{GMRES}$ if and only if $e_m^T y = -e_m^T u$, provided there is no stagnation (Ravibabu, 2019).

The admissibility of a harmonic Ritz value sequence for a prescribed sequence of GMRES residual norms is governed solely by these stagnation-compatibility (infinity-insertion) conditions (Du, 2016).

3. Convergence Theory of Harmonic Ritz Values and Vectors

The convergence of harmonic Ritz values and vectors under the Rayleigh–Ritz or harmonic Rayleigh–Ritz paradigm is subtle, especially for non-Hermitian matrices or when the search subspace nears the target eigenvector:

• Let $(\lambda, x)$ be a simple eigenpair of $A$, $\tau\neq\lambda$ a shift, and $K$ a subspace. The harmonic Ritz pair $(\widetilde{\lambda},\widetilde{x})$ converges to $(\lambda, x)$ as $\angle(x,K)\rightarrow 0$, provided a "uniform separation" is maintained: $$|\lambda-\tau| \gg \sigma_{\min}(A-\tau I)\cdot \sin\angle(x,K)$$ (Wu, 2016).
• The convergence bounds avoid the prior requirement of uniformly nonsingular Rayleigh quotient matrices by recasting the problem as a Ritz approximation for $(A-\tau I)^{-1}$ on the image $(A-\tau I)K$.
• The error in the harmonic Ritz value is: $$|\widetilde{\lambda}-\lambda| \leq C \left(\kappa(A-\tau I)\sin\angle(x,K)\right)^{1/m},$$ with $C$ depending on subspace dimension, spectral gap, and singular values of $A-\tau I$.
• The associated harmonic Ritz vector error satisfies a Stewart-type bound, depending on $\kappa(A-\tau I)$, the harmonic Ritz separation in $(A-\tau I)^{-1}$, and $\sin\angle(x,K)$.

For Hermitian $A$, a generalized Saad-type bound for harmonic Ritz vectors formalizes the key dependence: $$\sin\angle(x,v)\;\le\;\kappa(S)\; \sqrt{\,1 \,+\,\frac{\gamma^2}{\delta^2}\sin^2\angle(x,K)},$$ where $S=A-\sigma I$, $\gamma=\|P_Q S^{-1}(I-P_Q)\|$, and $\delta$ is the eigenvalue separation in the harmonic Ritz spectrum (Vecharynski, 2015).

4. Computability, Preconditioning, and Algorithmic Implications

Harmonic Ritz values are computed by solving a small ($m\times m$) generalized or regular eigenproblem at each iteration or restart, leveraging dense eigensolvers in practical implementations. The structure and computability are as follows:

• The generalized eigenproblem $$(H_m^* H_m + |h_{m+1,m}|^2 e_m e_m^T)u = \lambda_H H_m^* u$$ can be reliably solved at moderate subspace dimensions.
• In the preconditioned harmonic Rayleigh–Ritz approach, an HPD preconditioner $T\approx(A-\sigma I)^{-1}$ is incorporated, effecting the Petrov–Galerkin condition $A v-\theta v \perp_T (A-\sigma I)K$, with improved convergence if $\kappa(T^{1/2}(A-\sigma I))$ is reduced; $T\approx |A-\sigma I|^{-1}$ or $T\approx (A-\sigma I)^{-2}$ are typical strategies (Vecharynski, 2015).
• For large-scale problems, exact preconditioners are replaced by multigrid, incomplete factorization, or polynomial filtering strategies.

Practical guidelines for robust use include:

• Avoiding shifts $\tau$ too close to target eigenvalues to control $\kappa(A-\tau I)$.
• Ensuring the search subspace $K$ is enriched enough that $\angle(x,K)$ is small and the separation of harmonic Ritz values in $(A-\tau I)^{-1}$ is adequate.
• Monitoring $e_m^T y$ and the smallest singular value of $H_m$ to detect stagnation or near-stagnation and adapting the algorithm (e.g., deflation, block methods).

5. Implications for Residual Norms, Spectrum, and Communications with GMRES

A decisive result in (Du, 2016) establishes that for any nonincreasing, positive sequence of GMRES residual norms $\{\|r_k\|\}$, and any admissible sequence of harmonic Ritz spectra in the stagnation-compatible sense, there exists a matrix $A$ and initial vector $b$ such that the GMRES procedure realizes precisely this convergence and spectral history. The upshot is:

• The behavior of harmonic Ritz values conveys no additional predictive power regarding convergence beyond the information present in the GMRES residual norm sequence.
• Any spectral pattern consistent with stagnation rules can be engineered for a given convergence trajectory.
• Consequently, using harmonic Ritz values to forecast convergence, select restart points, or guide deflation must account for their nonuniqueness: they are one of many possible spectral surrogates consistent with residuals, not a canonical "approximate spectrum" of $A$.
• This demonstrates the flexibility—but also the limitations—of using harmonic Ritz values in Krylov space methods.

6. Stagnation, Deflation, and Recovery

The relationship between harmonic Ritz spectra and GMRES stagnation underpins both diagnosis and remedial strategies:

• At the onset of stagnation, the leading $m\times m$ Arnoldi matrix $H_m$ becomes singular, with zero eigenvalue entering the spectrum.
• Nonzero harmonic Ritz values persist from step $m-1$ to $m$; only the new zero eigenvalue is introduced (Ravibabu, 2019).
• This informs practical strategies:
  • Deflation of converged or nearly-converged harmonic Ritz pairs (by augmenting the subspace with corresponding harmonic Ritz vectors).
  • Employing look-ahead or block (augmented) Krylov methods to maintain progress when stagnation is detected.
  • Subspace recycling and targeted re-expansion when near-stagnation (small-but-nonzero $e_m^T y$) impedes convergence.

7. Summary Table: Algebraic Characterizations

Setting	Harmonic Ritz Value Definition	Key Eigenproblem
GMRES, traditional basis	$(A-\theta I)v \perp A K_k$	$H_k^*H_k z = \theta H_k^* z$
Arnoldi, Rayleigh–Ritz	$A v - \theta v \perp K$	$V^* A V y = \theta y$
Harmonic Rayleigh–Ritz, shift $\tau$	$A u - \lambda u \perp (A-\tau I)K$	$Cq = (\widetilde{\lambda}-\tau) B q$
Stagnation ($H_m$ singular)	$\Theta^{(m+1)} = \Theta^{(m)} \cup \{\infty\}$	$H_m$ adds $0$ eigenvalue; rest unchanged

This tabulation highlights the structural distinctions and unifying features across Krylov subspace projections.

---

The harmonic Ritz value framework provides a unifying, constructive, and diagnostic tool for analyzing and implementing Krylov subspace methods, especially in contexts where interior eigenvalue information, robust convergence, or stagnation recovery is critical. The theoretical results establish precise algebraic constraints, convergence rates, and algorithmic freedom, dictating both the power and the limitations of harmonic Ritz analysis in practical iterative solvers.