---
title: Arnoldi-Based Minimisation Method
url: https://www.emergentmind.com/topics/arnoldi-based-minimisation-method
type: topic
---

# Arnoldi-Based Minimisation Method

The Arnoldi-based minimisation method encompasses a spectrum of techniques exploiting Arnoldi's process within Krylov subspace frameworks for solving minimisation problems associated with linear systems, rational matrix functions, ill-posed inverse problems, high-dimensional optimisation, least-squares fitting, and model reduction. Central to all such methods is the use of the Arnoldi iteration to construct an orthonormal basis for Krylov subspaces, providing a well-conditioned coordinate system for efficient, reliable subspace-projected minimisation. These techniques have demonstrated significant efficacy for large-scale, high-dimensional, and/or ill-conditioned problems, especially where explicit operators (e.g., Hessians) are unavailable, or in the presence of noise and numerical instability.

## 1. The Arnoldi Process: Generation of Krylov Subspaces

The Arnoldi process generates an orthonormal basis $\{v_1,\ldots,v_m\}$ for the Krylov subspace $\mathcal{K}_m(A,b) = \mathrm{span}\{b, Ab, \ldots, A^{m-1}b\}$ associated with a matrix $A$ and a starting vector $b$ [1806.06599]. At each iteration $j$,
\[
w = A v_j,\quad h_{i,j} = v_i^*w,\quad w \leftarrow w - \sum_{i=1}^j v_i h_{i,j},\quad h_{j+1,j} = \|w\|,\quad v_{j+1} = w/h_{j+1,j}.
\]
The process yields an orthonormal basis $V_{m+1} = [v_1, \ldots, v_{m+1}]$ and an upper Hessenberg matrix $H_{m+1,m}$ satisfying $A V_m = V_{m+1} H_{m+1,m}$.

Extensions exist for functionals of matrices (e.g., rational functions, shifted-inverse forms), for approximating matrix exponentials, and for constructing rational Krylov spaces when denominator structures or regularization are imposed [2306.17308, 1009.5516].

## 2. Arnoldi-Based Minimisation Frameworks

### 2.1 Linear and Regularised Least-Squares

For linear systems $Ax=b$, the GMRES algorithm seeks $x_m\in\mathcal{K}_m(A,b)$ minimising $\|A x - b\|_2$ [1806.06599], which reduces to a small least-squares problem over $y\in\mathbb{R}^m$ using the Arnoldi basis:
\[
y_m = \arg\min_{y}\|H_{m+1,m} y - \beta e_1\|_2,\quad x_m = V_m y_m.
\]
Arnoldi-based variants regularise the subspace problem, for example:
- **Arnoldi–Tikhonov:** $\arg\min_y \|H_{m+1,m} y - \beta e_1\|_2^2 + \lambda\|y\|_2^2$.
- **Arnoldi–TSVD:** Truncates small singular values of $H_{m+1,m}$ to stabilise solutions to ill-posed problems.

### 2.2 Rational Function Approximations and Matrix Functions

Given a rational function $R(z) = D(z)^{-1} N(z)$ and matrix $A$, one seeks $x_k\in\mathcal{K}_k(A,b)$ minimising $\|R(A)b-x_k\|_{D(A)} = \|N(A)b-D(A)x_k\|_2$ in the $D(A)^*D(A)$-norm [2306.17308]. The Arnoldi-OR method extends Arnoldi for this setting, requiring extra $\nu = \max\{\deg D, \deg N\}$ steps and solving a $(k+\nu)\times k$ least-squares problem. The process uses only small blocks of Hessenberg matrices, avoiding explicit evaluation of high-degree matrix polynomials. For $R(z) = 1/z$, Arnoldi-OR reduces to GMRES.

Rational Arnoldi methods (RA) recast solutions of regularised systems or matrix equations as $f(Z)b$, with $Z$ a shifted-inverse or regularised operator, and $f$ a rational or matrix function; $x_m = \|b\| V_m f(H_m) e_1$ [1009.5516].

### 2.3 High-Dimensional Optimisation Under Noisy Data

The Stochastic Arnoldi Method (SAM) addresses optimisation of $f(x)$ in high-dimensional, noisy settings [1501.03735]. Arnoldi sampling, via finite-difference Hessian-vector approximations of noisy gradients, constructs a low-rank approximation of the Hessian in the subspace spanned by dominant Ritz vectors. A quadratic trust-region model is built in this subspace, using either step-averaged or directional-derivative-based linear parts, and trust-region subspace methods (e.g., Moré–Sorensen) are employed to update iterates.

### 2.4 Polynomial and Rational Least-Squares, Structured Fitting

In polynomial, rational, and Sobolev least-squares problems, Arnoldi orthogonalization renders the ill-conditioned Vandermonde (or Cauchy) coefficient matrix as a Krylov subspace basis, reducing the overdetermined system
\[
B_{n+1} c \approx W f
\]
to a small, well-conditioned upper-triangular (or block pentadiagonal) system [2407.05945]. For multivariate polynomial and Hermite least-squares, the G-Arnoldi method constructs a $G$-orthonormal basis appropriate for weighted and application-driven inner products [2404.09266].

### 2.5 Aggregation and Model Reduction for Markov Chains

Arnoldi aggregation projects the evolution of a Markov chain, defined by $p(t)^\top = p(0)^\top \exp(Q t)$, onto the Krylov span generated by $p(0)^\top$ and $Q$, providing a low-dimensional system $H_j$ governing the projected dynamics. The approximation $p_\mathrm{approx}(t)^\top = \beta e_1^\top \exp(t H_j) V_j$ is minimal, provably exact when an invariant subspace is reached, and allows for sharp, residual-based error estimation [2508.02078].

## 3. Key Algorithmic and Structural Elements

Arnoldi-based minimisation methods share several algorithmic features, often tailored for specific problem classes:

- **Orthonormal Basis Construction:** Modified Gram–Schmidt, with application-aware inner products (e.g., $G$-inner products for PDEs or data weighting).
- **Small Projected Systems:** The main minimisation problem in the full space is projected to a low-dimensional system (Hessenberg, block, or banded), exploiting the structure of $V_m$ and $H_m$ (or $(H,K)$ pencils in rational or confluent cases).
- **Implicit Function Action:** Matrix functions $f(A)$, rational functions, or exponentials are applied in the reduced subspace via $f(H_m)$, circumventing the need for full-matrix computations.
- **Finite-Difference/Subsampled Approximations:** For non-explicit operators (e.g., Hessian in optimisation), sampled Hessian-vector products or directional derivatives are used.
- **Short Recurrences:** For least-squares and fitting, Arnoldi-generated discrete orthogonal basis functions satisfy explicit recurrences, accelerating evaluation and stability [2404.09266].
- **Residual and Stopping Criteria:** Residuals in the last Arnoldi direction or reductions in model error (e.g., trust-region model vs. actual function decrement) provide natural stopping rules and adaptivity [2508.02078, 1501.03735].

## 4. Theoretical Properties and Numerical Behavior

Table: Principal Theoretical Features and Classes

| Class                       | Optimality/Exactness           | Conditioning                               | Error Analysis                |
|-----------------------------|--------------------------------|--------------------------------------------|-------------------------------|
| GMRES / Linear System [1806.06599]       | $\ell_2$-minimal residual in subspace    | Improved by preconditioning                | Spectrum, normality, and projections  |
| Arnoldi-OR / Rational Function [2306.17308] | $D(A)^*D(A)$-minimal residual             | Avoids explicit $R(A)$, uses residual norm | Crouzeix–Palencia, Ritz values        |
| SAM / Stochastic Optimisation [1501.03735] | Trust-region quadratic in dominant subspace | Low-rank, robust to noise                  | Bounded/truncated noise, empirical     |
| Polynomial/Rational LS [2407.05945]     | LS-optimal in polynomial/rational Krylov | Dramatically stabilised vs. Vandermonde    | Orthogonal polynomials/rationals       |
| Arnoldi Aggregation / Markov [2508.02078]    | Provably exact on invariant subspace      | Minimal dimension for accurate expansion   | Residual bounds, superlinear decay     |

For linear problems, minimisation in the Arnoldi Krylov subspace is classically optimal with respect to the chosen norm, with small projected systems advantageous for both accuracy and efficiency. Preconditioning and regularisation within the Arnoldi framework have proven effective at mitigating ill-posedness and semi-convergence [1806.06599, 1009.5516]. For rational approximations and matrix functions, sharp spectral-set bounds and the limitations of eigenvalue-based heuristics are established [2306.17308].

In stochastic or high-dimensional optimisation, the robustness of the model to noise is contingent on spectral decay of the Hessian and subspace parameter choices [1501.03735]. Empirically, Arnoldi-based minimisation demonstrates accelerated reduction of error and residual norm relative to naive approaches, particularly in state-space reduction and data-fitting settings [2508.02078, 2407.05945, 2404.09266].

## 5. Parameter Selection, Limitations, and Practical Guidelines

Parameter choices are application-dependent but the following guidelines emerge across classes:

- **Arnoldi subspace dimension ($m$, $k$):** 4–10 (optimisation), $2$–$3$ times larger for stable eigen-estimates (spectrum estimation for Hessian), or until diminishing returns in model error/residual.
- **Sampling radius in optimisation ($\alpha$):** Chosen to match expected trust-region scale or expected noise level [1501.03735].
- **Preconditioning:** Application-specific; circulant or low-rank operators can be derived from initial Arnoldi samples [1806.06599].
- **Orthogonality and Stability:** Re-orthogonalisation (“twice is enough”) can be employed in polynomial bases at the cost of increased flops [2407.05945].
- **Choice of weights and poles (rational/polynomial LS):** In weighted least-squares, quadrature-based weights enhance accuracy; adaptive pole selection for rational Arnoldi remains an open research direction [2407.05945].
- **Computational costs:** Each Arnoldi step involves a matrix-vector product and orthogonalisation (dominating cost for large systems).

Limitations include finite-precision breakdown for very large degrees or insufficiently decaying spectra, potential loss of orthogonality, and (in rational settings) sensitivity to pole placement.

## 6. Impact and Applications Across Disciplines

Arnoldi-based minimisation methods are widely adopted in scientific computing, optimisation, statistical inference, inverse problems, signal processing, uncertainty quantification, and model reduction. Applications include:
- Efficient iterative solvers for very large, sparse, or ill-posed linear and regularised systems [1806.06599, 1009.5516].
- Acceleration of objective minimisation in large-scale design under noise and uncertainty [1501.03735].
- Rational or polynomial data fitting and least-squares for high-accuracy approximation in numerical analysis and computational physics [2407.05945, 2404.09266].
- Reduced-order modelling for time-dependent Markov processes and chemical reaction networks [2508.02078].
- Regularised solution of PDEs via basis adaptation to non-standard inner products and functional constraints [2404.09266].

The unifying feature of Arnoldi-based minimisation is the reformation of diverse but challenging minimisation tasks into well-conditioned, low-dimensional subspace problems, enabled by the algebraic and numerical properties of Krylov sequences and Arnoldi-type recurrences.

Source: https://www.emergentmind.com/topics/arnoldi-based-minimisation-method