Karhunen–Loève Expansion Overview

• Karhunen–Loève expansion is a spectral representation that decomposes second-order stochastic processes into an infinite series of orthogonal deterministic functions with uncorrelated random coefficients.
• It optimally diagonalizes the covariance operator to achieve mean-square minimizing low-dimensional approximations, with error control based on eigenvalue decay.
• Widely applied in uncertainty quantification, signal processing, and numerical simulations, the method provides both theoretical insights and practical computational advantages.

The Karhunen–Loève expansion (KL expansion, sometimes KLE) is a canonical spectral representation that expresses a second-order stochastic process or random field as an infinite series of orthogonal deterministic functions multiplied by uncorrelated random coefficients. For centered, mean-square-continuous, square-integrable processes with continuous covariance on a bounded domain, the KL expansion furnishes an orthogonal transformation that diagonalizes the covariance operator and provides optimal (mean-square-minimizing) low-dimensional approximations. The expansion is widely used in stochastic process analysis, uncertainty quantification, signal processing, machine learning, and numerical simulation of random fields.

1. Mathematical Formulation and Spectral Theory

Given a stochastic process $X: D \times \Omega \to \mathbb{R}$ on a bounded domain $D \subset \mathbb{R}$, assume:

• Centered: $\mathbb{E}[X(t)] = 0$ for all $t \in D$.
• Mean-square continuous: $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$.
• Square-integrable: $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$.

Let

$$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$

be the autocovariance function, which is continuous and symmetric positive semidefinite.

Define the covariance operator $K: L^2(D) \rightarrow L^2(D)$ as

$(Ku)(s) = \int_D R_X(s, t) u(t)\,dt.$

By standard results, $K$ is Hilbert–Schmidt, compact, self-adjoint, and positive. The spectral theorem ensures the existence of an orthonormal basis $D \subset \mathbb{R}$0 and nonnegative eigenvalues $D \subset \mathbb{R}$1 such that

$D \subset \mathbb{R}$2

Equivalently, the eigenproblem is

$D \subset \mathbb{R}$3

Mercer's theorem guarantees the absolute and uniform convergence

$D \subset \mathbb{R}$4

and orthonormal completeness of $D \subset \mathbb{R}$5 in $D \subset \mathbb{R}$6 (Alexanderian, 2015).

2. Construction of the KL Expansion

For each $D \subset \mathbb{R}$7, expand $D \subset \mathbb{R}$8 in $D \subset \mathbb{R}$9: $\mathbb{E}[X(t)] = 0$0 This expansion converges in mean-square (i.e., $\mathbb{E}[X(t)] = 0$1), and by Mercer's theorem uniformly in $\mathbb{E}[X(t)] = 0$2. The random coefficients $\mathbb{E}[X(t)] = 0$3 satisfy (Alexanderian, 2015):

• $\mathbb{E}[X(t)] = 0$4,
• $\mathbb{E}[X(t)] = 0$5,
• $\mathbb{E}[X(t)] = 0$6.

Thus, the normalized coefficients

$\mathbb{E}[X(t)] = 0$7

have zero mean and unit variance, and are uncorrelated: $\mathbb{E}[X(t)] = 0$8 The series can be written as

$\mathbb{E}[X(t)] = 0$9

Convergence in $t \in D$0 is established for each $t \in D$1, with uniform mean-square convergence in $t \in D$2, and almost-sure uniform convergence under further conditions (Alexanderian, 2015).

3. Truncation, Optimality, and Error Control

The KL expansion provides the best rank-$t \in D$3 approximation of $t \in D$4 in mean-square sense: $t \in D$5 By orthogonality and optimality, the mean-square truncation error is

$t \in D$6

This error decays rapidly if the eigenvalues $t \in D$7 decay fast, which is typical for smooth or highly correlated random fields (Alexanderian, 2015). The truncation should be guided by desired mean-square accuracy or by capturing a specified fraction of variance $t \in D$8 (Alexanderian, 2015).

4. Specialization to Gaussian and Structured Processes

For a Gaussian process, all linear combinations of $t \in D$9 are jointly Gaussian, so all KL coefficients $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$0 are Gaussian and, because they are uncorrelated, independent: $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$1 Almost sure uniform convergence of the KL series thus holds (Alexanderian, 2015).

The KL expansion can be applied to more structured random processes. For example, processes with exponential covariance on $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$2: $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$3 yield eigenvalues decaying as $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$4 with eigenfunctions of increasing oscillation (Alexanderian, 2015). For Gaussian or “g-detrended” Wiener processes (such as the Wiener bridge), the covariance structure and KL expansion may admit closed-form expressions for eigenvalues and eigenfunctions, as detailed for various choices of detrending functions in (Barczy et al., 2016).

5. Numerical Realization and Simulation

Practical computation necessitates discretization. The integral operator is approximated by a quadrature rule (e.g., Nyström method): select nodes $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$5 and weights $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$6, then form the $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$7 discrete covariance matrix: $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$8 Eigenpairs $\lim_{h \to 0} \mathbb{E}[(X(t + h) - X(t))^2] = 0$9 are computed numerically, and the first $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$0 modes are used for the truncated approximation: $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$1 For field simulation, $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$2 are generated as independent $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$3 and inserted into the expansion. For small $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$4, realizations appear overly smooth; higher $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$5 captures finer field variability (Alexanderian, 2015).

In the studied exponential-covariance example, plotting partial reconstructions of the kernel

$$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$6

demonstrates rapid uniform convergence, with $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$7 yielding pointwise error of order $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$8 (Alexanderian, 2015).

6. Practical Guidance and Applications

KL expansion is most effective when the spectrum $$\int_\Omega \int_D X(t,\omega)^2\,dt\,dP(\omega) < \infty$$9 decays rapidly, allowing low-rank approximations. Practical recommendations include:

• Inspect eigenvalue decay before truncation.
• Use truncation thresholds based on mean-square error or captured variance.
• For Gaussian fields, coefficient sampling requires only independent standard normal draws.
• KL expansion is foundational in dimensionality reduction, field simulation, uncertainty quantification, and as an optimal decorrelation transform in statistics and engineering (Alexanderian, 2015).

The methodology is directly applicable to problems in Bayesian inversion, PDE models with random coefficients, model and state reduction, and spatial statistics; the underlying principles remain unchanged across these diverse settings.

7. Summary Table: Key Elements of the Karhunen–Loève Expansion

Element	Description	Reference Equation / Property
Covariance operator	$$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$0	Operator eigenproblem
Eigenfunctions	$$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$1	Spectral decomposition
KL coefficients	$$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$2	Zero mean, variance $$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$3
KL expansion	$$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$4	Convergence in mean square
Truncation error	$$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$5	Optimal in mean-square sense
Simulation	Sample $$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$6 i.i.d. $$R_X(s, t) = \mathrm{Cov}[X(s), X(t)] = \mathbb{E}[X(s) X(t)]$$7, sum as above	Monte Carlo realization
Gaussian process	KL coefficients are independent normal random variables	Independence in Gaussian case

This conceptual and computational framework underpins rigorous analysis and simulation of random fields in theory and application (Alexanderian, 2015).