Spectral Operator Learning

• Spectral operator learning is a paradigm that represents infinite-dimensional operators using spectral properties like eigenvalues and eigenfunctions.
• It utilizes SVD and Hankel matrix factorizations to extract latent structures, enabling efficient modeling in dynamical systems and inverse problems.
• The framework integrates non-convex loss minimization and convex relaxations, underpinning modern neural and physics-informed architectures for accurate operator approximation.

Spectral operator learning refers to a family of techniques for learning, approximating, or representing operators (often nonparametric, infinite-dimensional, or latent structure) in terms of their spectral properties—such as eigenvalues, eigenfunctions, factor decompositions, or their action in a suitable orthonormal basis. This paradigm drives powerful frameworks for learning in dynamical systems, structured prediction, inverse problems, PDE solvers, and neural operators, and forms the backbone of modern statistical and computational learning theory in operator-valued settings.

1. The Spectral Learning Framework for Operator Models

Spectral operator learning originated in the context of learning weighted automata, observable operator models, and latent variable models where an unknown operator governs functional relationships or sequence predictions. The classical method leverages the Hankel matrix $H_f$, whose entries are $H_f(u,v) = f(uv)$ for string function $f$ and sequences (prefixes $u$, suffixes $v$). If $f$ admits a finite-dimensional linear representation (e.g., minimal weighted automaton), then $H_f$ is (block-)finite rank and $f$ admits a factorization: $$f(x_1 \ldots x_t) = \alpha_1^\top A_{x_1} \ldots A_{x_t} \alpha_\infty,$$ where $\alpha_1$ is an initial vector, each symbol $a$ is associated with an observable operator $A_a$, and $\alpha_\infty$ is the terminal weighting. The crux of spectral learning is to recover these operators from empirical Hankel sub-blocks via singular value decomposition (SVD) factorization and to reconstruct model parameters through sub-block relationships. This procedure admits linear sample and computational complexity and provides global optimization without local minima, in contrast to EM or variational inference approaches (Balle et al., 2012).

The approach generalizes to observable operator models (OOMs), transfer and Koopman operators in dynamical systems, spectral kernel learning, and neural operator architectures, exploiting either explicit or implicit eigen-decompositions, approximations in compact operator bases, or direct parameterization through orthogonal/spectral bases (Wu et al., 2016, Kostic et al., 2023, Froyland et al., 8 May 2025, Kiruluta et al., 27 Jul 2025).

2. Local Loss, Non-Convexity, and Convex Relaxation

A foundational insight is the realization that spectral learning can be phrased as local loss minimization: fitting operator action over a finite local basis $(U, V)$ given by a choice of coordinate subspaces (subsets of prefixes/suffixes), using sub-blocks of the Hankel matrix and their shifted versions for each operator symbol. This yields a non-convex optimization: $$L_n(X, B, \{B_a\}) = \|HX B - h_c\|^2 + \sum_a \|HX B_a - H_a X\|^2$$ subject to $X^\top X = I$. Here, $X$ projects the basis, $B$ and $\{B_a\}$ are auxiliary and observable operator matrices. Despite its non-convexity (from orthogonality and non-separable quadratic terms), under certain conditions (e.g., embedding dimension $n$ larger than the target rank) any optimizer reconstructs the target function exactly, and the SVD-based method yields an approximate optimizer (Balle et al., 2012).

A regularized convex relaxation is formulated by dropping non-convex variables (i.e., fixing $X=I$) and imposing a nuclear norm penalty: $$L_\tau(B_\sharp) = \|B_\sharp\|_* + \tau \|H B_\sharp - H_\sharp\|_F,$$ where $B_\sharp$ collects the operator blocks, $H_\sharp$ collects shifted Hankel blocks, and $\tau$ is a continuous regularization parameter. This convex surrogate enables smooth trade-off between accuracy and complexity, outperforming discrete-rank tuning in the original spectral method.

3. Operator Learning for Non-Stationary and Continuous Data

Spectral operator learning generalizes to dynamical systems, especially with non-stationary or nonequilibrium sampling (Wu et al., 2016, Kostic et al., 2023, Froyland et al., 8 May 2025):

• OOMs: Spectral learning extends to observable operator models, with cross-covariance matrices constructed from feature blocks of past/future observations; SVD yields observable operator recovery.
• Nonequilibrium Data: By enforcing equilibrium constraints (quadratic programming to solve for stationary initial weights) spectral learning can recover the equilibrium dynamics even in the absence of identically distributed trajectories.
• Continuous Data: Binless approaches avoid discretization by working directly with feature maps (e.g., Gaussian basis functions) to construct Hankel analogs, yielding consistent operator approximations in infinite-dimensional spaces.

For kernel regression and nonlocal operator learning, adaptive spectral Sobolev spaces formalize convergence rates and regularization according to spectral decay, with minimax rates governed by bias–variance trade-offs dependent on operator compactness and eigenvalue asymptotics (Jin et al., 2022, Zhang et al., 27 Feb 2025).

4. Spectral Operator Learning in Neural and Physics-Informed Architectures

Spectral approaches underpin several classes of neural operator architectures for PDE solution and surrogate modeling:

• Spectral Neural Operators: Fourier Neural Operators (FNOs), SPFNOs, and similar architectures perform convolution, multiplication, or modulation in a spectral basis (Fourier, Chebyshev, polynomial, etc.), enabling efficient representation, resolution-invariance, and (via specific bases) exact constraint satisfaction for boundary conditions (Liu et al., 2023, Harandi et al., 24 Oct 2024, Choi et al., 2023).
• Physics-Informed Spectral Learning: Frameworks such as SPiFOL impose physical constraints (e.g., Lippmann–Schwinger operator in Fourier space) as part of loss functions, leveraging the structure for resolution-independence, differentiability, elimination of automatic differentiation, and exact periodicity enforcement (Harandi et al., 24 Oct 2024).
• Unified Spectral-Physical Learning: Holistic Physics Mixer (HPM) and variants fuse fixed spectral representations (e.g., Laplace–Beltrami eigenfunctions) with pointwise calibration for mesh adaptivity and frequency preference, providing robust zero-shot generalization and accuracy superior to purely attention- or spectral-based methods (Yue et al., 15 Oct 2024).

Significant performance improvements are demonstrated for benchmark PDE problems, multiscale systems, and operator-based surrogate modeling, with explicit spectral regularization mitigating bias toward low frequencies and enhancing the learning of high-frequency or derivative-sensitive phenomena (Liu et al., 2022).

5. Spectral Operator Learning in Inverse and Structured Problems

Spectral decompositions form the core of modern approaches to ill-posed inverse problems, tomography, and generalized linear operator regression:

• Spectral Function Learning: Two-step algorithms perform Gram–Schmidt orthonormalization of input functions followed by principal component analysis (PCA) of the corresponding output, yielding the spectral decomposition of the unknown operator $F$ directly from data. This pipeline can be unrolled as a fixed "linear algebra network," akin to a deep encoder–decoder architecture, facilitating interpretability and data-driven regularization (Aspri et al., 20 Aug 2024).
• Koopman/Transfer Operators: Spectral learning is leveraged to approximate invariant subspaces and spectral decompositions for Koopman and Perron–Frobenius operators, critical for linearizing and extracting modal structure from nonlinear dynamics. Methods employ joint neural learning of invariant, locally supported bases and linear operator action, enforcing subspace invariance and enabling robust recovery of spectral properties in dynamical systems (Froyland et al., 8 May 2025, Kostic et al., 2023).
• Numerical Linear Algebra and Data-Driven SVD: Regularization and shrinkage in the spectral domain govern both stability and expressivity, formally connecting gradient descent behavior, activation function design, and principal component selection to the bandwidth and bias in learned operators (Lucey, 25 Apr 2025).

6. Open Problems, Theoretical Guarantees, and Implications

Recent theory establishes sharp minimax rates for operator learning (in mean squared error and operator norm) under various spectral decay regimes, explicitly characterizing the impact of the eigenvalue decay of normal operators and the smoothness of target kernels. Adaptive spectral Sobolev spaces provide a unifying language for classical interpolation, nonparametric regression, and kernel methods, discarding non-identifiable components and controlling estimation variance—a key bridge between inverse problems and statistical learning (Zhang et al., 27 Feb 2025, Jin et al., 2022).

Robustness, scalability, and interpretability arise from the explicit use of functional analytic tools; for instance, operator learning in Hilbert and Reproducing Kernel Hilbert Spaces (RKHS) enables transparent regularization, stability proofs, and the utilization of basis-adaptive (wavelet, Fourier, or data-driven) expansions (Kiruluta et al., 27 Jul 2025). Scalability is further enhanced by leveraging structured fast transforms and spectral parameterizations, and interpretability is a direct consequence of working in basis function space or through mode-wise modulation (soft-thresholding, spectral filtering) instead of parameterizing arbitrary neural network weights.

Spectral operator learning thus serves as a comprehensive paradigm at the intersection of theory and practice, spanning explicit and implicit operator representations, stochastic and deterministic learning, and integrating data-driven and physics-informed constraints. Its applications are extensive: latent variable models, nonlinear system identification, PDE surrogate modeling, inverse problems, optical computation, control and reinforcement learning, and interpretable feature extraction in high-dimensional domains. Continued developments are focusing on optimal regularization in high dimensions, adaptive and hierarchical basis design, faithful enforcement of physical laws via spectral constraints, and principled trade-offs between generalization and expressivity.