---
title: Spectral Properties & NTK Analysis
url: https://www.emergentmind.com/topics/spectral-properties-and-neural-tangent-kernel-analysis
type: topic
---

# Spectral Properties & NTK Analysis

Spectral properties and Neural Tangent Kernel (NTK) analysis provide a rigorous mathematical framework for understanding learning dynamics, generalization, and inductive biases in wide neural networks. This perspective revolves around the eigendecomposition of kernels—especially the NTK—associated with neural architectures, which governs how different components of target functions are learned under gradient-based optimization. Key phenomena such as spectral bias, learning stages, and generalization bounds are direct consequences of the spectral properties of the underlying kernels. Below, the main principles, methods, results, and their implications are synthesized from contemporary research.

## 1. Kernel Spectral Decomposition and Eigenstructure

The NTK, as well as other associated kernels (e.g., Conjugate Kernel, Mercer kernels), admit spectral decompositions governed by Mercer's theorem when defined on compact domains:

\[
K(x, x') = \sum_{k=0}^\infty \lambda_k \sum_{m=1}^{N(d, k)} Y_{k m}(x) Y_{k m}(x')
\]

where $Y_{k m}$ are degree-$k$ spherical harmonics on $S^{d-1}$, $N(d, k)$ their multiplicity (degeneracy), and $\lambda_k$ the eigenvalue for the $k$th frequency [2002.02561][1912.01198]. For dot-product and neural tangent kernels, the spectral decomposition aligns with classical harmonics (e.g., Hermite polynomials for Gaussian input, Gegenbauer polynomials for the sphere, Boolean harmonics for the hypercube) [1907.10599][2111.03972]. The eigenspectrum quantitatively encodes which function spaces are "preferred" (fit more rapidly) by a given kernel.

## 2. Spectral Bias and Learning Dynamics

Wide neural networks—and their NTK-driven linearized training—exhibit a characteristic spectral bias: eigenmodes (features) corresponding to larger kernel eigenvalues are fit more rapidly under gradient descent. Concretely, for kernel regression or NTK-gradient flow, the coefficient $c_j$ on the $j$th eigenfunction evolves as:

\[
c_j(t) = c_j^* + (c_j(0) - c_j^*) \exp(-\lambda_j t)
\]

where learning speed in each direction is dictated by $\lambda_j$ [1912.01198][2002.02561]. Empirically and theoretically, low-frequency components (larger $\lambda_k$) are always fit and generalized earlier, whereas high-frequency components (small $\lambda_k$) decay slowly, leading to distinct learning stages or phase transitions as sample size or training time grows [2002.02561][2005.11879].

## 3. Influence of Network Architecture on Spectral Properties

The NTK spectrum is controlled both by depth, architecture, and activation function:

- **Fully Connected ReLU Networks**: The NTK is a rotationally invariant kernel whose spectrum decays polynomially in frequency. For the $k$th spherical harmonic mode: $\lambda_k \sim d^{-k}/k!$ in the high-dimensional regime [2002.02561][1907.10599]. Increasing network depth "whitens" the spectrum, slowing the eigenvalue decay and enabling faster learning of high-frequency modes.
- **Residual Networks**: The ResNet-NTK possesses the same eigenfunctions (spherical harmonics) and retains polynomial decay $\lambda_k \sim k^{-d}$. The spectrum's spikiness (localization near the diagonal and underrepresentation of mid-frequencies) can be controlled by adjusting the skip-connection hyperparameter [2104.03093].
- **Polynomial Nets (PNNs)**: PNNs equipped with Hadamard products display a much slower eigenvalue decay ($\lambda_k = \Omega(k^{-d/2})$ for degree $N$) than standard fully connected networks ($k^{-d}$), leading to more efficient learning of high-frequency modes and enhanced extrapolation capabilities beyond the support of training data [2209.07736].
- **Convolutional Architectures**: For neural tangent kernels derived from CNNs, the eigenfunctions are products of spherical harmonics over the channel and spatial dimensions. Eigenvalue decay can be quantified and is slower for spatially localized (few-pixel) patterns, giving CNNs a localized, hierarchical spectral bias and superior sample efficiency for local dependencies [2203.09255].

## 4. Spectral Decomposition and Generalization Error

The generalization error (for kernel regression or NTK-based training) admits an explicit mode-wise decomposition via the kernel eigenspectrum [2002.02561][2512.08264]:

\[
E_{\mathrm{gen}}(p) = \sum_\rho E_\rho(p)
\]
\[
E_\rho(p) = \frac{\langle \overline{w}_\rho^2 \rangle}{\lambda_\rho} \left[ \frac{1}{\lambda_\rho} + \frac{p}{\lambda + t(p)} \right]^{-2} \left[1 - \frac{p\,\gamma(p)}{(\lambda + t(p))^2} \right]^{-1}
\]

where each mode's error decays with sample size $p$ governed by its eigenvalue, and $t(p)$, $\gamma(p)$ are self-consistently determined by the spectrum. In the ridgeless limit ($\lambda \rightarrow 0$), mode-wise error simplifies:

\[
E_\rho(p) = \frac{\langle \overline{w}_\rho^2 \rangle}{\lambda_\rho} (1 + p \lambda_\rho)^{-2}
\]

yielding a $p^{-2}$ decay per mode and precise learning curves tied to the kernel and target function power spectra. These results, corroborated by empirical studies on synthetic data and real datasets such as MNIST, show that the NTK spectrum entirely orders generalization dynamics: modes with larger $\lambda_\ell$ are perfectly learned first as training/sample size increases [2002.02561][2507.18555][2512.08264].

## 5. Learning Stages, Architectural Effects, and Phase Transitions

Spectral analysis identifies learning stages corresponding to transitions where additional frequency bands are learned as the training set size or network capacity crosses certain thresholds. In high-dimensional spaces for dot product kernels (including NTK), $N(d, k) \sim d^k / k!$ and $\lambda_k \sim N(d, k)^{-1}$, so as $p \sim \alpha d^\ell$:

- Modes $k < \ell$: "perfectly learned" (modewise error approaches zero)
- Modes $k = \ell$: learning in transition
- Modes $k > \ell$: not yet fit, error remains near initial value

Consequently, learning proceeds as a series of spectral transitions, successively fitting higher-frequency modes. This phenomenon is robust to moderate deviations in data distribution and persists across architectures with the same underlying symmetry [2002.02561][1912.01198][2211.06506].

In linear-width regimes (where network width scales with dataset size), the empirical spectral distribution (ESD) of kernel matrices obeys deterministic limiting laws (e.g., Marčenko–Pastur), and bulk invariance is maintained under small learning rates. Large step sizes or adaptive optimization induce phase transitions: isolated "spike" eigenvalues emerge, corresponding to feature learning and alignment with target or spurious directions [2211.06506][2005.11879].

## 6. Spectral Generalization Bounds, Random Features, and Operator-Valued Extensions

Spectral properties directly control generalization bounds. For finite samples and regularization, the expected generalization error admits a spectral expansion:

\[
E_{\rm gen} \leq \sum_{i=1}^n \frac{(f_i - y_i)^2}{\lambda_i} + \epsilon
\]

with $\epsilon$ accounting for finite-width or stochastic effects [2512.08264][2002.02561]. Enriching the spectrum (e.g., through Fourier features or residual scaling) increases the smallest eigenvalues and sharpens generalization.

Random feature methods can be analyzed through the same spectral lens. Given appropriate spectral filters, minimax optimal rates for regression in the RKHS determined by the kernel hold provided the number of features $M$ matches the effective dimension, scaling as $M = O(\sqrt{n})$ for standard Tikhonov or as $M \propto n^{1/(2r + b)}$ for source regularity $r$ and eigenvalue decay rate $b$ [2506.16283][2603.00971][2104.01351].

These results extend to operator-valued kernels and neural operators, with Mercer-type expansions and tight control of the minimax learning rate, even in misspecified regimes [2603.00971].

## 7. Architectural and Optimization Mechanisms for Spectral Control

Recent work has established mechanisms to control and exploit spectral properties through informed architectural or algorithmic modifications:

- **Fourier feature embeddings** increase spectral support in high-frequency modes, reducing spectral bias and improving convergence on rapidly varying targets [2512.08264].
- **Residual scaling and stochastic depth** can be tuned to control the growth of maximal kernel eigenvalues (stability) and prevent departures from the linearized NTK regime [2512.08264][2104.03093].
- **Adaptive optimizers** (e.g., Adam) generate heavy-tailed spectra, learning multiple target directions and often correlating with improved test accuracy [2211.06506].

For physics-informed neural networks (PINNs), it is established that the introduction of a differential operator in the loss does not in general accelerate tail decay of the NTK spectrum or enhance learning of high-frequency modes. Advanced activation functions (e.g., periodic, SIREN-type) and loss-balancing heuristics can partially alleviate spectral bias in these setups [2211.11716][2503.11029][2506.07958].

---

## References Table

| Topic                             | Key Results and Methods                                      | arXiv ID      |
|------------------------------------|-------------------------------------------------------------|---------------|
| Mercer decomposition, spectral decay, learning stages | Analytical formulae for mode-wise error, "successive mode fit", $\lambda_k \sim d^{-k}/k!$, distinct learning stages | [2002.02561]  |
| NTK spectral bias theory           | Decomposition of training process along kernel eigenfunctions; low-frequency fit first | [1912.01198]  |
| Architectural effects: Residuals, PNNs, convolution | ResNTK and FC-NTK share spectrum; PNNs have heavier spectral tails, CNNs exhibit spatially localized spectra | [2104.03093], [2209.07736], [2203.09255] |
| Spectrum and generalization bounds | Explicit spectral error bounds; enrichment increases min-eigenvalue, improves error | [2512.08264]  |
| Linear-width & high-dim spectral analysis | Phase transition, bulk invariance, Marčenko–Pastur, emergence of spikes and heavy tails | [2005.11879], [2211.06506] |
| Random features and spectral analysis | Finite-sample, RF, and operator-valued kernel generalization rates, spectral approximation | [2506.16283], [2603.00971], [2104.01351] |
| Physics-informed networks & differential operators | Differential operators do not induce faster decay; spectral bias is robust; periodic activations flatten spectrum | [2211.11716], [2503.11029], [2506.07958] |

---
Spectral properties and NTK analysis provide a unified, quantitative, and predictive toolkit for understanding the differential learnability of features in wide neural networks. The eigenspectrum fully determines the learning curve, orders convergence by “simplicity,” and dictates both algorithmic limitations and pathways for architectural improvement. The core principle is that kernel spectra act as an "inductive filter," enforcing a bias toward low-complexity solutions, structuring generalization, and, through explicit manipulation, enabling informed control of network learning dynamics.

Source: https://www.emergentmind.com/topics/spectral-properties-and-neural-tangent-kernel-analysis