---
title: Eigenfunctions of the Neural Tangent Kernel
url: https://www.emergentmind.com/topics/eigenfunction-of-ntk
type: topic
---

# Eigenfunctions of the Neural Tangent Kernel

The eigenfunctions of the Neural Tangent Kernel (NTK) play a central role in characterizing the learning dynamics, inductive bias, and feature selection properties of (infinitely) wide neural networks in both theory and practice. For a given data distribution and neural network architecture, the NTK defines a positive semidefinite kernel whose spectrum—its eigenvalues and eigenfunctions—governs not only how target functions are represented, but also how quickly various modes of the target function are learned and retained during optimization.

## 1. Mathematical Formulation and Integral Operator Structure

The NTK for a neural network $f_\theta(x)$ with parameters $\theta$ is defined, at initialization, as the inner product of parameter-space gradients:
\[
K(x, x') = \nabla_\theta f_{\theta_0}(x) \cdot \nabla_\theta f_{\theta_0}(x').
\]
Under suitable regularity conditions, $K$ induces an integral operator $T_K$ on $L^2(\mu)$:
\[
(T_K \phi)(x) = \int K(x, x') \phi(x') \, d\mu(x'),
\]
where $\mu$ is the input distribution. The associated eigenvalue problem seeks $\lambda \geq 0$ and $\phi \in L^2(\mu)$ such that
\[
\int K(x, x') \phi(x') \, d\mu(x') = \lambda \phi(x),
\]
with normalization $\int \phi(x)^2 d\mu(x) = 1$. By Mercer's theorem, the spectrum $\{\lambda_i, \phi_i\}$ is discrete (for compact $\mathcal{X}$), and the eigenfunctions $\{\phi_i\}$ form an orthonormal basis for $L^2(\mu)$ [2007.01580][2201.04738]. On $N$ data points, the Gram matrix $G_{ij} = K(x_i, x_j)$ provides an empirical analogue, with eigenvectors $v_i$ approximating $\phi_i(x_k)$ on the dataset.

## 2. Spectral Decomposition and Empirical Computation

The NTK kernel admits the expansion
\[
K(x, x') = \sum_{i=1}^\infty \lambda_i\, \phi_i(x) \phi_i(x').
\]
In practice, one computes the leading $k$ eigenpairs of the empirical NTK Gram matrix by standard numerical linear algebra (e.g., Lanczos or randomized SVD algorithms). At new inputs $x$, the associated eigenfunction is obtained via the Nyström approximation:
\[
\phi_i(x) \approx \lambda_i^{-1} \sum_{j=1}^N K(x, x_j) v_i(j).
\]
Empirical evaluations show that in finite-width settings, the eigenfunctions and eigenvalues evolve throughout training, with the top eigenfunctions dynamically aligning to the target function and serving as a compact basis for the network output [1910.08720].

## 3. Analytical Characterizations in Key Data and Model Regimes

For fully connected networks and rotation-invariant distributions (e.g., uniform measure on $S^{d-1}$), the NTK is a zonal kernel: $K(x, x') = H(x \cdot x')$. The Mercer decomposition utilizes spherical harmonics $Y_{\ell, m}$:
\[
K(x, x') = \sum_{\ell=0}^{\infty} \sum_{m=1}^{N(d, \ell)} \lambda_\ell\, Y_{\ell, m}(x) Y_{\ell, m}(x'),
\]
with eigenvalues
\[
\lambda_\ell = \frac{a_\ell}{N(d, \ell)},
\]
where $a_\ell$ are Gegenbauer expansion coefficients of $H$ and $N(d, \ell)$ is the multiplicity. The eigenfunctions are the spherical harmonics, and asymptotically, the eigenvalues decay as $\Theta(\ell^{-d})$, directly matching the spectrum of the Laplace kernel [2007.01580]. Thus, the NTK and Laplace kernel yield RKHSs with the same Sobolev regularity on the sphere.

In the case of multilayer linear networks and Gaussian mixture data,
\[
K(x, x') \propto x^\top x',\quad \phi(x) = x^\top v,
\]
where $v$ solves $Mv = a v$, with $M$ as the second-moment matrix of the mixture, and the eigenvalue is $\lambda = \sum_k \pi_k \sigma_k^2 + a$ [2602.03066].

## 4. Dynamics of Learning and NTK Eigenfunctions

During gradient descent on mean squared error, each residual component along an NTK eigenfunction decays exponentially at a rate set by its eigenvalue:
\[
\langle r_t, \phi_i \rangle \approx e^{-\lambda_i t} \langle r_0, \phi_i \rangle,
\]
with corrections governed by the "damped deviations" framework in underparameterized regimes [2201.04738]. In overparameterized or NTK-linear regimes, the empirical spectrum aligns rapidly to its limiting form, and directions with large $\lambda_i$ are fitted significantly faster. For finite-width networks, NTK eigenfunctions are not perfectly static but instead rotate during training to align the top spectrum with the target function [1910.08720].

## 5. Spectral Bias, Shortcut Features, and Generalization

The NTK spectrum imposes a form of spectral bias: low-frequency (large-eigenvalue) modes are learned first, while high-frequency (small-eigenvalue) modes are fitted more slowly. In the presence of data with clustered structure or spurious correlation (e.g., shortcut features), the top NTK eigenfunctions can align with these dominant clusters or shortcuts. Their associated large eigenvalues ensure both rapid learning and persistent post-training influence, even after aggressive margin maximization [2602.03066]. This provides a principled operator-theoretic explanation for shortcut feature dominance and the slowness of learning rare or nuanced modes.

Empirical studies with two-layer ReLU nets and deep architectures (e.g., ResNet-18) reveal that practical NTK eigenmodes extracted from finite-width models localize on spurious or dominant cluster features, while lower-eigenvalue modes recover more generalizable structure [2602.03066].

## 6. Role of Depth, Width, and Training Procedures

Deeper architectures achieve higher spectral alignment (i.e., higher $E_t(y, k)$ for given $k$) with targets, improving convergence rates and generalization. Width, by contrast, exhibits diminishing returns for alignment in fixed data settings [1910.08720]. Further, practical training procedures such as learning rate decay induce monotonic jumps in eigenvalues and redistribute energy across the spectrum, with top eigenvalues growing more rapidly and the top eigenspaces remaining stable throughout learning. This preserves the basis functions that span the majority of the network output and ensures that the optimization proceeds efficiently along the dominant (high-$\lambda$) directions.

## 7. Interpretation and Broader Theoretical Consequences

The NTK eigenfunction perspective recasts neural network learning dynamics as a spectral filtering process in function space [2007.01580][2201.04738]. The empirical finding that top NTK eigenfunctions form a stable, low-dimensional subspace capturing nearly all label and output variance throughout training [1910.08720] clarifies why overparameterized networks generalize well on low-complexity functions, while also being susceptible to shortcut learning under distributional imbalance. The evolving NTK spectrum in finite-width networks can be interpreted as an "adaptive kernel method," in which the top spectrum is learned to match the structure of the target, providing both expressive power and implicit regularization.

## Summary Table: Key Facts on NTK Eigenfunctions

| Aspect                | Core Fact                                                                           | Reference        |
|-----------------------|-------------------------------------------------------------------------------------|------------------|
| Spectral basis        | Eigenfunctions $\{\phi_i\}$ form orthonormal basis of $L^2(\mu)$                    | [2007.01580]     |
| Data regime: sphere   | $\phi_i$ are spherical harmonics; $\lambda_\ell = \frac{a_\ell}{N(d, \ell)}$        | [2007.01580]     |
| Learning dynamics     | Residuals along $\phi_i$ decay as $e^{-\lambda_i t}$                                | [2201.04738]     |
| Spectral bias         | Large-$\lambda$ (smooth/low-frequency) modes learned faster                         | [2201.04738]     |
| Shortcut alignment    | Dominant clusters/spurious features align with top $\phi_i$ and large $\lambda_i$   | [2602.03066]     |
| NTK evolution         | Top eigenspace rotates to align with target during training                         | [1910.08720]     |
| Depth effect          | Greater depth enables better alignment and faster convergence                        | [1910.08720]     |

The NTK eigenfunction framework provides a precise spectral lens for interpreting training dynamics, feature selection, and inductive bias in both linear and deep nonlinear neural networks [1910.08720][2007.01580][2201.04738][2602.03066].

Source: https://www.emergentmind.com/topics/eigenfunction-of-ntk