NTK Eigenfunctions in Deep Learning

• NTK eigenfunctions are the spectral modes that decompose neural network training and generalization, illuminating the basis of shortcut learning.
• They are defined as eigenfunctions of the NTK-induced integral operator, with derivations validated in both linear and nonlinear architectures.
• Imbalanced input clusters and nonzero variances amplify shortcut eigenfunctions, causing them to dominate training even under strong regularization.

Neural Tangent Kernel (NTK) eigenfunctions are the spectral modes of the integral operator induced by the NTK, providing a functional basis in which the inductive biases and learning dynamics of infinite-width neural networks can be analyzed. These eigenfunctions underpin the spectral decomposition of training dynamics and generalization, and play a central role in understanding phenomena such as shortcut learning in deep neural networks. Theoretical analysis, especially in the case where input distributions are clustered or imbalanced, reveals how shortcut features emerge as top NTK eigenfunctions, with large eigenvalues and disproportionate influence over function fitting—even after the application of regularization techniques aimed at margin control (Lim et al., 3 Feb 2026).

1. Formal Definition of NTK Eigenfunctions

Let $\rho$ be a probability measure on the input space $X$. In the infinite-width (lazy training) limit, a neural network’s NTK becomes a fixed, positive semi-definite kernel function $K : X \times X \to \mathbb{R}$. This kernel induces a compact Hilbert-Schmidt integral operator,

$T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$

An NTK eigenfunction (or feature) is defined as any function $\phi_i \in L^2(X, \rho)$ that satisfies

$$\int_X K(x, x') \,\phi_i(x')\,d\rho(x') = \lambda_i\, \phi_i(x),$$

where $\lambda_i \geq 0$ is the corresponding NTK eigenvalue and $\|\phi_i\|^2 = \int\phi_i^2\,d\rho = 1$. Under mean squared error (MSE) loss, the solution after infinite time admits a decomposition in the NTK eigenbasis:

$$f_\infty(x) = \sum_{i} \frac{1}{\lambda_i}\langle\phi_i, y\rangle\,\phi_i(x).$$

Each NTK feature $\phi_i$ acts as a coordinate axis in function space, weighted by both label signal and the inverse NTK eigenvalue (Lim et al., 3 Feb 2026).

2. Closed-Form Eigenfunctions for Linear Networks

For a linear network, the NTK simplifies to $X$0. Consider the input distribution $X$1, a Gaussian mixture with cluster weights $X$2. Seeking eigenfunctions linear in $X$3, i.e., $X$4 for unit vector $X$5, the action of the NTK operator is:

$X$6

By setting $X$7 and $X$8, one establishes that $X$9 must be an eigenvector of $K : X \times X \to \mathbb{R}$0 with eigenvalue $K : X \times X \to \mathbb{R}$1, and the associated NTK eigenvalue is $K : X \times X \to \mathbb{R}$2.

The normalized eigenfunctions are

$K : X \times X \to \mathbb{R}$3

If one mixture component dominates (large $K : X \times X \to \mathbb{R}$4), the corresponding $K : X \times X \to \mathbb{R}$5 has the largest $K : X \times X \to \mathbb{R}$6 and thus $K : X \times X \to \mathbb{R}$7, marking it as a “shortcut” direction (Lim et al., 3 Feb 2026).

3. Influence of Cluster Imbalance and Variance

When the cluster weights $K : X \times X \to \mathbb{R}$8 are imbalanced, shortcut eigenfunctions arise. Spectral bias implies that, during gradient flow, modes with large $K : X \times X \to \mathbb{R}$9 are learned fastest, so shortcut-aligned features associated with large $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$0 emerge early. At convergence, the solution for linear MSE-regression has the form:

$T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$1

with

$T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$2

If the cluster means $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$3 are orthogonal, this simplifies to

$T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$4

The weight $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$5 grows monotonically in $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$6. The denominator $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$7 depends on within-cluster variance: nonzero variances amplify the dominance of shortcut features, whereas if $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$8 all weights share a denominator, suppressing shortcut amplification (Lim et al., 3 Feb 2026).

4. Robustness of NTK Spectral Bias to Margin Control

Previous work suggested that maximal $T_K[g](x) := \int_X K(x, x')\,g(x')\,d\rho(x').$9 margin bias (the result of cross-entropy optimization on separable data) is responsible for shortcut learning. Introducing an SD regularization term, $\phi_i \in L^2(X, \rho)$0, controls the margin, shifting solutions toward ridge regression. However, both cross-entropy with SD and MSE with SD approach the same optimal ratios of feature weights $\phi_i \in L^2(X, \rho)$1 as $\phi_i \in L^2(X, \rho)$2, matching the MSE solution. Thus, shortcut bias, favored by high $\phi_i \in L^2(X, \rho)$3 clusters, persists even under aggressive margin control; it is an intrinsic property of the NTK spectrum, not merely a consequence of margin-seeking dynamics (Lim et al., 3 Feb 2026).

5. Empirical Observations in Nonlinear Architectures

The dominance of shortcut NTK eigenfunctions extends beyond linear models. Two-layer ReLU networks, trained on synthetic Patched-MNIST and Colored-MNIST datasets, exhibit top empirical NTK eigenfunctions that align with spurious features (patch or color), verified via saliency maps of $\phi_i \in L^2(X, \rho)$4. Lower-order eigenfunctions correspond to semantically meaningful digit shapes.

A metric termed “availability,” $\phi_i \in L^2(X, \rho)$5, quantifies NTK alignment; shortcut labels typically have much higher availability than core labels throughout training, reflecting their spectral accessibility.

For pretrained ResNet-18 on Waterbirds, CelebA, and Dogs-vs-Cats, empirical NTK analysis confirms that shortcut labels maintain higher NTK availability regardless of the loss function (cross-entropy, MSE, SD). Notably, altering the shortcut-strength (e.g., patch size in Patched-MNIST) scales shortcut availability. Pretraining (e.g., on ImageNet) can sometimes invert these spectral relationships, making core-label features easier to access and revealing the role of initialization (Lim et al., 3 Feb 2026).

6. Summary and Implications

The spectral decomposition of the NTK, and the resulting dominance of shortcut-aligned eigenfunctions, offers a unifying mechanism for shortcut learning across architectures and datasets. Primary determinants are the presence of high-mixing-weight (large $\phi_i \in L^2(X, \rho)$6) clusters and nonzero within-cluster variances. Neural networks preferentially fit the highest-eigenvalue NTK features early and retain them after training, even in the presence of strong regularization or margin controls. These findings, established analytically for linear Gaussian mixtures and empirically validated in complex networks, underscore the robustness of NTK-based spectral bias as an explanatory paradigm for shortcut learning in deep learning (Lim et al., 3 Feb 2026).