Empirical Neural Tangent Kernel

• Empirical NTK is a finite-width, data-dependent kernel defined by the gradients of network outputs, capturing dynamic feature evolution during training.
• It serves as a diagnostic tool connecting deep learning dynamics, kernel methods, and feature learning through both theoretical insights and practical approximations.
• Efficient computation methods like Jacobian–vector products and low-rank approximations enable scalable NTK analysis across diverse neural architectures.

The empirical neural tangent kernel (NTK), often denoted as K_emp, is a data-dependent, finite-width instantiation of the neural tangent kernel, defined as a kernel over training samples using the gradient of the neural network output with respect to its parameters. Unlike the idealized infinite-width NTK, which is constant throughout training and admits closed-form analysis, the empirical NTK of finite neural networks generally moves in a high-dimensional, architecture- and data-dependent manner under stochastic gradient descent. Empirical NTK analysis has become a central tool in understanding the relationship between deep-learning dynamics, kernel methods, and the phenomenon of feature learning. This article documents precise definitions, computational approaches, regime validity, practical applications, and caveats of the empirical NTK.

1. Mathematical Definition and Finite-Width Construction

For a neural network $f_\theta(x)\in\mathbb{R}^L$ with parameters $\theta\in\mathbb{R}^P$ and input $x\in\mathbb{R}^d$, the empirical NTK Gram matrix on a dataset $\{x_i\}_{i=1}^n$ at parameter setting $\theta$ is defined as

$$K(\theta)_{ij} = \sum_{l=1}^L \langle \nabla_\theta f_l(\theta, x_i), \nabla_\theta f_l(\theta, x_j) \rangle,$$

where $f_l$ is the $l$-th output coordinate. For scalar outputs ($L=1$) this reduces to

$$K_\theta(x,x') = \nabla_\theta f(\theta, x)^\top \nabla_\theta f(\theta, x').$$

At initialization $\theta\in\mathbb{R}^P$0, the empirical NTK is $\theta\in\mathbb{R}^P$1. For a fixed parameter vector $\theta\in\mathbb{R}^P$2, as after training, this kernel becomes the "after kernel" $\theta\in\mathbb{R}^P$3, whose features have been shaped by data and training evolution (Long, 2021).

For typical architectures, computing $\theta\in\mathbb{R}^P$4 requires evaluating the parameter gradients $\theta\in\mathbb{R}^P$5 for each sample. For multi-output architectures, the empirical NTK can take the form of a block matrix in $\theta\in\mathbb{R}^P$6, or a scalar surrogate can be formed by summing or projecting across output indices (Mohamadi et al., 2022).

2. Computational Approaches and Approximations

Direct computation of the empirical NTK via full Jacobians scales as $\theta\in\mathbb{R}^P$7 in time and $\theta\in\mathbb{R}^P$8 in memory, often prohibitive for large $\theta\in\mathbb{R}^P$9 or architectures with $x\in\mathbb{R}^d$0. Multiple strategies have been developed to mitigate this:

• Explicit differentiation methods: Hand-derived formulas—especially for MLPs—reduce per-batch memory costs and accelerate computation (Engel et al., 2022). E.g., for MLPs with $x\in\mathbb{R}^d$1 layers and $x\in\mathbb{R}^d$2 samples, explicit methods attain $x\in\mathbb{R}^d$3 time per layer.
• Autodifferentiation / Jacobian–vector products: Leverage automatic differentiation frameworks (e.g., PyTorch autograd) to compute individual rows/columns of the NTK on the fly, avoiding explicit storage of the full Jacobian (Engel et al., 2022).
• Layerwise decomposition: The NTK decomposes additively over parameter blocks; layerwise computation allows efficient storage and inspection (Engel et al., 2022).
• Randomized trace estimation: For NTK statistics (e.g., trace, effective rank), matrix-free Hutch++ or Hutchinson estimators dramatically lower computational time, e.g., achieving $x\in\mathbb{R}^d$4–$x\in\mathbb{R}^d$5 speedups compared to explicit NTK formation (Hazelden, 13 Nov 2025).
• Low-rank and randomized projection approximations: Projection-based surrogates (e.g., proj-trNTK, proj-pNTK) reduce the parameter dimension, with provable error bounds under Johnson–Lindenstrauss (Engel et al., 2023).
• Sum-of-logits and diagonal surrogates: For classification, "pseudo-NTK" approximations that sum or average over output units yield $x\in\mathbb{R}^d$6 kernels, reducing both compute and memory by factors of $x\in\mathbb{R}^d$7 and $x\in\mathbb{R}^d$8 compared to the true $x\in\mathbb{R}^d$9 Gram matrix (Mohamadi et al., 2022).

These developments have made it feasible to analyze empirical NTK structure and statistics for CNNs, RNNs, and transformers on large datasets (Hazelden, 13 Nov 2025, Engel et al., 2022).

3. Evolution Under Training and Comparison to Infinite-Width Theory

The empirical NTK is in general nonstationary in finite-width networks. At width $\{x_i\}_{i=1}^n$0 and with infinitesimal learning rates, the NTK converges to a deterministic, data-independent kernel that remains exactly constant throughout training—this is the NTK theory regime (Samarin et al., 2020). However, empirical studies reveal that:

• For standard-width convolutional architectures (e.g., LeNet, AlexNet), the empirical NTK moves substantially during training; linearized models based on the initial kernel poorly track both early trajectories and final generalization, unless $\{x_i\}_{i=1}^n$1 is taken to be extremely large (e.g., $\{x_i\}_{i=1}^n$2 for LeNet on MNIST) (Samarin et al., 2020).
• In realistic training of deep networks, all key measures—NTK velocity, error barrier, loss landscape geometry—shift rapidly in a "chaotic transient" phase (first $\{x_i\}_{i=1}^n$3–$\{x_i\}_{i=1}^n$4 epochs), after which the NTK enters a regime of linear drift (Fort et al., 2020). Substantial feature learning occurs during the transient, poorly captured by the linearization at initialization.
• The empirical NTK learns class structure and adjusts its spectrum: leading eigenvectors rapidly align to the target function, with $\{x_i\}_{i=1}^n$5 of label energy captured by the top $\{x_i\}_{i=1}^n$6 eigenvectors (out of $\{x_i\}_{i=1}^n$7) shortly after training starts (Kopitkov et al., 2019). The empirical Gram matrix spectrum grows under learning rate decay and becomes sharply class block-structured in well-trained classifiers (Seleznova et al., 2023).
• Under mean-squared error loss, NTK drift is moderate; under cross-entropy, empirical NTK can diverge without bound for finite width, precluding uniform-in-time convergence to the infinite-width NTK (Yu et al., 15 Apr 2025). This marks a sharp distinction between kernel-regression and classification dynamics.

Therefore, the empirical NTK only approximates kernel methods (NTK regime) in the combined limit of large width, small learning rate, and minimal feature evolution, which rarely holds in practice for high-performance models (Samarin et al., 2020, Ortiz-Jiménez et al., 2021, Liu et al., 19 Jan 2025).

4. Applications: Surrogate Modeling, Uncertainty Quantification, and Data Attribution

Despite regime breakdowns, the empirical NTK enables a range of practical tasks:

• Surrogate model construction: The empirical NTK forms the basis for kernel-based generalized linear models (kGLM) that faithfully approximate neural network predictions. In classification, kernel machines fitted to the empirical NTK can attain up to $\{x_i\}_{i=1}^n$8–$\{x_i\}_{i=1}^n$9 Kendall’s $\theta$0 rank correlation with the original network’s softmax outputs, with only small drops in accuracy (Engel et al., 2023). Efficient approximate NTKs can yield similar fidelity at orders-of-magnitude lower cost.
• Data attribution and XAI: Empirical NTK surrogates offer principled, faithful explain-by-example and data attribution via influence scores that are linear in the kernel. Random projection variants trade small decreases in attribution fidelity for dramatic computational gains (Engel et al., 2023).
• Uncertainty quantification: Sampling-based ensemble methods on the linearized network about the trained weights (using the empirical NTK) efficiently generate GP-like posterior predictive distributions. These methods (e.g., NUQLS) match or exceed competitive UQ metrics at a fraction of the computational cost of deep ensembles, Laplace, or SWAG (Wilson et al., 5 Feb 2025).
• Differential privacy: Empirical NTK features serve as a competitively expressive, nonpublic-data-dependent basis for differentially private data generation, outperforming other synthetic-data methods on numerous benchmarks without relying on public perceptual features (Yang et al., 2023).

These applications rely on the discriminative power and local geometry encoded in the empirical NTK feature map $\theta$1, which can be efficiently extracted for moderate architectures.

5. Structure, Block Patterns, and Spectral Alignment

The empirical NTK displays remarkable structural adaptation during training:

• Block-structure and class alignment: In classification, the NTK develops a block-diagonal pattern, with high within-class correlations and weaker between-class interactions (Seleznova et al., 2023). This aligns with the target label Gram matrix and underpins theoretical analyses of neural collapse and class structure emergence.
• Spectral alignment: The top eigenspace of the empirical NTK rapidly aligns with the label vector; the network output remains close to the span of these leading eigenvectors throughout training (Kopitkov et al., 2019).
• Invariance and globalness: Features derived from the after kernel (empirical NTK at end of training) are more global—less invariant to "global" image perturbations—and more invariant to small shifts/rotations/zooms than those at initialization. Architecture and learning rate control the degree and type of invariance acquired (Long, 2021).

Such structure emerges even in the absence of infinite width, with depth, architecture, and learning rate controlling the extent and rate of spectrum adaptation.

6. Practical Limitations and Validity Concerns

Despite its utility, the empirical NTK is not a globally reliable surrogate for neural network learning:

• Failure to match generalization and accuracy in finite width: In moderate-width CNNs (e.g., AlexNet, LeNet), the linearized model based on empirical NTK at initialization systematically underperforms the true neural net, with gaps in accuracy up to $\theta$2 on benchmarks like CIFAR-10 and ImageNet subsets (Samarin et al., 2020, Ortiz-Jiménez et al., 2021).
• Lack of correspondence with network improvements: Depth-induced improvements in trained network accuracy are not captured by any NTK variant; adding layers to the network changes performance, but the corresponding NTK-induced kernel predictor does not reflect this improvement (Liu et al., 19 Jan 2025).
• Divergence under cross-entropy loss: For classification, the empirical NTK can diverge over training time, invalidating the assumption that it converges to or remains close to the infinite-width kernel. This leads to a breakdown of NTK-based generalization guarantees in realistic classification training (Yu et al., 15 Apr 2025).
• Computational scaling: For multi-output or large datasets, even efficient approximations can soon become memory-bound; e.g., for $\theta$3 and $\theta$4, the true NTK would require terabytes of RAM (Mohamadi et al., 2022). Approximate schemes based on sum-of-logits, random projections, or block approximations are essential for tractability, but can degrade fidelity for narrow, highly structured, or non-ReLU architectures.

These results collectively highlight that the empirical NTK serves as a valuable, theoretically motivated diagnostic in specific regimes, but it does not provide a universally faithful surrogate for real neural network learning, feature evolution, or generalization outside the extremely wide, lazy-training limit.

7. Summary Table: Key Regimes and Limitations of Empirical NTK

Training Regime / Task	Empirical NTK Validity	Failure Modes / Observations
Infinite-width, small LR, regression	Excellent	Kernel stays constant, matches GP behavior (Samarin et al., 2020, Ortiz-Jiménez et al., 2021)
Standard-width CNN, vision classification	Fails	Nonlinear model outperforms kernel; significant feature learning (Samarin et al., 2020, Ortiz-Jiménez et al., 2021)
Overtraining (classification)	Diverges	Empirical NTK grows unbounded after loss minimization (Yu et al., 15 Apr 2025)
Block structure (late training)	Accurate	Reveals neural collapse and class alignment (Seleznova et al., 2023)
Data attribution / XAI	Surrogate, approximate	High rank correlation for attributions with low-cost variants (Engel et al., 2023)

In conclusion, empirical NTK analysis is indispensable for understanding feature learning, early kernel alignment, and the onset of spectral structures in deep networks. However, researchers should be cognizant of its regime of validity, computational trade-offs, and failure to predict learned network improvements, especially as networks depart from infinite-width, lazy dynamics (Samarin et al., 2020, Ortiz-Jiménez et al., 2021, Liu et al., 19 Jan 2025, Yu et al., 15 Apr 2025).