--- title: Empirical Neural Tangent Kernel url: https://www.emergentmind.com/topics/empirical-neural-tangent-kernel type: topic --- # Empirical Neural Tangent Kernel The empirical neural tangent kernel (NTK), often denoted as K_emp, is a data-dependent, finite-width instantiation of the neural tangent kernel, defined as a kernel over training samples using the gradient of the neural network output with respect to its parameters. Unlike the idealized infinite-width NTK, which is constant throughout training and admits closed-form analysis, the empirical NTK of finite neural networks generally moves in a high-dimensional, architecture- and data-dependent manner under stochastic gradient descent. Empirical NTK analysis has become a central tool in understanding the relationship between deep-learning dynamics, kernel methods, and the phenomenon of feature learning. This article documents precise definitions, computational approaches, regime validity, practical applications, and caveats of the empirical NTK. ## 1. Mathematical Definition and Finite-Width Construction For a neural network $f_\theta(x)\in\mathbb{R}^L$ with parameters $\theta\in\mathbb{R}^P$ and input $x\in\mathbb{R}^d$, the empirical NTK Gram matrix on a dataset $\{x_i\}_{i=1}^n$ at parameter setting $\theta$ is defined as $$ K(\theta)_{ij} = \sum_{l=1}^L \langle \nabla_\theta f_l(\theta, x_i), \nabla_\theta f_l(\theta, x_j) \rangle, $$ where $f_l$ is the $l$-th output coordinate. For scalar outputs ($L=1$) this reduces to $$ K_\theta(x,x') = \nabla_\theta f(\theta, x)^\top \nabla_\theta f(\theta, x'). $$ At initialization $\theta_0$, the empirical NTK is $K^\mathrm{emp} = K(\theta_0)$. For a fixed parameter vector $\theta^*$, as after training, this kernel becomes the "after kernel" $K_T = K(\theta_T)$, whose features have been shaped by data and training evolution [2105.10585]. For typical architectures, computing $K^\mathrm{emp}$ requires evaluating the parameter gradients $\nabla_\theta f$ for each sample. For multi-output architectures, the empirical NTK can take the form of a block matrix in $nL\times nL$, or a scalar surrogate can be formed by summing or projecting across output indices [2206.12543]. ## 2. Computational Approaches and Approximations Direct computation of the empirical NTK via full Jacobians scales as $\mathcal{O}(n^2 P L^2)$ in time and $\mathcal{O}(nP L)$ in memory, often prohibitive for large $n$ or architectures with $P\gg n$. Multiple strategies have been developed to mitigate this: - **Explicit differentiation methods**: Hand-derived formulas—especially for MLPs—reduce per-batch memory costs and accelerate computation [2205.12372]. E.g., for MLPs with $L$ layers and $n$ samples, explicit methods attain $\mathcal{O}(Ln^2)$ time per layer. - **Autodifferentiation / Jacobian–vector products**: Leverage automatic differentiation frameworks (e.g., PyTorch autograd) to compute individual rows/columns of the NTK on the fly, avoiding explicit storage of the full Jacobian [2205.12372]. - **Layerwise decomposition**: The NTK decomposes additively over parameter blocks; layerwise computation allows efficient storage and inspection [2205.12372]. - **Randomized trace estimation**: For NTK statistics (e.g., trace, effective rank), matrix-free Hutch++ or Hutchinson estimators dramatically lower computational time, e.g., achieving $100\times$–$1000\times$ speedups compared to explicit NTK formation [2511.10796]. - **Low-rank and randomized projection approximations**: Projection-based surrogates (e.g., proj-trNTK, proj-pNTK) reduce the parameter dimension, with provable error bounds under Johnson–Lindenstrauss [2305.14585]. - **Sum-of-logits and diagonal surrogates**: For classification, "pseudo-NTK" approximations that sum or average over output units yield $N \times N$ kernels, reducing both compute and memory by factors of $O^2$ and $O^3$ compared to the true $(NO)\times(NO)$ Gram matrix [2206.12543]. These developments have made it feasible to analyze empirical NTK structure and statistics for CNNs, RNNs, and transformers on large datasets [2511.10796, 2205.12372]. ## 3. Evolution Under Training and Comparison to Infinite-Width Theory The empirical NTK is in general nonstationary in finite-width networks. At width $m\to\infty$ and with infinitesimal learning rates, the NTK converges to a deterministic, data-independent kernel that remains exactly constant throughout training—this is the NTK theory regime [2006.13645]. However, empirical studies reveal that: - For **standard-width** convolutional architectures (e.g., LeNet, AlexNet), the empirical NTK moves substantially during training; linearized models based on the initial kernel poorly track both early trajectories and final generalization, unless $m$ is taken to be extremely large (e.g., $m \gtrsim 60$ for LeNet on MNIST) [2006.13645]. - In realistic training of deep networks, all key measures—NTK velocity, error barrier, loss landscape geometry—shift rapidly in a "chaotic transient" phase (first $2$–$3$ epochs), after which the NTK enters a regime of linear drift [2010.15110]. Substantial feature learning occurs during the transient, poorly captured by the linearization at initialization. - The empirical NTK learns class structure and adjusts its spectrum: leading eigenvectors rapidly align to the target function, with $90\%$ of label energy captured by the top $20$ eigenvectors (out of $10^4$) shortly after training starts [1910.08720]. The empirical Gram matrix spectrum grows under learning rate decay and becomes sharply class block-structured in well-trained classifiers [2305.16427]. - Under mean-squared error loss, NTK drift is moderate; under cross-entropy, empirical NTK can diverge without bound for finite width, precluding uniform-in-time convergence to the infinite-width NTK [2504.11130]. This marks a sharp distinction between kernel-regression and classification dynamics. Therefore, the empirical NTK only approximates kernel methods (NTK regime) in the combined limit of large width, small learning rate, and minimal feature evolution, which rarely holds in practice for high-performance models [2006.13645, 2106.06770, 2501.10929]. ## 4. Applications: Surrogate Modeling, Uncertainty Quantification, and Data Attribution Despite regime breakdowns, the empirical NTK enables a range of practical tasks: - **Surrogate model construction**: The empirical NTK forms the basis for kernel-based generalized linear models (kGLM) that faithfully approximate neural network predictions. In classification, kernel machines fitted to the empirical NTK can attain up to $0.78$–$0.85$ Kendall’s $\tau$ rank correlation with the original network’s softmax outputs, with only small drops in accuracy [2305.14585]. Efficient approximate NTKs can yield similar fidelity at orders-of-magnitude lower cost. - **Data attribution and XAI**: Empirical NTK surrogates offer principled, faithful explain-by-example and data attribution via influence scores that are linear in the kernel. Random projection variants trade small decreases in attribution fidelity for dramatic computational gains [2305.14585]. - **Uncertainty quantification**: Sampling-based ensemble methods on the linearized network about the trained weights (using the empirical NTK) efficiently generate GP-like posterior predictive distributions. These methods (e.g., NUQLS) match or exceed competitive UQ metrics at a fraction of the computational cost of deep ensembles, Laplace, or SWAG [2502.02870]. - **Differential privacy**: Empirical NTK features serve as a competitively expressive, nonpublic-data-dependent basis for differentially private data generation, outperforming other synthetic-data methods on numerous benchmarks without relying on public perceptual features [2303.01687]. These applications rely on the discriminative power and local geometry encoded in the empirical NTK feature map $x\mapsto\nabla_\theta f(x;\theta)$, which can be efficiently extracted for moderate architectures. ## 5. Structure, Block Patterns, and Spectral Alignment The empirical NTK displays remarkable structural adaptation during training: - **Block-structure and class alignment**: In classification, the NTK develops a block-diagonal pattern, with high within-class correlations and weaker between-class interactions [2305.16427]. This aligns with the target label Gram matrix and underpins theoretical analyses of neural collapse and class structure emergence. - **Spectral alignment**: The top eigenspace of the empirical NTK rapidly aligns with the label vector; the network output remains close to the span of these leading eigenvectors throughout training [1910.08720]. - **Invariance and globalness**: Features derived from the after kernel (empirical NTK at end of training) are more global—less invariant to "global" image perturbations—and more invariant to small shifts/rotations/zooms than those at initialization. Architecture and learning rate control the degree and type of invariance acquired [2105.10585]. Such structure emerges even in the absence of infinite width, with depth, architecture, and learning rate controlling the extent and rate of spectrum adaptation. ## 6. Practical Limitations and Validity Concerns Despite its utility, the empirical NTK is not a globally reliable surrogate for neural network learning: - **Failure to match generalization and accuracy in finite width**: In moderate-width CNNs (e.g., AlexNet, LeNet), the linearized model based on empirical NTK at initialization systematically underperforms the true neural net, with gaps in accuracy up to $20\%$ on benchmarks like CIFAR-10 and ImageNet subsets [2006.13645, 2106.06770]. - **Lack of correspondence with network improvements**: Depth-induced improvements in trained network accuracy are not captured by any NTK variant; adding layers to the network changes performance, but the corresponding NTK-induced kernel predictor does not reflect this improvement [2501.10929]. - **Divergence under cross-entropy loss**: For classification, the empirical NTK can diverge over training time, invalidating the assumption that it converges to or remains close to the infinite-width kernel. This leads to a breakdown of NTK-based generalization guarantees in realistic classification training [2504.11130]. - **Computational scaling**: For multi-output or large datasets, even efficient approximations can soon become memory-bound; e.g., for $N\sim50,000$ and $O\sim10$, the true NTK would require terabytes of RAM [2206.12543]. Approximate schemes based on sum-of-logits, random projections, or block approximations are essential for tractability, but can degrade fidelity for narrow, highly structured, or non-ReLU architectures. These results collectively highlight that the empirical NTK serves as a valuable, theoretically motivated diagnostic in specific regimes, but it does not provide a universally faithful surrogate for real neural network learning, feature evolution, or generalization outside the extremely wide, lazy-training limit. ## 7. Summary Table: Key Regimes and Limitations of Empirical NTK | Training Regime / Task | Empirical NTK Validity | Failure Modes / Observations | |:--------------------------------:|:-------------------------:|:-------------------------------------| | Infinite-width, small LR, regression | Excellent | Kernel stays constant, matches GP behavior [2006.13645, 2106.06770] | | Standard-width CNN, vision classification | Fails | Nonlinear model outperforms kernel; significant feature learning [2006.13645, 2106.06770] | | Overtraining (classification) | Diverges | Empirical NTK grows unbounded after loss minimization [2504.11130] | | Block structure (late training) | Accurate | Reveals neural collapse and class alignment [2305.16427] | | Data attribution / XAI | Surrogate, approximate | High rank correlation for attributions with low-cost variants [2305.14585] | In conclusion, empirical NTK analysis is indispensable for understanding feature learning, early kernel alignment, and the onset of spectral structures in deep networks. However, researchers should be cognizant of its regime of validity, computational trade-offs, and failure to predict learned network improvements, especially as networks depart from infinite-width, lazy dynamics [2006.13645, 2106.06770, 2501.10929, 2504.11130]. Source: https://www.emergentmind.com/topics/empirical-neural-tangent-kernel