Neural Tangent Kernels Overview

• Neural Tangent Kernels are a mathematical framework that links the training dynamics of wide neural networks with kernel methods and Gaussian processes.
• The NTK formalism transforms non-convex training into a tractable, convex optimization in function space, enabling precise analytic insights.
• Its positive-definiteness and constant behavior in the infinite-width limit underpin convergence guarantees and explain benefits like early stopping.

The Neural Tangent Kernel (NTK) is a mathematical framework that connects the training dynamics of wide neural networks to kernel methods and Gaussian processes. Developed to analyze the behavior of artificial neural networks (ANNs) in the infinite-width limit, the NTK provides a rigorous means of studying the evolution, convergence, and generalization of neural networks by lifting analysis from high-dimensional parameter space to function space. The NTK is defined via the inner product of network gradients with respect to parameters and, in the infinite-width regime, becomes a deterministic and constant kernel during training. This underpinning enables analytic treatment of network learning dynamics and influences modern theoretical and empirical understanding of deep learning.

1. Definition and Formal Properties

The Neural Tangent Kernel in an L-layer neural network with realization function $F^{(L)}(\theta) = f_\theta$ is defined as the sum of the outer products of gradients of the output with respect to network parameters: $$\Theta^{(L)}(\theta) = \sum_p \left(\partial_{\theta_p} F^{(L)}(\theta)\right) \otimes \left(\partial_{\theta_p} F^{(L)}(\theta)\right)$$ In the infinite-width limit, the output of the network at initialization converges to a Gaussian process with covariance $\Sigma^{(L)}$. The NTK then acts as a kernel defining the inner product on function space, allowing for the paper of $f_\theta$'s evolution by working directly in that space, as opposed to the parameter space, thereby leveraging convexity properties.

2. Training Dynamics and Kernel Gradient Descent

Even though network training typically involves gradient descent in a highly non-convex parameter landscape, the function $f_\theta$ follows the gradient flow with respect to the NTK in function space: $$\partial_t f_{\theta(t)} = -\nabla_{\mathbb{K}} C|_{f_{\theta(t)}}$$ where the functional gradient is determined by the NTK: $$\nabla_{\mathbb{K}} C|_{f}(x) = \frac{1}{N} \sum_j K(x, x_j)\, d|_f(x_j)$$ for empirical data. This makes the optimization convex and analytically tractable in function space. The original non-convexity is bypassed, and the network's output evolution is defined by a deterministic kernel dynamic in the infinite-width limit.

3. Infinite-Width Limit: Convergence and Constancy

As the widths $n_1,\ldots, n_{L-1} \to \infty$, the NTK $\Theta^{(L)}(t)$ converges in probability—at initialization—to a deterministic kernel: $$\Theta^{(L)}(\theta) \longrightarrow \Theta^{(L)}_{\infty} \otimes \mathrm{Id}_{n_L}$$ The limiting NTK is recursively computable, beginning at

$$\Theta^{(1)}_{\infty}(x,x') = \Sigma^{(1)}(x,x') = \frac{1}{n_0} x^\top x' + \beta^2$$

and inductively via

$$\Theta^{(L+1)}_{\infty}(x,x') = \Theta^{(L)}_{\infty}(x,x')\cdot d\Sigma^{(L+1)}(x,x') + \Sigma^{(L+1)}(x,x')$$

where $$d\Sigma^{(L+1)}(x,x') = \mathbb{E}_{f \sim N(0, \Sigma^{(L)})} [\sigma'(f(x))\sigma'(f(x'))]$$. Throughout training, the NTK remains asymptotically constant in the infinite-width regime, leading to purely linear differential equation dynamics for the evolution of $f_\theta$.

4. Positive-Definiteness and Global Convergence

Guaranteeing convergence in function space requires that the limiting NTK be positive-definite. The NTK's positive-definiteness is proved for data on the unit sphere and for non-polynomial, Lipschitz activation functions (with depth $L\geq 2$). The proof employs recursive expressions for the NTK and Hermite expansions of activation functions, connecting to Schoenberg’s theorem to ensure that kernels $f(x^\top x')$ with infinitely many positive coefficients are positive-definite.

The positive-definiteness condition ensures:

• The kernel gradient norm is strictly positive except at the minimum,
• The cost decreases monotonically,
• Convergence to the global minimum in function space.

5. Least-Squares Regression and Linear Dynamics

For the least-squares loss,

$C(f) = \frac{1}{2}||f-f^*||^2$

the evolution under NTK gradient descent in the infinite-width regime is given by: $$\partial_t f_{\theta(t)} = -\Phi_{\Theta^{(L)}_{\infty} \otimes \mathrm{Id}} \Big( \left\langle f_\theta - f^*, \cdot \right\rangle \Big)$$ The solution is explicitly: $f_t = f^* + e^{-t\Pi}(f_0-f^*)$ where $\Pi$ is the NTK-induced linear operator. Decomposing onto kernel principal components, each component $(f_0-f^*)_i$ decays exponentially at rate $\exp(-\lambda_i t)$, with $\lambda_i$ the corresponding NTK eigenvalues.

6. Principal Components and Early Stopping

The NTK's eigenfunctions (“kernel principal components”) set the convergence rates: fast for high-eigenvalue directions (typically low frequency), slow for low-eigenvalue ones (often noise or high frequency). Early stopping thus acts as an effective regularization mechanism: halting training before overfitting to slower-decaying, less significant directions. This provides a concrete theoretical motivation for the empirical effectiveness of early stopping in practice.

7. Numerical Observations and Practical Implications

Empirical studies conducted in the paper support theoretical predictions:

• For wide networks, the empirical NTK converges rapidly to the theoretical NTK and becomes almost invariant over the course of training. Narrow networks show greater NTK variation.
• Regression settings reveal function evolution along NTK principal components with decaying Gaussian-distributed error as predicted.
• Experiments on MNIST demonstrate that rapid initial convergence occurs along dominant NTK principal components, and early stopping yields improved generalization.

The NTK enables analyzing the training and generalization of deep, overparameterized neural networks. By mapping non-convex optimization to kernel regression in function space, it unifies deep learning’s empirical observations (such as strong overfitting resistance and the merits of early stopping) with kernel theory and Gaussian processes. Positive-definiteness of the NTK in the infinite-width setting, explicit linear dynamics for regression, and eigenstructure-driven training flow underscore the power and limitations of this approach. It also clarifies when and how overparameterization leads to favorable optimization and generalization—bridging classical kernel methods and modern deep learning.