Form of the Projection Matrix Linking Infinite-Width NTK Eigenfunctions to Finite-Width NTK Features

Determine the form and structural properties of the projection matrix A that maps infinite-width neural tangent kernel (NTK) eigenfunctions to the finite-width NTK feature subspace in realistic neural network architectures, including its distributional characteristics and dependence on architecture and initialization, to enable precise modeling of finite-width dynamics.

Background

To model finite-width neural networks in the lazy/linearized regime, the paper represents finite-width NTK features as linear projections of infinite-width NTK eigenfunctions via a matrix A. This matrix encapsulates how finite-width features relate to the infinite-width basis and is crucial for analytically tractable models of training dynamics and generalization.

While the paper assumes A has i.i.d. entries with mean zero and unit variance for tractability and to satisfy consistency with the infinite-width limit, the authors explicitly note that in realistic scenarios—when projecting actual infinite-width NTK eigenfunctions to the empirical finite-width NTK—the true structure of A is generally unknown. Characterizing A would improve the fidelity of theoretical models to practical architectures.