Clarify width growth requirements in neural tangent kernel analyses

Determine whether the number of neurons required to satisfy the "sufficiently large" condition in neural tangent kernel (NTK) analyses of over-parameterized neural networks must grow exponentially with the sample size or at a slower rate, and specify the minimal width-growth regime under which NTK-derived guarantees hold.

Background

In surveying NTK-based results, the paper points out that equivalence to deep networks is often established only pointwise and not for global L2 errors. Moreover, these analyses typically assume that the number of neurons is "sufficiently large" without a precise quantitative requirement.

The authors explicitly note the lack of clarity on how the required width should scale with sample size, highlighting the need for a definitive characterization of the minimal growth rate necessary for NTK conclusions to transfer to practical deep networks.