A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel

Abstract: A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of $\textbf{when}$ and $\textbf{by how much}$ has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its $\textbf{Fourier complexity}$, which controls NTK kernel regression, and its $\textbf{architectural complexity}$, which controls learning over depth-$L$, width-$w$ ReLU networks with the variation norm of the weights bounded by $R$. We first characterize the minimax rate of the architecture class $\mathcal{C}_{L,w,R}$, pinning it down up to a single factor of $L$: between $Ω(Lw<sup>2R<sup>2/n)$ and $\tilde{O}(L<sup>2w<sup>2R<sup>2/n)$. We then show the NTK estimator sits $\textbf{exponentially}$ above this floor whenever the two complexities decouple: for the depth-$L$ iterated sawtooth, NTK regression needs $Ω(4<sup>L)$ samples while the minimax floor is polynomial in $L$. Numerical experiments confirm the theoretical claims: on bandlimited smooth targets, the NTK is competitive or better, while on the hypercube sparse-parity model, a standard two-layer network beats the NTK by four to six orders of magnitude in test error. The gap is thus a function-space property, a mismatch between the kernel's smoothness bias and the target's compositional structure, rather than a generic kernel-versus-network phenomenon.

• The paper shows that when targets have high Fourier complexity but low architectural complexity, NTK regression becomes exponentially sub-optimal compared to deep networks.
• It rigorously analyzes deep ReLU architectures by establishing minimax learning rates and a clear function-space separation between NTK and compositional regimes.
• Empirical and theoretical evidence confirms that NTK’s smoothness bias leads to an exponential sample complexity gap for structured, high-frequency targets.

Function-Space Dichotomy and the Exponential Sub-Optimality of the Neural Tangent Kernel

Introduction

This paper investigates the relationship between compositional function structure and the statistical performance of the Neural Tangent Kernel (NTK) regime, contrasting it with weight-decay-trained deep ReLU networks in the "rich" (i.e., non-kernel, non-linearized) regime. The central theme is the function-space mismatch that arises between NTK's smoothness bias—quantified via its RKHS (Sobolev) norm—and the architectural (deep variation) complexity that governs actual compositional network learning. The main contribution is a rigorous function-space dichotomy: when targets have high Fourier complexity yet low architectural complexity, the NTK estimator is shown to be exponentially sub-optimal relative to the minimax optimal rates achievable by the deep network architecture class.

Complexity Dichotomy: Fourier vs. Architectural

Central to the analysis is the dichotomy between two measures of target complexity:

• Fourier complexity: Governs NTK kernel regression; for targets whose spectral mass is at frequency $k^\star$, the learning cost in the NTK's RKHS scales as $(k^\star)^2$.
• Architectural (variation) complexity: Governs the minimax rate for weight-decay-trained ReLU networks, quantified via the deep variation norm. For functions with succinct compositional realization—i.e., constructed by a compact deep network—the architectural complexity can remain low even as Fourier complexity grows.

This decoupling leads to a pronounced gap: targets like the depth-$L$ Telgarsky sawtooth, efficiently described by a compact depth-$L$ network but with exponentially high-frequency content, require exponentially many more samples in the NTK regime than in the compositional regime.

After introducing this dichotomy, the paper provides a detailed theoretical and empirical analysis showcasing and quantifying this gap.

Figure 1: The Telgarsky sawtooth $g_L$ for $L=1,2,3$, exhibiting exponential growth in oscillations with only linear increase in the variation norm.

Minimax Rates for Deep ReLU Architecture Classes

A fundamental result is the characterization of the minimax learning rate for compositional ReLU networks, specifically the class $C_{L,w,R}$ of depth-$L$, width-$w$ ReLU networks with $\leq R$ variation norm. The minimax $(k^\star)^2$0 risk is established as:

$(k^\star)^2$1

This positions the statistical "floor" for sample complexity at a rate polynomial in $(k^\star)^2$2, $(k^\star)^2$3, and $(k^\star)^2$4, up to a single factor of $(k^\star)^2$5. This baseline crucially enables a nontrivial functional gap quantification: the NTK estimator is measured not against a specific neural network, but against the best possible estimator within the architecture class.

Exponential Sample Complexity Gap for NTK

By leveraging the well-established equivalence of the NTK RKHS norm and the Sobolev norm (i.e., a penalization of high-frequency content), the paper rigorously proves that NTK regression requires exponentially more samples than the compositional minimax rate whenever a target's Fourier complexity significantly exceeds its architectural complexity.

Explicitly, for the depth-$(k^\star)^2$6 sawtooth function with dominant frequency $(k^\star)^2$7, the NTK sample requirement is

$(k^\star)^2$8

while the compositional minimax rate is polynomial in $(k^\star)^2$9, $L$0, and $L$1. The leading mechanism is the slow eigenvalue decay of the NTK on $L$2, which, due to the incapacity of depth to accelerate spectral decay (the "deep equals shallow" property), cannot efficiently represent targets with high compositional oscillatory structure.

Figure 3: NTK eigenvalue decay on $L$3 for multiple depths, showing parallel log-log spectra with slope approximately $L$4, confirming slow decay and the inability to efficiently capture high-frequency components.

Empirical Demonstrations and the Role of Target Structure

The empirical section rigorously confirms three main predictions:

1. Spectral Gap Validity: The empirical eigenvalue spectrum of the NTK matches the theoretical $L$5 decay, independent of NTK depth.
2. No Gap on Smooth (Bandlimited) Targets: When the target is bandlimited (e.g., $L$6 for small $L$7), both NTK and ERM converge at similar (polynomial) rates, with no exponentially increasing gap:

Figure 2: Test error curves for bandlimited targets $L$8 under both NTK and deep network ERM. No exponential gap emerges as $L$9 increases within the tested range.

1. Large Gap and Realizability for Parity-Type Targets: On structured targets that instantiate the dichotomic gap—such as sparse parity functions on the hypercube—the empirical gap between NTK and network ERM is four to six orders of magnitude, and the gap is realized end-to-end by standard SGD-based learning.

Figure 4: Sparse parity $L$0 on $L$1. NTK-KRR errors remain near chance for all tested $L$2, while deep network ERM achieves essentially zero error as $L$3 rises above the polynomial floor.

These results robustly confirm that the exponential gap is not generic, but tightly associated with the spectral-architectural decoupling of the target—validating the theoretical predictions.

Theoretical and Practical Implications

Theoretical: This work establishes that the statistical advantage of deep networks over their NTK limit does not arise merely from dimension or nonlinearity, but has a precisely quantifiable function-space origin. The precise boundary is determined by the interplay of Fourier and architectural complexity. The paper broadens previous kernel-vs-network lower bounds (which focused on low-dimensional subspace dependence) by producing an exponential-in-depth gap at fixed input dimension, and, critically, by using the minimax floor as the comparative baseline.

Practical: For practitioner scenarios involving compositionally structured/hierarchical or parity-like tasks, reliance on NTK or comparable fixed-kernel surrogates can be drastically sub-optimal. The results concretely justify the use of compositional architectural bias and weight-decay learning when prior knowledge suggests the presence of such structure. Additionally, the sharp minimax rates for the architectural class $L$4 provide actionable guidelines for estimation and benchmarking in structured settings, independent of ill-matched smoothness-based complexity metrics.

Limitations and Future Directions

• The main function-space results are currently established for $L$5; multivariate extensions, particularly to $L$6 and beyond, remain to be worked out in detail, though the spectral mechanism generalizes.
• The minimax rate's residual factor of $L$7 reflects an open technical question; achieving a tight depth-linear bound will require advances in Rademacher complexity arguments for the deep variation space.
• Critically, the paper's guarantees are statistical, not algorithmic—hard targets within the architectural class may still elude practical optimization by gradient descent. Future investigation targeting the algorithmic learnability (versus pure statistical learnability) boundary, especially in terms of depth and compositionality, is a compelling direction.

Conclusion

This work presents a rigorous function-space separation between NTK kernel regression and compositional network learning, grounded in an explicit complexity dichotomy. The NTK limit is shown to be exponentially sub-optimal on compositionally rich, spectrally complex targets, relative to the statistical rates achievable by the associated deep network architecture—yet remains minimax-optimal for smooth (bandlimited) targets. Whether kernel surrogates "suffice" is hence determined by comparing the target's Fourier and architectural complexities, rather than by any general dichotomy between kernels and neural networks.

(2607.06382)