Loop Corrections to the Training and Generalization Errors of Random Feature Models

Abstract: We investigate random feature models in which neural networks sampled from a prescribed initialization ensemble are frozen and used as random features, with only the readout weights optimized. Adopting a statistical-physics viewpoint, we study the training, test, and generalization errors beyond the mean-kernel approximation. Since the predictor is a nonlinear functional of the induced random kernel, the ensemble-averaged errors depend not only on the mean kernel but also on higher-order fluctuation statistics. Within an effective field-theoretic framework, these finite-width contributions naturally appear as loop corrections. We derive the loop corrections to the training, test, and generalization errors, obtain their scaling laws, and support the theory with experimental verification.

• The paper introduces an effective field theory framework to derive loop corrections for training, test, and generalization errors in random feature models.
• It employs spectral analysis and Monte Carlo validation to reveal that finite-width corrections scale as O(n⁻¹), bridging the gap between mean-kernel predictions and empirical data.
• The approach refines model selection and diagnostic methods in overparameterized kernel models by incorporating nonlinear kernel fluctuations and their impact on generalization.

Loop Corrections and Finite-Width Effects in Random Feature Models

Introduction and Motivation

The paper "Loop Corrections to the Training and Generalization Errors of Random Feature Models" (2604.12827) develops a formal perturbative framework grounded in effective field theory for analyzing random feature models (RFMs), specifically addressing finite-width effects. In the RFM setting, an ensemble of neural networks, sampled and frozen at initialization, defines nonlinear feature maps, with only the downstream readout optimized. Kernels induced by these random features are foundational, and their limiting behavior in infinite-width regimes connects RFMs to kernel ridge regression and neural tangent kernel (NTK) theory. However, empirical architectures operate at finite width, where kernel fluctuations—departures from the mean kernel—can drive substantial deviations from tree-level kernel predictions, especially in deep or moderately wide settings. These fluctuations induce higher-order corrections, known as loop corrections in field-theoretic terminology, which the paper systematically derives for the ensemble-averaged training error, test error, and generalization gap.

Effective Field-Theoretic Formulation

The field-theoretic perspective addresses the nonlinear dependency of observables on the random kernel. Since the prediction depends on the resolvent $(K_\Theta + N\lambda I_N)^{-1}$, kernel fluctuations—timed by their connected correlation functions—must be incorporated beyond simple replacement by the mean kernel. The framework identifies the Gaussian theory (mean-kernel approximation) as the tree-level contribution, while loop corrections, introduced as higher-order cumulants and correlation tensors, systematically encode finite-width deviations. The expansion organizes perturbative terms in powers of the inverse feature width $n$, with the leading correction scaling as $O(n^{-1})$ and governed by the connected four-point correlator (vertex $V$).

Analytical Derivation of Error Expansions

The paper rigorously details the expansions for three observables:

• Training Error: The ensemble-averaged training error admits a perturbative series where the leading term relies on the mean kernel, and the next-to-leading-order term (the one-loop correction) is a contraction of the kernel covariance vertex $V$. Beyond the Gaussian theory, finite-width corrections emerge as systematic loop insertions.
• Test Error: The test error depends additionally on mixed fluctuations between train and test kernels, inducing new contraction structures involving $V$ and integrals over the population distribution. This richer structure encapsulates the complexity of generalization in finite-width RFMs.
• Generalization Gap: The generalization gap is defined as the difference between the test and training errors, and captures both pure train kernel fluctuations and mixed train--test corrections. As a result, its loop expansion incorporates both intra-domain and cross-domain cumulants.

These expansions are derived with explicit spectral decompositions, elucidating how eigenmodes of the mean kernel amplify or suppress correction terms. The analytical results identify concrete scaling laws: loop corrections scale as $n^{-1}$, and are dominated by the spectral response of the mean kernel and the contraction structure of $V$.

Spectral Analysis and Scaling Laws

A detailed spectral analysis provides modal insight into the perturbative regime. Tree-level terms are diagonalized via the kernel eigenbasis, highlighting amplification effects when regularization is small or kernel eigenvalues are near zero. One-loop corrections are explicitly shown to involve contractions of $V$ against spectral propagators—these terms become dominant as width decreases and characterize the breakdown of the mean-kernel approximation. The generalization gap, in particular, is shown to be more sensitive to spectral structure and mixed kernel fluctuations than training error alone.

Experimental Validation

Empirical validation is conducted via Monte Carlo estimation on synthetic one-dimensional regression tasks, with ensemble averaging across random feature realizations. The paper presents:

Figure 1: Training error versus feature width $n$, with empirical averages converging to mean-kernel predictions and one-loop corrections matching finite-width deviations.

Figure 2: Test error as a function of feature width $n$0, demonstrating improved agreement with empirical results upon inclusion of one-loop corrections.

Figure 3: Generalization gap versus feature width $n$1, with one-loop predictions closely tracking the empirical generalization gap, revealing rich finite-width structure.

Figure 4: Log–log plot of deviations from tree-level prediction, fitted slope supports $n$2 scaling of finite-width loop corrections.

These figures confirm both the dominance of tree-level (mean-kernel) behavior at large $n$3 and the necessity of incorporating loop corrections for accurate modeling at moderate widths. The $n$4 scaling predicted analytically is empirically validated, and deviations in generalization gap are explained by the richer mixed fluctuation structure identified in the theoretical expansion.

Implications and Future Directions

The results have immediate implications for understanding finite-width neural networks, particularly in quantifying sources of generalization error beyond classical kernel theory. Practically, the loop expansion provides refinements for model selection and diagnostics in kernel-based learning, especially when operating away from the infinite-width limit. Theoretically, the framework offers groundwork for extending perturbative analysis to deep and fully trainable networks, where dynamics of parameter evolution may be mapped into effective-theory expansions with action-based loop corrections. Future research may focus on non-frozen, dynamically trained architectures and the mapping of learning trajectories into field-theoretic perturbative structures.

Conclusion

This study establishes a systematic effective-theoretic framework for quantifying finite-width corrections in random feature models, deriving explicit loop expansions for training, test, and generalization errors and pinpointing scaling laws via spectral analysis. Empirical validation corroborates theoretical predictions, confirming both the accuracy of tree-level theory at large widths and the necessity of loop corrections for intermediate regimes. The approach enriches the understanding of kernel fluctuations and generalization phenomena in overparameterized models, setting a rigorous foundation for future field-theoretic investigations in learning theory.