Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime (2509.24882v1)

Abstract: Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with matrix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish a precise link between these regimes and the spectral properties of the trained network weights, which we characterize in detail. As a consequence, we provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance, yielding an interpretation from first principles.

• The paper presents a rigorous mapping from neural network training to sparse estimation, deriving universal phase diagrams for excess risk and generalization.
• It characterizes the spectral properties of learned weights, revealing bulk and spike structures that correlate with various overfitting regimes.
• The study leverages approximate message passing and state evolution to predict finite-dimensional behaviors, offering practical insights for model design and optimal pruning.

Scaling Laws and Spectral Properties in Shallow Neural Networks with Feature Learning

This paper presents a rigorous theoretical and empirical analysis of scaling laws and spectral properties in shallow neural networks operating in the feature learning regime. By leveraging exact mappings to sparse vector and low-rank matrix estimation (LASSO and matrix compressed sensing), the authors derive universal phase diagrams for excess risk, characterize the spectra of learned weights, and provide a first-principles explanation for the connection between spectral statistics and generalization performance.

Problem Formulation and Model Equivalences

The central object of paper is the empirical risk minimization (ERM) for two-layer neural networks:

$$\min_{W, a} \sum_{\mu=1}^n \left(y_\mu - f(x_\mu; W, a)\right)^2 + \lambda \left(\|W\|_{\rm F}^2 + \|a\|_2^2\right)$$

where $f$ is either a diagonal linear network or a quadratic network. The teacher-student setup is adopted, with targets generated by a network of the same architecture and additive Gaussian label noise.

• Diagonal Linear Networks: The ERM reduces exactly to LASSO regression, with $\ell_2$ weight decay on both layers mapping to an $\ell_1$ penalty on the effective weights.
• Quadratic Networks: The ERM maps to matrix compressed sensing with nuclear norm regularization, enabling analysis via low-rank matrix estimation.

These equivalences allow the application of high-dimensional inference tools, notably approximate message passing (AMP) and state evolution (SE), to derive precise predictions for generalization error and spectral properties.

Universal Scaling Laws and Phase Diagram

A comprehensive phase diagram is established for the excess risk as a function of sample complexity and regularization strength, under power-law/quasi-sparse targets (coefficients decay as $i^{-2\gamma}$ or $i^{-\gamma}$ for diagonal and quadratic cases, respectively).

• Unified Sample Complexity: The effective sample size is $n$ for diagonal networks and $n/d$ for quadratic networks, allowing a unified presentation of scaling laws.
• Excess Risk Rates: Multiple regimes are identified, including benign and harmful overfitting, interpolation peaks, and fast/slow decay regions. The rates match classical minimax results for high-dimensional regression over $\ell_q$-balls, with optimal regularization achieving Bayes-optimal rates.

(Figure 1)

Figure 1: Excess risk phase diagram as a function of effective sample size and regularization, with corresponding spectral properties of learned weights.

Key findings include:

• Benign Overfitting: For $1 \ll n_{\rm eff} \ll d^{2\gamma}$, excess risk decays as $\Theta(n_{\rm eff}^{-1+1/(2\gamma)})$.
• Harmful Overfitting: As $n_{\rm eff} \to d$, excess risk is dominated by noise fitting, with a non-universal scale.
• Double Descent: Non-monotonic risk behavior at interpolation, with a peak scaling as $\lambda^{-2/3}$.
• Optimal Regularization: Tuning $\lambda$ avoids harmful overfitting and achieves minimax rates.

Spectral Analysis of Learned Weights

The spectral properties of the trained weights are characterized analytically across all regimes. The learned spectrum is shown to be a noisy, soft-thresholded version of the target spectrum, with the following structure:

• Bulk and Spikes: The spectrum consists of a bulk (learned noise), spikes (learned features), and a spike at zero (unlearned features due to regularization).
• Phase Transitions: The emergence and disappearance of spikes correspond to transitions between learning and underfitting regimes.

  Figure 2: Comparison between spectra from simulations and theory across different training phases. Blue: eigenvalue histograms after training. Purple/orange: theoretical predictions for bulk and spikes.

This spectral behavior mirrors empirical observations in deep learning, such as heavy-tailed eigenvalue distributions and their correlation with generalization [martin2021implicit, thamm2024random].

Universal Error Decomposition

A universal decomposition of the excess risk is derived, partitioning it into underfitting, overfitting, and approximation error terms, each directly linked to spectral statistics:

• Underfitting: Power of unlearned spikes (features lost due to regularization or insufficient data).
• Overfitting: Second moment of the bulk (learned noise).
• Approximation Error: Error in learned spikes, governed by signal-to-noise ratio and regularization.

This decomposition provides a principled explanation for the spectra–generalization connection, grounding empirical findings in rigorous theory.

Non-Asymptotic State Evolution and Robustness

The analysis relies on AMP and SE, which are formally valid in the proportional asymptotic regime. However, extensive numerical experiments demonstrate that SE predictions remain accurate down to finite dimensions and arbitrary scalings, far beyond proven guarantees.

Figure 3: Excess risk in simulations (dots) versus non-asymptotic state evolution (solid lines) as a function of effective sample size, showing excellent agreement across regimes.

This robustness suggests that AMP-based frameworks may have predictive power well outside their standard asymptotic assumptions, motivating further work on non-asymptotic control.

Practical Implications and Extensions

• Pruning Optimality: Pruning small eigenvalues (setting them to zero) achieves optimal error rates except near the interpolation peak, aligning with practical weight pruning strategies in neural network compression.
• Universality: The phase diagram and error decomposition are universal across diagonal and quadratic networks, and potentially extendable to broader architectures.
• Resource Allocation: The results inform resource-conscious model design, highlighting the necessity of scaling data, model size, and regularization jointly to avoid bottlenecks.

Conclusion

This work provides a comprehensive theoretical and empirical framework for understanding scaling laws and spectral properties in shallow neural networks with feature learning. By mapping neural training to sparse estimation problems, the authors derive universal phase diagrams, analytic spectral characterizations, and a principled error decomposition. These results bridge empirical observations and theory, offering actionable insights for model design and motivating future research on universality, non-Gaussian inputs, and compute scaling laws.