Statistical Properties of Training & Generalization

Abstract: Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.

• The paper reveals how overparameterized deep networks exhibit double descent and benign overfitting, challenging classical tradeoff theory.
• The paper uses rigorous empirical and theoretical analyses, employing SGD dynamics and kernel regimes to uncover universal scaling laws and phase transitions.
• The paper highlights the importance of architectural choices and hyperparameter transfer in achieving reliable generalization under practical constraints.

Statistical Properties of Training and Generalization in Deep Learning

Introduction

The statistical properties governing the training and generalization of deep neural networks (DNNs) diverge in critical ways from classical statistical learning frameworks, largely due to high dimensionality, the non-convexity of neural architectures, and regime shifts driven by overparameterization. This article provides a structured analysis—emphasizing physical analogies—of universal phenomena in modern deep learning, such as double descent, benign overfitting, the role of inductive biases, neural scaling laws, and the statistical implications of key design and hyperparameter choices. Particular attention is given to the practical and theoretical disconnects between idealized scaling regimes and real-world, constraint-laden applications in scientific domains, with a focus on physics-informed AI.

Universal Phenomena in Overparameterized Regimes

Non-Convexity, Loss Landscape Geometry, and SGD

Unlike traditional convex models, DNNs present non-convex loss surfaces with myriad critical points and intricate geometries shaped by overparameterization. Empirical and theoretical results suggest that, far from being detrimental, high-dimensional non-convexity often produces primarily saddle points rather than poor local minima. In the highly overparameterized regime, minima frequently reside on large, connected subspaces, making it feasible for optimizers like stochastic gradient descent (SGD) to find generalizing solutions. Importantly, SGD's inherent stochasticity and the use of momentum facilitate effective escape from saddle points, further reducing the practical complexity of optimization [dauphin_identifying_2014, choromanska_loss_2015, draxler_essentially_2018].

Double Descent, Overparameterization, and Benign Overfitting

Classical bias-variance tradeoff theory predicts a U-shaped curve for test error with increasing model complexity: underfitting (high bias) for simple models, a "sweet spot" of optimal capacity, then a sharp rise in error due to overfitting (high variance). Modern DNNs, however, empirically exhibit a double descent phenomenon—test error first increases at the interpolation threshold (where the number of parameters matches the number of training samples) but then decreases again as model capacity further increases, often surpassing the classical optimum (Figure 1). This regime permits interpolating (zero training loss) solutions that retain excellent generalization, contradicting classical parametric intuition.

Figure 1: Bias-variance trade-off and double descent in test error for increasing model complexity; the second descent emerges when models are highly over-parameterized.

This benign overfitting effect is best understood in linear settings, such as minimum-norm interpolators in linear regression, but extends to neural networks in both lazy NTK (Neural Tangent Kernel) and rich feature learning regimes. SGD implicitly biases solutions toward simpler, lower-norm functions, closely mirroring classical forms of regularization [belkin_reconciling_2019, bartlett_benign_2020, hastie_surprises_2020].

Figure 2: Conceptual taxonomy of overfitting—benign, tempered, and catastrophic—illustrating that overparameterized interpolators can yield either robust or fragile generalization depending on inductive bias and noise alignment.

Inductive Bias and Kernel Regimes

The critical role of inductive bias, both explicit (architectural choices, equivariances) and implicit (optimizer, initialization), is highlighted through exact analyses in the kernel and infinite-width limits. In these lazy regimes, the generalization is fully characterized by the eigenspectrum of the kernel induced by the architecture and initialization, and generalization is limited by spectral alignment between the kernel and target function. Feature learning in the rich regime, in contrast, enables dynamic adaptation toward optimal task-specific representations and can produce qualitatively enhanced learning curves [canatar_spectral_2021, cohen_learning_2021, seroussi_separation_2023].

Neural Scaling Laws and Their Origins

Empirical scaling laws indicate that test loss decreases predictably as a power-law function of model size, dataset size, and compute budget, even across several orders of magnitude. The observed exponents are not fixed universal quantities, but depend on statistical properties of the data manifold, the architecture, and the effectiveness of feature learning. Theoretical analyses suggest that the scaling exponents reflect the intrinsic data structure, supporting a physically-inspired hypothesis that natural data possessing power-law correlations in their spectrum permit steady performance gains via continued scaling [kaplan_scaling_2020, hoffmann_training_2022, bahri_explaining_2021, cagnettaDerivingNeuralScaling2026].

Scaling laws also inform optimal allocation of compute: given budgetary constraints, model size and dataset size should both be increased in lockstep along the compute-optimal frontier, contradicting earlier practice of fixing dataset size and only enlarging model capacity.

Dynamics of Generalization: Solvable Regimes and Phase Transitions

Statistical mechanics approaches—such as the teacher-student framework—permit precise derivation of learning curves, phase transitions, and sharp thresholds in high-dimensional limits. These analyses identify critical sample-to-parameter ratios demarcating transitions between learnable, overfitting, and underdetermined regimes, and link these transitions to phenomena such as the jamming threshold in constraint satisfaction and generalization in deep nonlinear networks [barbier_optimal_2019, geiger_jamming_2019, spigler2020asymptotic]. These models provide concrete predictors for the effects of architectural choices and data covariance on sample complexity and achievable risk.

Hyperparameters, Architectures, and Scaling Strategies

Parameterization and Hyperparameter Transfer

Scaling models efficiently requires principled parameterizations that preserve learning dynamics and enable hyperparameter transfer from small (affordable) proxies to large (expensive) models. Maximal update parameterization (μP) and its generalizations guarantee that learning rates, initialization variances, and weight decay can be safely translated as models grow—provided the scaling rules for width and depth are followed [yang_tensor_2021, bordelon_depthwise_2023, dey_dont_2025]. Empirically, these parameterizations yield nearly universal learning dynamics and effective transfer across scales up to billion-parameter LLMs, although there are nontrivial limitations regarding transfer outside of training set distributions and when models are too small or far from the asymptotic regime.

Depth and Emergence of Hierarchical Representations

Although width enables kernel-like behavior and smoothing of loss landscapes, depth is directly linked to hierarchical feature extraction. In certain limits, increasing depth does not inherently increase expressivity in terms of function space covered (e.g., in infinite-width MLPs with ReLU activation [bietti_deep_2021]); however, for compositional or hierarchical data-generating processes—such as those common in vision and physics—depth can confer provable representational or sample-efficiency advantages. Theoretical models of data with hierarchical latent structure demonstrate that truly deep architectures are sometimes necessary for efficient learning and generalization [cagnetta2023deep].

Optimization Algorithms and Training Dynamics

Choice of Optimizer and Learning Rate

Optimization in deep learning is nuanced: well-chosen learning rates and adaptive methods (Adam, RMSProp, Shampoo, Muon) implicitly shape the geometry of optimization and interact with loss landscape curvature to govern both convergence and generalization [cohenUnderstandingOptimizationDeep2024, gupta_shampoo_2018, jordan_muon_2024]. Empirical and theoretical analyses reveal that best practice is to operate near the edge of stability, maximizing the learning rate until reaching instability thresholds controlled by the largest Hessian eigenvalues [cohen_edge_2021, arora_edge_2022]. Batch size, noise scale, and schedule also interact in subtle ways with implicit regularization mechanisms and the preference for flat minima [hochreiter_flat_1997, keskar_largebatch_2016].

Emergent Phenomena: Grokking and Late-Time Transients

Extended training in highly expressive models can yield grokking: a regime where zero training loss and poor generalization persist for long epochs, followed by abrupt improvement in test accuracy. Analytical results now demonstrate that such long transients and phase transitions in linear estimators arise from the interaction between regularization, problem geometry, and implicit dynamical bias induced by optimization protein [levi_grokking_linear_estimators_2024, beck_grokking_edge_linear_separability_2024].

Generalization Under Practical Constraints

Data Limited Scenarios

When labeled data are scarce, statistical estimation error dominates. Effective strategies include the introduction of strong inductive biases via architecture (symmetry, equivariance), explicit kernelization, and leveraging unsupervised or weakly supervised pretraining. Kernel and Gaussian process approaches are especially effective for quantifying the number of modes that can be robustly learned and for ensuring generalization in low-$n$ regimes [cohen_learning_2021, canatar_spectral_2021].

Parameter, Compute, and Latency Constraints

Constraints on architecture size, training budget, or deployment serve to inflate approximation, optimization, and estimation errors, respectively. The statistical literature informs efficient model compression (distillation, pruning, quantization) and the deployment of parameter-efficient adaptation methods (LoRA, fine-tuning). Notably, compressibility of overparameterized models provides Occam-style guarantees that empirically link model size and generalization gap [arora_compression_2018, lotfi_pac-bayes_2022].

Physics-Informed Strategies

Physics applications demand both trustworthy generalization and robust uncertainty quantification. Here, explicit encoding of known symmetries, invariances, and conservation laws can dramatically enhance learning efficiency and reduce variance. Furthermore, physics-motivated optimization and architecture co-design can be essential for respecting deployment constraints (e.g., FPGA inference) and for supporting principled validation strategies.

Conclusion

This comprehensive analysis of the statistical properties of training and generalization in deep learning demonstrates that many surprising empirical phenomena—such as double descent, benign overfitting, predictable scaling, and the effectiveness of simple optimization techniques—emerge naturally from the high-dimensional structure, implicit and explicit biases, and the physics-informed framework underpinning modern DNNs. Rather than serving as anomalies, these behaviors signal the presence of universal mechanisms, many of which have precise parallels in statistical physics. Future progress in AI will likely arise from sharpening this connection, unifying rigorous theory, data-driven empirical study, and domain expertise to develop targeted, efficient, and robust learning systems.