High-Dimensional SGD Dynamics

• High-dimensional SGD Dynamics is the study of how stochastic gradient descent navigates overparameterized models by exploiting the Hessian's spectral structure.
• It reveals that a few dominant outlier eigenvalues drive optimization through low-dimensional updates while the numerous near-zero bulk directions offer implicit stability.
• This insight informs practical strategies for tuning learning rates, batch sizes, and regularization to achieve stable training and superior generalization.

High-dimensional stochastic gradient descent (SGD) dynamics refers to the behavior and underlying principles governing the evolution of SGD when applied to highly overparameterized models—especially neural networks and kernel machines—whose parameter counts and/or data dimensions are very large. In this regime, SGD interacts with rich spectral properties of the loss landscape’s curvature (as encoded by the Hessian), exhibits low-rank structure in its updates, and experiences phenomena such as bulk-and-outlier spectral decompositions, emergent low-dimensional subspaces for optimization, and characteristic scaling behaviors with respect to sample size and model parameters.

1. Spectral Structure of the Hessian and Implications for SGD

The Hessian of the loss function in deep and wide neural networks possesses a distinctive “spiked” structure: a small number of large, isolated “outlier” eigenvalues (one per class in classification, in typical cases) are separated from a bulk of near-zero eigenvalues. The outliers arise entirely from the Gauss-Newton (expected Fisher) term, specifically the low-rank $A_1$ term in the Hessian’s decomposition; the continuous bulk is contributed mainly by the architecture-dependent $H$ term. Empirical findings show that as the number of data points increases, the bulk eigenvalues compress toward zero, yet outliers remain distinct and are relatively insensitive to sample size.

This spectral picture critically shapes SGD’s behavior:

• SGD dynamics are low-rank: The trajectory of SGD aligns rapidly with the subspace spanned by the leading outlier eigenvectors. Most parameter-space directions are flat (bulk), and movement in those directions does not strongly affect the loss.
• Learning rates must accommodate curvature: Large eigenvalues (outliers) force step sizes to be reduced for stability, while in flat directions, SGD can take large steps with little effect.
• Generalization and sharpness: Flat landscapes (narrow bulk) are statistically linked to better generalization, whereas persistence of sharp, isolated eigenvalues may be indicative of sensitivity or memorization.

2. Decomposition of the Hessian: Gauss-Newton and Beyond

The Hessian, $Hess(\theta)$, naturally decomposes into an $H$ component related to neural network architecture and a Gauss-Newton component $G$ connected to gradient covariance and the model’s Fisher information. The Gauss-Newton term itself further splits: $G = A_1 + A_2 + B_1 + B_2$ where $A_1$ and $A_2$ capture low-rank inter-class and intra-class gradient correlations, and $B_1$, $B_2$ represent bulk contributions from gradient variances.

In practice:

• $A_1$ (low-rank spike): Responsible for the emergence of outlier eigenvalues that direct most optimization progress.
• $B_1$/$B_2$ (bulk): Feed the dense cluster of small eigenvalues, underlying the many flat directions observed.

This layered structure reinforces why SGD can navigate overparameterized settings: it ignores vast degenerate directions, focusing computational effort on a data-derived, low-dimensional space.

3. Evolution of the Spectrum with Training and Data

As training progresses, the separation and magnitudes of outliers and the bulk in the Hessian spectrum evolve:

• Early epochs: Outlier eigenvalues may grow and peak as discriminative structure is learned and class separation is established.
• Later stages: Spectrum may compress as the optimization converges, with bulk mass condensing near zero and outlier magnitude reducing.
• With increasing sample size: The spectral bulk shrinks and sharpens, but outlier locations are largely unaffected, indicating that the directions associated with outlier eigenvalues are robust to the addition of more data, while bulk directions become increasingly “uninformative.”

For SGD:

• Robustness from flatness: The enormous number of near-zero bulk directions yields implicit stability, as perturbations in those directions alter the loss function minimally.
• Risk of instability: Negative eigenvalues in the Hessian, corresponding to saddles or unconverged non-convex directions, may persist during training and threaten instability if learning rates are not annealed.

4. SGD’s Low-Rank Trajectory and Generalization

SGD naturally aligns updates with high-curvature directions—the outlier eigenspace. This “active” subspace is typically very low-dimensional (number of classes), even in networks with millions of parameters. As a result:

• Most SGD updates live within the subspace spanned by outlier eigenvectors.
• SGD’s statistical and optimization properties (such as convergence rates and the probability of escaping sharp minima or saddles) are determined by the spectrum’s outlier-bulk structure.
• Flatter solutions—where the spectral norm is reduced for both outlier and bulk—are empirically correlated with superior generalization.

These phenomena underpin the effectiveness of batch size selection, regularization methods, and learning rate tuning: all leverage (either intentionally or implicitly) the structure revealed by the Hessian spectrum.

5. Practical Applications: Model Selection and Optimization

Understanding high-dimensional SGD dynamics has led to concrete guidance in deep learning practice:

• Batch size and learning rate schedule: Large learning rates must be constrained by the largest outlier eigenvalue to avoid divergence; small batch sizes, by injecting gradient noise, can delay overfitting and act as implicit early stopping.
• Regularization strategies: Techniques such as weight decay or label smoothing may be interpreted as attempts to compress the Hessian spectrum—reducing outlier magnitudes, thereby achieving sharper or flatter minima.
• Network design: Choice of architecture (e.g., width, depth, activation functions) directly affects the $H$ component of the Hessian, which in turn shapes the spectrum’s bulk.

A summary comparison:

Aspect	Role of Hessian Spectrum	SGD Implication
Outliers (spikes)	Large, isolated eigenvalues (inter-class structure)	Control learning rate, stability, and information content
Bulk (continuous)	Dense near-zero (flat) eigenvalues (overparametrization)	Robustness, flexibility, and high-dimensional traversal
Sample size	Bulk compresses; outliers stable	More data reduces degeneracy in non-informative directions
Training epoch	Outlier growth/decay, bulk migration	Dynamics adapt between informative and flat subspaces

6. Concluding Insights

The spiked spectrum of the Hessian in deepnets is a structural law of high-dimensional learning, shaping the path and possibilities for stochastic gradient descent. SGD exploits this structure by confining optimization efforts to a small, data-relevant subspace, effectively yielding rapid convergence, stable learning, and—when combined with control of outlier sharpness—good generalization. The interplay between bulk flatness (robustness) and low-rank outlier structure (statistical efficiency) distinguishes the high-dimensional setting, guiding both theoretical investigations and practical algorithm design.