Heavy-Tailed Self-Regularization

• Heavy-Tailed Self-Regularization (HT-SR) Theory is a framework that characterizes how DNN weight matrices develop heavy-tailed eigenvalue distributions, reflecting multi-scale feature learning.
• It combines random matrix theory, statistical mechanics, and spectral diagnostics to link these spectral signatures to generalization, model selection, and algorithmic regularization.
• Practical applications include using metrics like AlphaHat for adaptive learning rate scheduling, weight decay adjustment, and layer-wise pruning to enhance model performance.

Heavy-Tailed Self-Regularization (HT-SR) Theory is a framework for characterizing and understanding the emergent spectral properties of deep neural network weight matrices after training. Modern DNNs implicitly sculpt the eigenvalue spectrum of their layer weight correlations into heavy-tailed forms, reflecting multi-scale correlation and feature learning. HT-SR theory employs random matrix theory (RMT), statistical mechanics, and spectral diagnostics to connect these spectral signatures to generalization, model selection, and algorithmic regularization.

1. Spectral Foundations and Universality

HT-SR theory studies the empirical spectral density (ESD) of layer-wise weight correlation matrices $X_\ell = (1/N_\ell) W_\ell^\top W_\ell$. In random matrix theory, the ESD of i.i.d. Gaussian matrices follows the Marchenko–Pastur (MP) law, presenting a bulk with compact support and no outlier eigenvalues. Trained DNN weight matrices, however, display ESDs that deviate from MP: the right tail decays slowly according to a power law, $\rho(\lambda) \sim \lambda^{-\alpha}$ over $x_\text{min} \leq \lambda \leq x_\text{max}$, and $\alpha$ quantifies the degree of "heavy-tailedness" (Martin et al., 2018, Martin et al., 2019, Martin et al., 2021).

Three heavy-tailed regimes are predicted from RMT:

• Weakly heavy-tailed ($\mu > 4$): MP-like bulk, finite support.
• Moderately heavy-tailed ($2 < \mu < 4$): ESD tail follows $-\left(1+\frac{\mu}{2}\right)$, Fréchet largest-eigenvalue statistics.
• Very heavy-tailed ($0 < \mu < 2$): Pure power law, outlier domination.

This empirical universality—the prevalence of heavy-tailed ESDs and similar tail exponents across architectures and training regimes—is termed Heavy-Tailed Mechanistic Universality (HT-MU) (Martin et al., 2019).

2. AlphaHat Metric and Shape/Scale Decomposition

A central object in HT-SR theory is the AlphaHat metric, which unifies two complementary spectral diagnostics:

• Shape ($\hat{\alpha}$): The layer-wise average power-law exponent, $(1/L) \sum_{l=1}^L \alpha_\ell$.
• Scale ($\hat{\sigma}$): The average layer log-spectral norm, $(1/L) \sum_{l=1}^L \log \lambda_\ell^\text{max}$.

AlphaHat combines these via a weighted sum, $$\hat{\alpha}^{(H)} = \sum_{\ell=1}^L \alpha_\ell \cdot \log \lambda_\ell^\text{max}$$, equivalently $L \cdot \langle \alpha \sigma \rangle$. Estimation employs maximum-likelihood fits (Clauset–Shalizi–Newman, Hill estimator) over the right tail, with tail index selection via Kolmogorov–Smirnov minimization (Martin et al., 2021, Martin et al., 2019).

Neither shape nor scale alone reliably predicts generalization: shape captures multi-scale feature learning, scale tracks capacity control across model architecture. AlphaHat resolves their blind-spots, providing strong monotonic alignment with out-of-sample accuracy across both architecture and hyperparameter variation.

3. Mechanism: Implicit Self-Regularization

Training dynamics in DNNs (SGD, GD, Adam) drive the weight spectra through several phases (Martin et al., 2018, Martin et al., 2019):

1. Random-like: MP bulk; noise-dominated weights.
2. Bleeding-out: Mass leaks just above bulk edge.
3. Bulk+Spikes: Few outlier eigenvalues appear; signal emerges.
4. Bulk-decay: Bulk deforms, more continuous right tail.
5. Heavy-Tailed: Pure power law; scale-free correlations.
6. Rank-collapse: Over-regularized regime; many zero eigenvalues.

Multi-scale correlations arise naturally, with larger batch sizes or aggressive regularization freezing the spectrum in early phases (weaker generalization), while smaller batches or tuned regularization induce deeper heavy-tails. Training induces self-organized criticality, where the system naturally tunes its spectral structure for optimal feature learning and generalization (Martin et al., 2018, Martin et al., 2019).

HT-SR theory extends to noise-free regimes, showing that large deterministic updates (e.g., full-batch Adam with high learning rates) can induce a sequence of rank-one perturbations whose repeated rotation and aggregation drive bulk+spike spectra into a heavy-tailed regime without stochastic noise (Kothapalli et al., 2024).

4. Generalization Bounds and Capacity Metrics

HT-SR theory demonstrates a quantitative link between spectral heavy-tailedness and generalization bounds. For SGD modeled as a Feller process under heavy-tailed additive noise, the sample-path Hausdorff dimension (controlled by tail index $\alpha$) governs the intrinsic capacity of the learning trajectory (Şimşekli et al., 2020, Lim et al., 2022):

$$\sup_{t \in [0,1]} \left| \widehat{R}(W_t, S) - R(W_t) \right| \leq B \sqrt{\frac{2 \bar{d} \log(nL^2) + \log(1/\gamma)}{n}}$$

where $\bar{d} \leq \alpha$ is the upper Blumenthal–Getoor index, and $B$ bounds the loss. Heavier tails ($\alpha$ small) produce lower Hausdorff dimension, smaller covering numbers, and thus enhanced generalization capacity. Notably, the tail index is agnostic to raw parameter count and thus avoids the curse of dimensionality intrinsic to VC or norm-based bounds.

Further, deterministic chaotic gradient perturbations (MPGD) converge to heavy-tailed Lévy-driven SDEs, where dynamical regularization induces effective Hessian penalties on loss, smoothing sharp minima, and promoting generalization through flatness (Lim et al., 2022).

5. Algorithmic Design: Adaptive Regularization via HT-SR

HT-SR metrics have been operationalized to improve model selection, regularization, and compression:

• Layer-wise LR scheduling: TempBalance adaptively tunes per-layer learning rates via the measured HT-SR exponent, steering layers to the optimal regularization regime ($\alpha \approx 2$), outperforming global or spectral norm schemes (Zhou et al., 2023).
• Module-wise weight decay: AlphaDecay assigns weaker decay to modules with heavier-tailed spectra, balancing structural diversity and spectral learning between modules, improving perplexity and generalization in LLM pre-training (He et al., 17 Jun 2025).
• Layer-wise pruning: AlphaPruning employs block-level shape metrics (Hill PL fit) to allocate layer-wise sparsity ratios, prunes less aggressively in layers with strong heavy-tails, and attains higher sparsity with minimal degradation in accuracy across LLM and vision models (Lu et al., 2024).

In each case, the empirical ESD is computed per module or layer, the tail index $\alpha$ estimated, and the regularization assigned via a monotonic mapping (linear or otherwise) between $\alpha$ and the regularization strength.

6. Empirical Validation and Simpson's Paradox

Across diverse architectures (VGG, ResNet, DenseNet, ViT, LLaMA) and optimization regimes, HT-SR metrics robustly correlate with downstream accuracy, generalization gaps, and transfer performance (Martin et al., 2019, Martin et al., 2021, Martin et al., 2019, Lu et al., 2024, He et al., 17 Jun 2025). Quantitative results:

Model Family	Metric	Correlation R
VGG/ResNet	AlphaHat	–0.99
DenseNet	AlphaHat	–0.97
LLaMA-7B	Perplexity vs PL	see (Lu et al., 2024)

In cross-sectional studies, classical norm-based metrics or capacity measures exhibit Simpson's paradox: trends that hold within fixed-depth subgroups can reverse when across architectures. AlphaHat, by blending implicit scale and shape, resolves these confounds and provides reliable, monotonic predictive power (Martin et al., 2021).

7. Theoretical and Practical Implications

HT-SR theory (i) reframes generalization as an emergent spectral self-regularization phenomenon; (ii) delivers metrics for post-hoc, data-free model selection; (iii) offers mechanistically grounded, empirically validated recipes for automatic regularization scheduling and model compression; and (iv) connects statistical-physics perspectives with classical learning theory (Martin et al., 2018, Martin et al., 2019, Martin et al., 2019, Şimşekli et al., 2020, Martin et al., 2021).

Open directions include tighter chaining-based generalization bounds, extension to adaptive optimizers and attention architectures, unsupervised early-stopping via spectral phase monitoring, and robust integration of activation statistics with spectral shape metrics. The HT-SR paradigm provides a physics-inspired lens for deep learning, unifying model spectral analysis, regularization, and generalization.