Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning (1810.01075v1)

Abstract: Random Matrix Theory (RMT) is applied to analyze weight matrices of Deep Neural Networks (DNNs), including both production quality, pre-trained models such as AlexNet and Inception, and smaller models trained from scratch, such as LeNet5 and a miniature-AlexNet. Empirical and theoretical results clearly indicate that the DNN training process itself implicitly implements a form of Self-Regularization. The empirical spectral density (ESD) of DNN layer matrices displays signatures of traditionally-regularized statistical models, even in the absence of exogenously specifying traditional forms of explicit regularization. Building on relatively recent results in RMT, most notably its extension to Universality classes of Heavy-Tailed matrices, we develop a theory to identify 5+1 Phases of Training, corresponding to increasing amounts of Implicit Self-Regularization. These phases can be observed during the training process as well as in the final learned DNNs. For smaller and/or older DNNs, this Implicit Self-Regularization is like traditional Tikhonov regularization, in that there is a "size scale" separating signal from noise. For state-of-the-art DNNs, however, we identify a novel form of Heavy-Tailed Self-Regularization, similar to the self-organization seen in the statistical physics of disordered systems. This results from correlations arising at all size scales, which arises implicitly due to the training process itself. This implicit Self-Regularization can depend strongly on the many knobs of the training process. By exploiting the generalization gap phenomena, we demonstrate that we can cause a small model to exhibit all 5+1 phases of training simply by changing the batch size. This demonstrates that---all else being equal---DNN optimization with larger batch sizes leads to less-well implicitly-regularized models, and it provides an explanation for the generalization gap phenomena.

• The paper demonstrates that deep neural networks exhibit implicit self-regularization, shown through distinct spectral phases in weight matrices.
• The methodology employs Random Matrix Theory to analyze the empirical spectral density across architectures, highlighting interlayer correlations.
• These findings imply that training hyperparameters, such as batch size, can be adjusted to enhance model generalization without explicit regularization.

Implicit Self-Regularization in Deep Neural Networks: Insights from Random Matrix Theory

Deep Neural Networks (DNNs) have become a cornerstone of modern machine learning, yet their theoretical underpinnings often remain elusive, particularly regarding their generalization capabilities. The paper "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning" by Martin and Mahoney leverages insights from Random Matrix Theory (RMT) to shed light on these phenomena. The authors propose that DNNs implicitly implement a form of Self-Regularization, observed through the analysis of weight matrices, without the need for traditional explicit regularization techniques.

Key Contributions and Results

The authors apply RMT to explore the Empirical Spectral Density (ESD) of the weight matrices of various DNN architectures, including AlexNet, Inception, and LeNet5. A notable finding is the presence of Self-Regularization in DNNs, observable through the spectral properties of these matrices. Importantly, they identify 5+1 Phases of Training based on increasing amounts of implicit Self-Regularization:

1. Random-like: The ESD mimics that of a purely random matrix.
2. Bleeding-out: A moderate signal starts to appear just outside the expected bulk of eigenvalues.
3. Bulk+Spikes: This phase is characterized by an MP bulk with distinct spikes, similar to a Spiked-Covariance model.
4. Bulk-decay: The onset of Heavy-Tailed behavior disrupts clear separation between bulk and spikes, leading to a decay into the ESD.
5. Heavy-Tailed: The ESD becomes better described by Heavy-Tailed distributions, indicative of strong signal correlations at all scales.

The additional Rank-collapse phase represents over-regularization scenarios where substantial rank loss occurs.

The authors find that for smaller, older models like LeNet5, the ESDs align with traditional MP theory with clear isolated spikes, indicative of weak Self-Regularization akin to Tikhonov regularization. However, modern architectures exhibit Heavy-Tailed behavior in their weight matrices, suggesting a novel form of Self-Regularization that arises from interlayer correlations amplified by the training process itself.

Theoretical Implications

This research provides compelling evidence that DNNs engage in Self-Regularization implicitly, offering a nuanced understanding that challenges conventional wisdom relying solely on model complexity or overparameterization. By framing this regularization within the context of RMT and statistical physics, the paper links DNN training dynamics to systems with similar complex behavior, like disordered materials.

The use of RMT to elucidate these principles is notable because it operates in a regime where traditional large-data assumptions (e.g., VC dimension) are invalid. The focus on the distributional properties of weight matrices, rather than individual elements, supports a framework in which the learning machine restrictively carves out a functional subset efficiently representing the data.

Practical Implications

Practically, this work suggests methodologies for evaluating and potentially enhancing model generalization without explicit regularization. For instance, understanding that smaller batch sizes can implicitly increase Self-Regularization gives rise to strategies aimed at optimizing hyperparameters for improved generalization. Additionally, the consideration of Unsregularized ESDs provides insights into overfitting prevention, akin to traditional concepts such as early stopping and cross-validation but grounded in spectral properties.

Directions for Future Research

The implications of this paper suggest several future directions. Foremost is the exploration of how these spectral characteristics relate to adversarial robustness and transfer learning. Additionally, expanding this framework to convolutional and attention-based neural architectures, as well as different data modalities like text or time-series data, may reveal further unified principles.

In conclusion, Martin and Mahoney's work foundationally influences our understanding of DNNs by redirecting the focus from traditional empirical risk minimization towards the inherently self-regularizing nature of these models, an insight that holds promise for designing more robust and efficient neural architectures.