Random matrix theory and the loss surfaces of neural networks (2306.02108v1)

Abstract: Neural network models are one of the most successful approaches to machine learning, enjoying an enormous amount of development and research over recent years and finding concrete real-world applications in almost any conceivable area of science, engineering and modern life in general. The theoretical understanding of neural networks trails significantly behind their practical success and the engineering heuristics that have grown up around them. Random matrix theory provides a rich framework of tools with which aspects of neural network phenomenology can be explored theoretically. In this thesis, we establish significant extensions of prior work using random matrix theory to understand and describe the loss surfaces of large neural networks, particularly generalising to different architectures. Informed by the historical applications of random matrix theory in physics and elsewhere, we establish the presence of local random matrix universality in real neural networks and then utilise this as a modeling assumption to derive powerful and novel results about the Hessians of neural network loss surfaces and their spectra. In addition to these major contributions, we make use of random matrix models for neural network loss surfaces to shed light on modern neural network training approaches and even to derive a novel and effective variant of a popular optimisation algorithm. Overall, this thesis provides important contributions to cement the place of random matrix theory in the theoretical study of modern neural networks, reveals some of the limits of existing approaches and begins the study of an entirely new role for random matrix theory in the theory of deep learning with important experimental discoveries and novel theoretical results based on local random matrix universality.

• The paper extends prior RMT work by generalizing the analysis of loss surfaces across various neural network architectures.
• The paper identifies local random matrix universality, demonstrating that statistical properties of random matrices model neural loss landscapes.
• The paper analyzes Hessian spectra to derive insights that lead to enhanced training optimization methods for neural networks.

The paper "Random matrix theory and the loss surfaces of neural networks" explores the application of random matrix theory (RMT) to enhance the theoretical understanding of neural network loss surfaces. Despite the practical success of neural networks, their theoretical foundations haven't been as extensively developed. This work seeks to bridge that gap by employing RMT, which has been successfully used in fields like physics, to delve into the phenomenology of neural networks.

Key contributions of this paper include:

1. Extension of Prior Work: The author significantly extends existing research on using RMT to understand neural network loss surfaces, generalizing across various architectures. This is vital as different architectures can exhibit diverse loss surface characteristics.
2. Local Random Matrix Universality: The paper identifies local random matrix universality in real neural networks. This concept is a critical insight, as it implies that certain statistical properties of random matrices can model aspects of neural network loss surfaces.
3. Analysis of Hessians and Spectra: By leveraging the modeling assumptions informed by RMT, the paper derives new insights into the Hessians of neural network loss surfaces and their spectral properties. Understanding these properties is essential for comprehending the behavior of neural network optimization processes.
4. Impact on Optimization: The research uses random matrix models to investigate contemporary neural network training methodologies. Notably, this leads to the development of a novel variant of a widely-used optimization algorithm, showcasing practical implications of the theoretical findings.
5. Theoretical and Experimental Insights: The paper not only contributes to theory but also highlights experimental discoveries, pointing to the limits of current approaches and proposing new roles for RMT in deep learning theory.

Overall, this thesis positions random matrix theory as a pivotal tool in the theoretical paper of neural networks, offering novel theoretical results and practical advancements in optimization strategies. The introduction of local random matrix universality opens new avenues for understanding and improving deep learning models.