Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction (2106.15013v4)

Abstract: Recently there has been significant theoretical progress on understanding the convergence and generalization of gradient-based methods on nonconvex losses with overparameterized models. Nevertheless, many aspects of optimization and generalization and in particular the critical role of small random initialization are not fully understood. In this paper, we take a step towards demystifying this role by proving that small random initialization followed by a few iterations of gradient descent behaves akin to popular spectral methods. We also show that this implicit spectral bias from small random initialization, which is provably more prominent for overparameterized models, also puts the gradient descent iterations on a particular trajectory towards solutions that are not only globally optimal but also generalize well. Concretely, we focus on the problem of reconstructing a low-rank matrix from a few measurements via a natural nonconvex formulation. In this setting, we show that the trajectory of the gradient descent iterations from small random initialization can be approximately decomposed into three phases: (I) a spectral or alignment phase where we show that that the iterates have an implicit spectral bias akin to spectral initialization allowing us to show that at the end of this phase the column space of the iterates and the underlying low-rank matrix are sufficiently aligned, (II) a saddle avoidance/refinement phase where we show that the trajectory of the gradient iterates moves away from certain degenerate saddle points, and (III) a local refinement phase where we show that after avoiding the saddles the iterates converge quickly to the underlying low-rank matrix. Underlying our analysis are insights for the analysis of overparameterized nonconvex optimization schemes that may have implications for computational problems beyond low-rank reconstruction.

• The paper demonstrates that small random initialization in gradient descent for overparameterized low-rank matrix reconstruction resembles spectral initialization, guiding optimization towards globally optimal and generalizable solutions.
• It analyzes the optimization process through three phases (spectral/alignment, saddle avoidance, local refinement) showing how gradient descent avoids degenerate saddle points.
• The theoretical findings offer insights into optimization and generalization in highly nonconvex and overparameterized settings, potentially impacting areas like neural networks.

Overview of "Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction"

This paper by Dominik Stöger and Mahdi Soltanolkotabi offers an in-depth theoretical analysis of gradient descent algorithms for overparameterized low-rank matrix reconstruction problems. Key to their exploration is the role of small random initialization, which, they argue, mimics spectral learning methods, thereby providing optimization and generalization guarantees even in nonconvex regimes.

The paper centers on low-rank Positive Semidefinite (PSD) matrix recovery, a foundational problem prevalent in domains like recommendation systems, phase retrieval, and quantum tomography. Given linear measurements, the goal is to reconstruct the underlying low-rank matrix. The authors leverage a nonconvex optimization approach via matrix factorization, employing gradient descent from a small random initialization.

Key Contributions

1. Spectral Phase Analysis: The authors demonstrate that small random initialization followed by gradient descent iterations resembles spectral initialization. Spectral methods have been historically popular due to their ability to find globally optimal solutions. Here, gradient descent implicitly integrates a spectral bias directing it towards solutions that are globally optimal and generalize well.
2. Optimization in Overparameterized Regimes: The paper provides thorough analyses divided into three phases: spectral/alignment, saddle avoidance, and local refinement.
   • Spectral/Alignment Phase: The authors prove that initial iterations align the gradient descent trajectory with the spectral path, effectively learning the column space of the signal.
   • Saddle Avoidance Phase: They show that the trajectory avoids degenerate saddle points, moving towards global optima.
   • Local Refinement Phase: Quick convergence towards the low-rank matrix occurs after avoiding saddle points, achieving a test error commensurate with initialization scale.
3. Sample Complexity Insights: Despite operating in nonconvex landscapes with overparameterization, the paper ensures that gradient descent leads to generalizable optima. It achieves this using sample complexities that are suboptimal in $r_*$ but pertinent for empirical settings.

Theoretical Implications

The paper’s theoretical framework extends beyond matrix reconstruction to a variety of machine learning tasks, potentially impacting areas involving highly nonconvex landscapes and overparameterization, such as neural networks. Results suggest that small initialization may mitigate pitfalls associated with lazy training paradigms, providing a foundation for practical machine learning insights.

Practical Implications & Future Directions

Practically, these findings enable enhanced performance models by suggesting initialization strategies that harness spectral biases without needing sophisticated initialization techniques. The framework could be generalized, impacting a broader class of models, including deep networks and beyond. Future research may focus on bounding sample complexities rigorously and extending spectral alignment principles in broader applications.

This work is crucial not only for theoretical advancements in understanding optimization landscapes but also for practical considerations in machine learning, especially in settings requiring efficient model training with guarantees on generalization and convergence.