Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates (1708.07120v3)

Abstract: In this paper, we describe a phenomenon, which we named "super-convergence", where neural networks can be trained an order of magnitude faster than with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with one learning rate cycle and a large maximum learning rate. A primary insight that allows super-convergence training is that large learning rates regularize the training, hence requiring a reduction of all other forms of regularization in order to preserve an optimal regularization balance. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. Experiments demonstrate super-convergence for Cifar-10/100, MNIST and Imagenet datasets, and resnet, wide-resnet, densenet, and inception architectures. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. The architectures and code to replicate the figures in this paper are available at github.com/lnsmith54/super-convergence. See http://www.fast.ai/2018/04/30/dawnbench-fastai/ for an application of super-convergence to win the DAWNBench challenge (see https://dawn.cs.stanford.edu/benchmark/).

• The paper introduces super-convergence, demonstrating that large cyclical learning rates can cut training times by up to an order of magnitude.
• Empirical results across architectures like ResNet and datasets such as CIFAR-10 reveal significantly improved accuracies, for example, 92.4% in 10,000 iterations.
• Super-convergence especially benefits scenarios with limited data by reducing the need for extensive regularization while efficiently optimizing neural networks.

Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

The paper "Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates" by Leslie N. Smith and Nicholay Topin presents a method to accelerate the training of neural networks significantly, termed "super-convergence." This phenomenon enables an order of magnitude faster training compared to conventional approaches. The method involves utilizing a cyclical learning rate (CLR) schedule, which incorporates a single cycle with a large maximum learning rate.

Key Findings

1. Super-Convergence Mechanism
   • Super-convergence exploits large learning rates achieved through CLR, resulting in faster convergence and improved model performance. This approach involves reducing traditional forms of regularization since large learning rates inherently regularize the training process.
2. Empirical Observations
   • Results across several architectures, including ResNet, DenseNet, and Inception, and datasets such as CIFAR-10/100, MNIST, and ImageNet, demonstrate accelerated training with improved accuracy. For instance, on CIFAR-10 with ResNet-56, super-convergence achieved a 92.4% accuracy in only 10,000 iterations compared to 91.2% in 80,000 iterations using typical schedules.
3. Training with Limited Data
   • When the amount of labeled data is restricted, super-convergence results in more significant performance improvements, highlighting its utility in data-scarce scenarios.
4. Hessian-Free Optimization
   • The paper introduces a simplification of the Hessian-Free optimization method to estimate optimal learning rates, further supporting the theoretical understanding of large learning rates finding wide, flat minima.

Theoretical and Practical Implications

The results challenge existing paradigms regarding learning rate boundaries and the necessity of extensive regularization practices. The implication is that super-convergence can dramatically modify existing training protocols, reducing resource consumption while achieving competitive or superior performance.

Theoretically, the phenomenon suggests a need to reconsider the interplay of learning rate dynamics, generalization properties, and optimization landscapes in deep learning. Practically, this could reshape how neural networks are trained, particularly in resource-limited environments or applications demanding rapid model deployment.

Discussion and Future Directions

The introduction of super-convergence prompts numerous questions and potential research directions. It necessitates deeper exploration into the underlying mechanisms that allow large learning rates to enhance generalization and stability. Further investigation could focus on:

• The impact of various forms of data augmentation and batch normalization in enhancing or limiting super-convergence.
• Developing adaptive learning rate strategies that dynamically adjust based on the data distribution and model architecture.
• Extending analysis to other deep learning paradigms, such as reinforcement learning or unsupervised learning, where labeling or computation is a constraint.

Overall, super-convergence represents a significant shift in neural network training methodology, combining theoretical insights with practical efficacy. The lessons from this work can lead researchers to new avenues in both deep learning optimization and model generalization.