An Exponential Learning Rate Schedule for Deep Learning (1910.07454v3)

Abstract: Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN. 1. Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some $(1 +\alpha)$ factor in every epoch for some $\alpha >0$. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the net stays well-behaved due to normalization. 2. Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + StandardRate Tuning + Weight Decay + Momentum. This equivalence holds for other normalization layers as well, Group Normalization, LayerNormalization, Instance Norm, etc. 3. A worked-out toy example illustrating the above linkage of hyper-parameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.

An Exponential Learning Rate Schedule for Deep Learning

The paper "An Exponential Learning Rate Schedule for Deep Learning" by Zhiyuan Li and Sanjeev Arora presents a novel approach to optimizing deep learning models through the use of an exponential learning rate (LR) schedule. The research investigates how batch normalization (BN) interacts with learning rates and other common optimization techniques like weight decay and momentum to stabilize and improve performance in deep neural networks.

Key Contributions and Results

1. Exponential Increase in Learning Rate: The paper outlines that SGD with momentum can be effectively paired with an exponentially increasing learning rate where the rate grows by a factor of $(1+\alpha)$ each epoch. This approach, despite increasing the network weights significantly during training, maintains network stability due to normalization techniques like BN.
2. Mathematical Equivalence: A major contribution is the derivation of mathematical equivalence showing that the proposed exponential LR schedule achieves results similar to those attained using traditional methods such as SGD with a conventional learning rate schedule plus weight decay and momentum. This equivalence extends to other normalization methods like Group Normalization, Layer Normalization, and Instance Normalization.
3. Interplay Between Hyperparameters: A detailed examination of hyperparameters reveals that using both weight decay and normalization layers leads to a unique optimization process, distinct from using either technique alone. The paper provides examples where convergence fails if both weight decay and normalization are employed, underlining the complex interplay between these aspects.
4. Trajectory Analysis and Scale Invariance: The authors advocate for a trajectory analysis approach, proposing that this lens better captures the dynamics of optimization because scale invariance plays a critical role in the successful training of deep networks.
5. Generalization to Other Schedules: Exploring other learning rate schedules, such as the Triangular and Cosine LR schedules, the paper shows that these can also be interpreted within the exponential framework allowed by BN.

Implications and Future Directions

• Simplification of Hyperparameter Tuning: By demonstrating the compatibility and equivalence of exponential LR schedules with traditional methods, the paper potentially simplifies the search space for hyperparameter tuning in practical settings.
• Robustness and Stability: The paper suggests that networks trained with an exponential LR schedule may inherently exhibit more robust behavior over variable initialization scales, contributing to better stability during prolonged training.
• Theoretical Framework Revisions: The results call for a reassessment of existing optimization frameworks in the context of deep learning. The traditional analysis does not readily account for the kind of dynamics observed with scale-invariant networks, especially when large learning rates are used.

In conclusion, this paper advances the understanding of learning rate schedules in deep learning, particularly in settings involving batch normalization. It introduces a theoretically sound and empirically verified exponential schedule that could redefine norms in optimizing deep neural network training. Further exploration into adapting these findings to other architectural and optimization frameworks might reveal new dimensions of the capabilities in contemporary neural network models.