A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation (1810.13243v1)

Abstract: The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz., mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.

• The paper finds that cosine learning rate restarts do not merely enable escape from local minima but guide trajectories through wider loss basins.
• It demonstrates that learning rate warmup stabilizes training by limiting drastic weight changes in deeper layers, similar to freezing fully-connected layers.
• It reveals that knowledge distillation primarily benefits deeper layers by transferring 'dark knowledge' from teacher to student to boost model discrimination.

Analysis of Deep Learning Heuristics: Learning Rate Restarts, Warmup, and Distillation

The paper "A closer look at Deep Learning heuristics: Learning rate restarts, warmup and distillation" delivers a rigorous empirical investigation into the efficacy of commonly used heuristics in deep learning training protocols. These heuristics, though often employed to enhance convergence rates and improve final model performance, lack comprehensive theoretical foundations. The paper revisits these strategies using two contemporary analytical tools: Mode Connectivity (MC) and Canonical Correlation Analysis (CCA).

Key Heuristics Investigated

Learning Rate Schedules: Cosine Restarts

Cosine learning rate decay, exemplified in the form of stochastic gradient descent with restarts (SGDR), is scrutinized under the paper's experimental lens. The research undermines the widely held assumption about SGDR aiding optimization by facilitating escape from local minima. Through loss surface interpolation and MC, it finds that paths indeed cross barriers, but the explanation of converging to and escaping from local minima is oversimplified. The paper further suggests that iterates between restarts operate over barriers more due to their trajectory into wider basins than due to true local minima escape.

Learning Rate Warmup

The warmup strategy, crucial in stabilizing training with large batch sizes, is re-examined through CCA. The paper determines that learning rate warmup notably affects deeper network layers by limiting drastic weight changes, hinting at its role in averting training instability. By freezing the fully-connected layers initially, a comparable stabilization was observed, indicating an alternate possible heuristic.

Knowledge Distillation

This heuristic is typically engaged to transfer knowledge from a large teacher model to a smaller student model, purportedly enhancing student performance. Investigating this through CCA revealed that the primary beneficiary layers of latent knowledge transfer are the deeper layers of the student network. This suggests the weight of the "dark knowledge" resides primarily in the network's discriminative areas rather than its feature extraction stage.

Methodological Tools

• Mode Connectivity: MC is employed to explore loss surface landscapes, notably revealing high-accuracy connections between local optima obtained from disparate training regimes. This robust approach underscores the limitations of relying on intuitive and simplistic low-dimensional visualizations.
• Canonical Correlation Analysis: By comparing neuron activations, CCA provides insights into the layer-wise representational changes over training epochs. This effectively highlights portions of the network most impacted by various heuristics.

Implications and Future Directions

The paper's findings imply several implications. For instance, the escape from local minima facilitated by SGDR may be more about the quality of traversal paths rather than frequency of restarts. Moreover, understanding that deeper layers are critical in both stabilization (warmup) and knowledge transfer (distillation) suggests enhanced focus on these layers could yield optimized training strategies and architectures. The results invite deeper scrutiny and potential recalibration of prevalent heuristic dependencies across varied architectures and applications.

While the paper provides compelling arguments substantiated through rigorous empirical trials, it invites further exploration into the scalability of such heuristics, especially on larger datasets and models beyond benchmark paradigms. Expanding the suite of analytical tools and exploring combinations of existing ones could unveil more about the underlying dynamics at play in model training, especially in the burgeoning field of AI model interpretability and robustness.