On Empirical Comparisons of Optimizers for Deep Learning (1910.05446v3)

Abstract: Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

• The paper reveals that meticulous hyperparameter tuning dramatically changes performance rankings among deep learning optimizers.
• The study employed comprehensive experiments across tasks like ImageNet classification and language modeling to verify optimizer inclusion relations.
• The findings underscore that general optimizers like Adam, when properly tuned, consistently match or exceed the performance of specialized variants.

Analysis of "On Empirical Comparisons of Optimizers for Deep Learning"

The paper presents a meticulous examination of the intrinsic effects of hyperparameter tuning on the empirical evaluation of deep learning optimizers. The research articulates distinct insights into the role of hyperparameters in shaping the comparative efficacy of popular neural network optimizers, such as SGD, Momentum, RMSprop, Adam, and Adam-like adaptive gradient methods including NAdam.

Core Contributions

1. Sensitivity to Hyperparameter Tuning:
   • The paper underscores the criticality of hyperparameter tuning in optimizer evaluation, demonstrating that minor adjustments can lead to significant changes in performance rankings. This revelation challenges prior results and stresses the importance of a detailed and nuanced tuning protocol to capture a true comparison.
2. Inclusion Relationships:
   • Importantly, the authors articulate inclusion relationships between optimizers, arguing that more general optimizers (e.g., Adam) should theoretically never underperform compared to their specializations (e.g., Momentum) when properly tuned. This observation challenges some empirical studies in the literature but is convincingly corroborated by the results in this analysis.
3. Empirical Validation Across Diverse Workloads:
   • The research conducts experiments across various tasks, from image classification with deeply layered models like ResNet-50 on ImageNet to LLMing with Transformers on the LM1B dataset. These experiments consistently show that broader optimizers indeed encapsulate and potentially extend the performance of more simplistic variants when hyperparameter spaces are expansively explored.

Numerical Strengths and Methodological Integrity

The paper is exemplary in its methodical approach to tuning all conceivable hyperparameters. The authors make an impressive effort to test these optimizers across a voluminous parameter space, rather than succumbing to default or minimal tuning, which is common in prior art. This approach ensures that the performance-reducing limitations observed in earlier works are meticulously addressed.

Through rigorous bootstrapping techniques and the consideration of a variable number of trials structured through a quasi-random uniform search strategy, the authors precisely measure their claims. This ensures statistical robustness in their reported performance metrics of training error, validation, and test accuracy.

Theoretical and Practical Implications

The primary implication of this paper for both theory and practice is the reaffirmation that in practice, the inclusion principles hold: a general optimizer cannot underperform its simpler counterparts given sufficiently tuned hyperparameters. Practically, this suggests that practitioners should allocate substantial resources towards hyperparameter tuning, notably for adaptive methods like Adam, to achieve optimal performance on their specific tasks.

From a theoretical vantage point, this research prompts a need for consistency in the definition and tuning of optimization algorithms as it pertains to the generalizability of empirical findings across different workloads. It opens up avenues for further investigation into devising adaptive or automated mechanisms to navigate these hyperparameter spaces more efficiently.

Future Directions

Future research should aim to develop theoretically grounded and empirically validated methodologies that can expedite this extended hyperparameter tuning process. Moreover, it would be beneficial to explore the implications of these findings across different architectural paradigms possibly with even larger batch sizes, as speculated by the authors, to delineate the boundaries of these inclusion relationships under different conditions clearly.

Conclusion

The paper decisively challenges and refines the understanding of optimizer performance in deep learning, presenting compelling evidence that with comprehensive hyperparameter tuning, adaptive gradient methods hold significant promise in practical applications. This work presents a clarion call for the deep learning community to reconsider optimized evaluation methodologies, prioritizing thorough hyperparameter exploration in empirical comparisons of optimization algorithms.