Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers

Abstract: Choosing the optimizer is considered to be among the most crucial design decisions in deep learning, and it is not an easy one. The growing literature now lists hundreds of optimization methods. In the absence of clear theoretical guidance and conclusive empirical evidence, the decision is often made based on anecdotes. In this work, we aim to replace these anecdotes, if not with a conclusive ranking, then at least with evidence-backed heuristics. To do so, we perform an extensive, standardized benchmark of fifteen particularly popular deep learning optimizers while giving a concise overview of the wide range of possible choices. Analyzing more than $50,000$ individual runs, we contribute the following three points: (i) Optimizer performance varies greatly across tasks. (ii) We observe that evaluating multiple optimizers with default parameters works approximately as well as tuning the hyperparameters of a single, fixed optimizer. (iii) While we cannot discern an optimization method clearly dominating across all tested tasks, we identify a significantly reduced subset of specific optimizers and parameter choices that generally lead to competitive results in our experiments: Adam remains a strong contender, with newer methods failing to significantly and consistently outperform it. Our open-sourced results are available as challenging and well-tuned baselines for more meaningful evaluations of novel optimization methods without requiring any further computational efforts.

• The paper presents an exhaustive benchmark of 15 popular deep learning optimizers, revealing similar performance profiles across standard tasks.
• It evaluates performance on diverse tasks—from simple landscapes to CIFAR and VAE models—highlighting impacts of hyperparameter tuning and learning rate schedules.
• Findings suggest that testing default settings can identify promising optimizers, advocating for shifts towards more innovative algorithmic strategies.

Benchmarking Deep Learning Optimizers

The paper "Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers" (2007.01547) provides an exhaustive empirical analysis of various deep learning optimizers to derive insights into their relative performance across a range of tasks. This work addresses the prevalent challenge in machine learning of selecting the appropriate optimizer from a vast array of potential options, each with its respective tunable hyperparameters. The key contributions include a comprehensive benchmark of fifteen popular optimizers, public dissemination of extensive performance data for further research, and the proposition that many current optimizers exhibit similar performance profiles across standard tasks, prompting a reconsideration of ongoing optimizer development strategies.

Methodology

Optimizers and Tasks

The authors selected fifteen widely recognized optimization algorithms, including stochastic gradient methods and adaptive algorithms such as Adam, AdaBelief, and RMSProp. The benchmark evaluates these optimizers on eight diverse problems sourced from the DeepOBS suite. This approach ensures coverage of a broad spectrum of practical deep learning applications. The tasks involve varying complexities, model architectures, and data sets, ranging from simple artificial landscapes to image classification on CIFAR datasets and generative models like VAEs.

Hyperparameter Tuning and Learning Rate Schedules

The study employs four distinct hyperparameter tuning budgets—spanning from reliance on default parameters to extensive random searches with up to 75 tuning runs. Additionally, the impact of learning rate schedules is examined using different strategies: constant, cosine decay, cosine decay with warm restarts, and trapezoidal warm-up. The findings suggest notable variations in performance with different schedules and highlight that optimizers with well-chosen default parameters often narrowly outperform optimizers subjected to extensive hyperparameter tuning.

Key Findings and Insights

Performance Variability and Default Settings

The empirical results indicate significant performance variability across different tasks, confirming the absence of a universally dominant optimizer. This variability supports the notion that each problem may naturally favor distinct optimizers due to unique characteristics or model-fit dynamics. While adaptive methods like Adam often benefit from robust default parameters, fixed-methods such as SGD generally require more nuanced tuning.

Strategy Recommendations for Practitioners

For practitioners embarking on new deep learning challenges, the study recommends a pragmatic approach of testing several optimizers using default settings to identify promising candidates, followed by more targeted tuning of top contenders. This strategy enables efficient use of computational resources without exhaustive hyperparameter searches for a single optimizer.

Implications for Optimization Research

The analysis underscores the redundancy observed among the many optimizers designed in recent years, emphasizing a need for research directed at more innovative methodologies offering tangible improvements in efficiency, adaptability, or performance stability. Future work might better serve the community by exploring automatic tuning mechanisms, inner-loop parameter adjustments, or by focusing on niche optimizers optimized for specific problem domains.

Conclusion

The paper offers extensive empirical evidence underpinning several heuristic insights regarding the use of optimizers in deep learning. Open-sourcing the data associated with this benchmark provides a valuable resource for the community, offering pre-established baseline comparisons to facilitate the meaningful evaluation of new methods. It raises important considerations regarding the current trajectory of optimizer development, suggesting a broader emphasis on significant algorithmic advancements over incremental methodological variations.