Tunability: Importance of Hyperparameters of Machine Learning Algorithms (1802.09596v3)

Abstract: Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.

Tunability: Importance of Hyperparameters of Machine Learning Algorithms

The paper "Tunability: Importance of Hyperparameters of Machine Learning Algorithms," authored by Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl, addresses the challenges associated with hyperparameter tuning in ML algorithms. The work stands out by formalizing a statistical framework for hyperparameter tuning and providing an extensive empirical analysis across different ML techniques.

Key Contributions

1. Formalization of Hyperparameter Tuning: The authors present a statistical point of view for hyperparameter tuning, offering definitions and measures that describe the tunability of both entire algorithms and individual hyperparameters. This formalization is a valuable contribution as it provides a structured method to evaluate the necessity and impact of tuning.
2. Large-Scale Benchmarking Study: Using a set of 38 datasets from the OpenML platform, the authors have performed an extensive comparison across six popular machine learning algorithms including gradient boosting, random forest, and SVM. This benchmarking serves as the foundation for their proposals on data-based default hyperparameters and tunability measures.
3. Surrogate Model Approach: The paper introduces the application of surrogate regression models to estimate algorithm performance across hyperparameter configurations. The surrogate models serve as a cornerstone for evaluating the risk profiles for hyperparameters, which are otherwise computationally expensive to determine.

Numerical Results

The research highlights some key numerical results:

• Algorithm Tunability: Algorithms like glmnet and SVM showed considerable tunability, whereas random forest exhibited minor tunability. This is corroborated by quantitative measures which showed mean tunability scores of 0.069 for glmnet and 0.010 for ranger.
• Optimal Defaults: The paper proposes new default parameters that often yield performance improvements over package defaults. For instance, the improvement for glmnet using optimal defaults compared to package defaults was noted to be 0.045.
• Surrogate Model Performance: Random forest models were identified as the most effective surrogate models across various datasets, boasting high R² scores and robust predictive performance.

Implications

The exploration into hyperparameter tunability has direct implications both practically and theoretically. Practically, the ability to identify when and which hyperparameters to tune can streamline model development, save computational resources, and improve predictive performance. Theoretically, understanding the impact of hyperparameter configurations across numerous datasets can guide the design of more robust learning algorithms.

Future Directions

Looking ahead, extending this framework to incorporate adaptive hyperparameter tuning strategies informed by dataset-specific characteristics could further enhance performance. Additionally, applying these methods to other learning domains such as multi-class classification or regression and evolving hyperparameter exploration strategies for high-dimensional spaces represent promising avenues for future work.

Overall, this paper provides a well-grounded approach to understanding and optimizing hyperparameters, contributing valuable insights to the ML research community. The methodical approach and empirical evidences presented form a strong basis for continued research and application in machine learning algorithm development and optimization.