Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2007.04074v3)

Abstract: Automated Machine Learning (AutoML) supports practitioners and researchers with the tedious task of designing machine learning pipelines and has recently achieved substantial success. In this paper, we introduce new AutoML approaches motivated by our winning submission to the second ChaLearn AutoML challenge. We develop PoSH Auto-sklearn, which enables AutoML systems to work well on large datasets under rigid time limits by using a new, simple and meta-feature-free meta-learning technique and by employing a successful bandit strategy for budget allocation. However, PoSH Auto-sklearn introduces even more ways of running AutoML and might make it harder for users to set it up correctly. Therefore, we also go one step further and study the design space of AutoML itself, proposing a solution towards truly hands-free AutoML. Together, these changes give rise to the next generation of our AutoML system, Auto-sklearn 2.0. We verify the improvements by these additions in an extensive experimental study on 39 AutoML benchmark datasets. We conclude the paper by comparing to other popular AutoML frameworks and Auto-sklearn 1.0, reducing the relative error by up to a factor of 4.5, and yielding a performance in 10 minutes that is substantially better than what Auto-sklearn 1.0 achieves within an hour.

• The paper demonstrates a 4.5-fold error reduction using Portfolio Successive Halving to efficiently allocate computational resources.
• The paper introduces dynamic model selection via successive halving, quickly evaluating candidate pipelines with partial training data.
• The paper automates high-level design decisions with a meta-learning layer, enabling a truly hands-free AutoML experience under strict time constraints.

Overview of Auto-Sklearn 2.0: A New Paradigm in AutoML

The paper, "Auto-sklearn 2.0: Hands-free AutoML via Meta-Learning," introduces a significant advancement in the domain of Automated Machine Learning (AutoML) by detailing the development and empirical evaluation of Auto-sklearn 2.0. This system evolves from previous iterations by integrating sophisticated techniques such as meta-learning and efficient budget allocation strategies, offering practical solutions for hands-free machine learning. The methodological innovations presented have been rigorously validated on extensive benchmark datasets, emphasizing their utility in real-world scenarios.

AutoML has emerged as a pivotal domain to democratize machine learning by automating the creation of machine learning pipelines. The complexity involved in designing optimal pipelines for diverse datasets often acts as a barrier for rapid experimentation and deployment. AutoML systems such as Auto-sklearn attempt to bridge this gap by automating both algorithm selection and hyperparameter optimization within the constraints of limited computational budgets.

Key Contributions

1. Portfolio Successive Halving (PoSH): A keystone of Auto-sklearn 2.0 is PoSH, a strategic method that enhances traditional AutoML techniques by amalgamating bandit strategies for judicious budget allocation with an innovative portfolio-building approach. Unlike meta-feature-based meta-learning, this portfolio method is empirical and static, providing a set of well-performing initial configurations that can rapidly yield competitive models across various datasets. This approach demonstrated a reduction of relative error by a factor of 4.5 compared to its predecessors.
2. Improved Model Selection Strategies: The manuscript delineates how Auto-sklearn 2.0 improves upon its earlier versions by extending model selection strategies with the addition of successive halving (SH). This enables more dynamic evaluation of candidate pipelines based on partial training data, thus efficiently allocating computational resources and preventing time-intensive configurations from monopolizing the limited time available.
3. Automating High-level Design Decisions: The system intelligently automates decisions about the AutoML process itself, employing a meta-learning layer that selects the most pertinent configuration strategy given a dataset's specific characteristics. This design choice takes the systemic approach a step closer to fully hands-off AutoML by considering not only specific algorithmic choices but also overarching procedural options influenced by time and resource constraints.

Empirical Validation and Comparative Performance

Significant empirical evaluation supports the proposed methods, with experiments conducted on a suite of 39 AutoML benchmark datasets. The results demonstrate that Auto-sklearn 2.0 reliably outperforms current popular frameworks such as TPOT and Auto-WEKA under strict time budgets. For example, in a 10-minute competition condition, Auto-sklearn 2.0 yielded performance superior to what its predecessor could achieve in 60 minutes. These results attest to the efficacy of the system in producing high-performance models rapidly, underscoring the practical applicability of these innovations in diverse machine learning tasks.

Implications and Future Directions

The practical implications of these advances are far-reaching, primarily in scenarios where quick prototyping with robust models is requisite, such as in business intelligence and any domain requiring rapid adaptation of models to changing data distributions. The integration of meta-learning for high-level decision automatization points towards a future where AutoML frameworks are capable of intelligently contextualizing their strategies based on content-driven insights from the data itself.

A natural progression of this work would focus on extending these methodologies to multi-objective optimization scenarios and exploring further algorithm adaptations that leverage incremental datasets and online learning environments. Developing finer-grained portfolios and dynamically adaptive budget strategies that respond in real-time to ongoing model evaluation metrics holds promising potential as well.

Conclusion

The paper embodies a significant stride towards truly hands-free AutoML by incorporating meta-learning dynamics traditionally reserved for algorithm selection and adapting them to the AutoML process itself. Auto-sklearn 2.0 emerges as a robust system capable of handling the intricate interplay of constraints and objectives inherent in machine learning model design, presenting an effective toolset for both experts and novices in the field. Through meticulous experimentation and validated improvements, the paper contributes a foundational framework and methodology likely to inspire successive innovations within the automated machine learning space.