Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization (2403.04720v4)

Published 7 Mar 2024 in cs.LG

Abstract: Effectively representing heterogeneous tabular datasets for meta-learning purposes remains an open problem. Previous approaches rely on predefined meta-features, for example, statistical measures or landmarkers. The emergence of dataset encoders opens new possibilities for the extraction of meta-features because they do not involve any handmade design. Moreover, they are proven to generate dataset representations with desired spatial properties. In this research, we evaluate an encoder-based approach to one of the most established meta-tasks - warm-starting of the Bayesian Hyperparameter Optimization. To broaden our analysis we introduce a new approach for representation learning on tabular data based on [Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from Tasks with Heterogeneous Attribute Spaces. In Advances in Neural Information Processing Systems, 2020]. The validation on over 100 datasets from UCI and an independent metaMIMIC set of datasets highlights the nuanced challenges in representation learning. We show that general representations may not suffice for some meta-tasks where requirements are not explicitly considered during extraction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. Meta-learning from Tasks with Heterogeneous Attribute Spaces. In Advances in Neural Information Processing Systems, volume 33, pages 6053–6063, 2020.
  2. Joaquin Vanschoren. Meta-learning. In Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren, editors, Automated Machine Learning: Methods, Systems, Challenges, pages 35–61. Springer International Publishing, 2019. doi: 10.1007/978-3-030-05318-5_2.
  3. Dataset2Vec: Learning Dataset Meta-Features. Data Mining and Knowledge Discovery, 35:964–985, 2021. doi: https://doi.org/10.1007/s10618-021-00737-9.
  4. The potential for artificial intelligence in healthcare. Future Healthcare Journal, 6(2):94–98, 2019. doi: 10.7861/futurehosp.6-2-94.
  5. Abdullah Alanazi. Using machine learning for healthcare challenges and opportunities. Informatics in Medicine Unlocked, 30:100924, 2022. doi: 10.1016/j.imu.2022.100924.
  6. Meta-features for meta-learning. Knowledge-Based Systems, 240:108101, 2022. doi: 10.1016/j.knosys.2021.108101.
  7. Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems, 25, 2012.
  8. Learning Hyperparameter Optimization Initializations. In 2015 IEEE International Conference on Data Science and Advanced Analytics. Institute of Electrical and Electronics Engineers Inc., 2015a. doi: 10.1109/DSAA.2015.7344817.
  9. Sequential Model-Free Hyperparameter Tuning. In 2015 IEEE International Conference on Data Mining, pages 1033–1038, 2015b. doi: 10.1109/ICDM.2015.20.
  10. Scalable Gaussian Process-Based Transfer Surrogates for Hyperparameter Optimization. Machine Learning, 107(1):43–78, 2018. doi: 10.1007/s10994-017-5684-y. Springer US.
  11. Scalable Meta-Learning for Bayesian Optimization. 2018. arXiv: 1802.02219.
  12. Consolidated learning: a domain-specific model-free optimization strategy with validation on metaMIMIC benchmarks. Machine Learning, pages 1–25, 2023. doi: 10.1007/s10994-023-06359-0. Publisher: Springer.
  13. OpenML: Networked Science in Machine Learning. SIGKDD Explorations, 15(2):49–60, 2013. doi: 10.1145/2641190.2641198.
  14. Kolby Nottingham Markelle Kelly, Rachel Longjohn. UCI Machine Learning Repository, 2021. URL https://archive.ics.uci.edu.
  15. Deep Metric Learning via Lifted Structured Feature Embedding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4004–4012, 2015.
  16. Deep Sets. In Advances in Neural Information Processing Systems, volume 30, 2017.
  17. Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems, volume 24, 2011.
  18. Methods for Hyperparameters Optimization in Learning Approaches: An Overview. pages 100–112, 2020. doi: 10.1007/978-3-030-64583-0_11.
  19. Efficient Hyperparameters optimization Through Model-based Reinforcement Learning and Meta-Learning. 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), pages 1036–1041, 2020. doi: 10.1109/hpcc-smartcity-dss50907.2020.00139.
  20. Automatic tuning of hyperparameters using Bayesian optimization. Evolving Systems, 12(1):217–223, 2021. doi: 10.1007/s12530-020-09345-2.
  21. MIMIC-IV, 2023.
  22. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
  23. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785–794, 2016. doi: 10.1145/2939672.2939785.
  24. Tunability: Importance of Hyperparameters of Machine Learning Algorithms. The Journal of Machine Learning Research, 20(1):1934–1965, 2019.
  25. Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems, volume 35, pages 507–520, 2022.
  26. Efficient transfer learning method for automatic hyperparameter tuning. In Artificial intelligence and statistics, pages 1077–1085. PMLR, 2014.
  27. Tabular few-shot generalization across heterogeneous feature spaces. arXiv preprint arXiv:2311.10051, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets