Adaptive Hyperparameter Optimization for Continual Learning Scenarios (2403.07015v2)
Abstract: Hyperparameter selection in continual learning scenarios is a challenging and underexplored aspect, especially in practical non-stationary environments. Traditional approaches, such as grid searches with held-out validation data from all tasks, are unrealistic for building accurate lifelong learning systems. This paper aims to explore the role of hyperparameter selection in continual learning and the necessity of continually and automatically tuning them according to the complexity of the task at hand. Hence, we propose leveraging the nature of sequence task learning to improve Hyperparameter Optimization efficiency. By using the functional analysis of variance-based techniques, we identify the most crucial hyperparameters that have an impact on performance. We demonstrate empirically that this approach, agnostic to continual scenarios and strategies, allows us to speed up hyperparameters optimization continually across tasks and exhibit robustness even in the face of varying sequential task orders. We believe that our findings can contribute to the advancement of continual learning methodologies towards more efficient, robust and adaptable models for real-world applications.
- Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623–2631, 2019.
- Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
- Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920–15930, 2020.
- Rethinking experience replay: a bag of tricks for continual learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2180–2187. IEEE, 2021.
- Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420, 2018.
- A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
- Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):141–142, 2012.
- Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pages 3–33, 2019.
- Adaptive regularization for class-incremental learning. arXiv preprint arXiv:2303.13113, 2023.
- Class-incremental learning with repetition. arXiv preprint arXiv:2301.11396, 2023.
- Continual learning for predictive maintenance: Overview and challenges. Intelligent Systems with Applications, page 200251, 2023.
- An efficient approach for assessing hyperparameter importance. In International conference on machine learning, pages 754–762. PMLR, 2014.
- Learning multiple layers of features from tiny images. 2009.
- Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
- Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges. Information fusion, 58:52–68, 2020.
- Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
- Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118, 2018.
- Core50: a new dataset and benchmark for continuous object recognition. In Conference on robot learning, pages 17–26. PMLR, 2017.
- Avalanche: an end-to-end library for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3600–3610, 2021.
- Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
- Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28–51, 2022.
- Practical recommendations for replay-based continual learning methods. In International Conference on Image Analysis and Processing, pages 548–559. Springer, 2022.
- Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 2019.
- Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 524–540. Springer, 2020.
- Distilled replay: Overcoming forgetting through synthetic samples. In International Workshop on Continual Semi-Supervised Learning, pages 104–117. Springer, 2021.
- Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016.
- Continual-learning-as-a-service (claas): On-demand efficient adaptation of predictive models. arXiv preprint arXiv:2206.06957, 2022.
- Hyperparameter transfer across developer adjustments. arXiv preprint arXiv:2010.13117, 2020.
- Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2367–2376, 2018.
- Renate: A library for real-world continual learning. arXiv preprint arXiv:2304.12067, 2023.
- Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987–3995. PMLR, 2017.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.