Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box Optimization (2404.07539v1)

Published 11 Apr 2024 in cs.NE

Abstract: The recently proposed MA-BBOB function generator provides a way to create numerical black-box benchmark problems based on the well-established BBOB suite. Initial studies on this generator highlighted its ability to smoothly transition between the component functions, both from a low-level landscape feature perspective, as well as with regard to algorithm performance. This suggests that MA-BBOB-generated functions can be an ideal testbed for automated machine learning methods, such as automated algorithm selection (AAS). In this paper, we generate 11800 functions in dimensions $d=2$ and $d=5$, respectively, and analyze the potential gains from AAS by studying performance complementarity within a set of eight algorithms. We combine this performance data with exploratory landscape features to create an AAS pipeline that we use to investigate how to efficiently select training sets within this space. We show that simply using the BBOB component functions for training yields poor test performance, while the ranking between uniformly chosen and diversity-based training sets strongly depends on the distribution of the test set.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Evolutionary Algorithms for Parameter Optimization—Thirty Years Later. Evolutionary Computation 31, 2 (2023), 81 – 122.
  2. Benchmarking in Optimization: Best Practice and Open Issues. CoRR abs/2007.03488 (2020). arXiv:2007.03488 https://arxiv.org/abs/2007.03488
  3. Per instance algorithm configuration of CMA-ES with limited budget. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO). ACM, 681–688. https://doi.org/10.1145/3071178.3071343
  4. SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO). ACM. https://doi.org/10.1145/3512290.3528809
  5. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’16). ACM, New York, NY, USA, 785–794. https://doi.org/10.1145/2939672.2939785
  6. Iohexperimenter: Benchmarking platform for iterative optimization heuristics. Evolutionary Computation (2023), 1–6.
  7. Konstantin Dietrich and Olaf Mersmann. 2022. Increasing the Diversity of Benchmark Function Sets Through Affine Recombination. In Proceedings of Parallel Problem Solving from Nature (PPSN’22) (LNCS, Vol. 13398), Günter Rudolph, Anna V. Kononova, Hernán E. Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tusar (Eds.). Springer, 590–602. https://doi.org/10.1007/978-3-031-14714-2_41
  8. Reproducibility files and additional figures. Code and data repository (Zenodo): doi.org/10.5281/zenodo.10911264 Figure repository (Figshare): doi.org/10.6084/m9.figshare.25127501.
  9. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software 36, 1 (2021), 114–144. https://doi.org/10.1080/10556788.2020.1808977
  10. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633/document
  11. Automated Machine Learning - Methods, Systems, Challenges. Springer. https://doi.org/10.1007/978-3-030-05318-5
  12. Automated Algorithm Selection: Survey and Perspectives. Evolutionary Computation 27, 1 (2019), 3–45. https://doi.org/10.1162/evco_a_00242
  13. Detecting funnel structures by means of exploratory landscape analysis. GECCO 2015 - Proceedings of the 2015 Genetic and Evolutionary Computation Conference (2015), 265–272. https://doi.org/10.1145/2739480.2754642
  14. Pascal Kerschke and Heike Trautmann. 2019a. Automated Algorithm Selection on Continuous Black-Box Problems by Combining Exploratory Landscape Analysis and Machine Learning. Evolutionary Computation 27, 1 (2019), 99–127. https://doi.org/10.1162/evco_a_00236
  15. Pascal Kerschke and Heike Trautmann. 2019b. Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-package flacco. In Applications in Statistical Computing – From Music Data Analysis to Industrial Quality Improvement, Nadja Bauer, Katja Ickstadt, Karsten Lübke, Gero Szepannek, Heike Trautmann, and Maurizio Vichi (Eds.). Vol. 17. Springer, 93–123. https://doi.org/10.1007/978-3-030-25147-5_7
  16. Per-run algorithm selection with warm-starting using trajectory-based features. In International Conference on Parallel Problem Solving from Nature. Springer, 46–60.
  17. Comparing Algorithm Selection Approaches on Black-Box Optimization Problems. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO, companion material). ACM, 495–498. https://doi.org/10.1145/3583133.3590697
  18. Challenges of ELA-guided Function Evolution using Genetic Programming. arXiv preprint arXiv:2305.15245 (2023).
  19. BBOB Instance Analysis: Landscape Properties and Algorithm Performance Across Problem Instances. Springer-Verlag, 380–395. https://doi.org/10.1007/978-3-031-30229-9_25
  20. Monte Lunacek and Darrell Whitley. 2006. The dispersion metric and the CMA evolution strategy. GECCO 2006 - Genetic and Evolutionary Computation Conference 1 (2006), 477–484. https://doi.org/10.1145/1143997.1144085
  21. Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms. arXiv:2404.02031
  22. Exploratory landscape analysis. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO). ACM, 829–836. https://doi.org/10.1145/2001576.2001690
  23. Variance Reduction for Better Sampling in Continuous Domains. In Proceedings of Parallel Problem Solving from Nature (PPSN) (LNCS, Vol. 12269). Springer, 154–168. https://doi.org/10.1007/978-3-030-58112-1_11
  24. Rachael Morgan and Marcus Gallagher. 2014. Sampling Techniques and Distance Metrics in High Dimensional Continuous Landscape Analysis: Limitations and Improvements. IEEE Trans. Evol. Comput. 18, 3 (2014), 456–461. https://doi.org/10.1109/TEVC.2013.2281521
  25. Exploratory landscape analysis of continuous space optimization problems using information content. IEEE Transactions on Evolutionary Computation 19 (2015), 74–87. Issue 1. https://doi.org/10.1109/TEVC.2014.2302006
  26. Mario A. Muñoz and Kate Smith-Miles. 2020. Generating New Space-Filling Test Instances for Continuous Black-Box Optimization. Evol. Comput. 28, 3 (2020), 379–404. https://doi.org/10.1162/evco_a_00262
  27. RF+ clust for Leave-One-Problem-Out Performance Prediction. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar). Springer, 285–301.
  28. Algorithm Instance Footprint: Separating Easily Solvable and Challenging Problem Instances. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO). ACM, 529–537. https://doi.org/10.1145/3583131.3590424
  29. Raphael Patrick Prager and Heike Trautmann. 2023. Pflacco: Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems in Python. Evolutionary Computation (07 2023), 1–25. https://doi.org/10.1162/evco_a_00341
  30. Jérémy Rapin and Olivier Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad.
  31. Towards Explainable Exploratory Landscape Analysis: Extreme Feature Selection for Classifying BBOB Functions. In Proceedings of Applications of Evolutionary Computation (EvoApplications) (LNCS, Vol. 12694). Springer, 601–617. https://doi.org/10.1007/978-3-030-72699-7_2
  32. John R. Rice. 1976. The Algorithm Selection Problem. Advances in Computers 15 (1976), 65–118. https://doi.org/10.1016/S0065-2458(08)60520-3
  33. A collection of deep learning-based feature-free approaches for characterizing single-objective continuous fitness landscapes. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO). ACM, 657–665. https://doi.org/10.1145/3512290.3528834
  34. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Applied Soft Computing 90 (2020), 106138. https://doi.org/10.1016/j.asoc.2020.106138
  35. Transfer Learning Analysis of Multi-Class Classification for Landscape-Aware Algorithm Selection. Mathematics 10, 3 (2022). https://doi.org/10.3390/math10030432
  36. A Recommender System for Metaheuristic Algorithms for Continuous Optimization Based on Deep Recurrent Neural Networks. IEEE Transactions on Artificial Intelligence 1, 1 (2020), 5–18. https://doi.org/10.1109/TAI.2020.3022339
  37. Evolving the structure of Evolution Strategies. In Proceedings of IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 1–8. https://doi.org/10.1109/SSCI.2016.7850138
  38. Modular Differential Evolution. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2023, Lisbon, Portugal, July 15-19, 2023, Sara Silva and Luís Paquete (Eds.). ACM, 864–872. https://doi.org/10.1145/3583131.3590417
  39. MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and Shifts. arXiv preprint arXiv:2312.11083 (2023).
  40. MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts. In Proceedings of the AutoML Conference (AutoML). PMLR. Available at https://openreview.net/forum?id=uN70Dum6pC2.
  41. Using Affine Combinations of BBOB Problems for Performance Assessment. In Proceedings of Genetic and Evolutionary Computation Conference (GECCO’23), Vol. abs/2303.04573. ACM. https://doi.org/10.1145/3583131.3590412

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now