Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MA-BBOB: Many-Affine Combinations of BBOB Functions for Evaluating AutoML Approaches in Noiseless Numerical Black-Box Optimization Contexts (2306.10627v1)

Published 18 Jun 2023 in cs.LG, cs.NE, and math.OC

Abstract: Extending a recent suggestion to generate new instances for numerical black-box optimization benchmarking by interpolating pairs of the well-established BBOB functions from the COmparing COntinuous Optimizers (COCO) platform, we propose in this work a further generalization that allows multiple affine combinations of the original instances and arbitrarily chosen locations of the global optima. We demonstrate that the MA-BBOB generator can help fill the instance space, while overall patterns in algorithm performance are preserved. By combining the landscape features of the problems with the performance data, we pose the question of whether these features are as useful for algorithm selection as previous studies suggested. MA-BBOB is built on the publicly available IOHprofiler platform, which facilitates standardized experimentation routines, provides access to the interactive IOHanalyzer module for performance analysis and visualization, and enables comparisons with the rich and growing data collection available for the (MA-)BBOB functions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Enhanced instance space analysis for the maximum flow problem. Eur. J. Oper. Res. 304, 2 (2023), 411–428. https://doi.org/10.1016/j.ejor.2022.04.012
  2. Anne Auger and Nikolaus Hansen. 2020. A SIGEVO Impact Award for a Paper Arising from the COCO Platform: A Summary and Beyond. https://evolution.sigevo.org/issues/HTML/sigevolution-13-4/home.html. Issue 3.
  3. Per instance algorithm configuration of CMA-ES with limited budget. In Proc. of Genetic and Evolutionary Computation (GECCO’17). ACM, 681–688. https://doi.org/10.1145/3071178.3071343
  4. Evolving diverse TSP instances by means of novel and creative mutation operators. In Proc. of Conference on Foundations of Genetic Algorithms (FOGA’19), Tobias Friedrich, Carola Doerr, and Dirk V. Arnold (Eds.). ACM, 58–71. https://doi.org/10.1145/3299904.3340307
  5. Jakob Bossek and Markus Wagner. 2021. Generating instances with performance differences for more than just two algorithms. In Proc. of Genetic and Evolutionary Computation Conference (GECCO’21, Companion material), Krzysztof Krawiec (Ed.). ACM, 1423–1432. https://doi.org/10.1145/3449726.3463165
  6. Tuning as a means of assessing the benefits of new ideas in interplay with existing algorithmic modules. In Proc. of Genetic and Evolutionary Computation Conference (GECCO’21, Companion material). ACM, 1375–1384. https://doi.org/10.1145/3449726.3463167
  7. IOHexperimenter: Benchmarking Platform for Iterative Optimization Heuristics. CoRR abs/2111.04077 (2021). arXiv:2111.04077 https://arxiv.org/abs/2111.04077
  8. Konstantin Dietrich and Olaf Mersmann. 2022. Increasing the Diversity of Benchmark Function Sets Through Affine Recombination. In Proc. of Parallel Problem Solving from Nature (PPSN’22) (LNCS, Vol. 13398), Günter Rudolph, Anna V. Kononova, Hernán E. Aguirre, Pascal Kerschke, Gabriela Ochoa, and Tea Tusar (Eds.). Springer, 590–602. https://doi.org/10.1007/978-3-031-14714-2_41
  9. IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics. CoRR abs/1810.05281 (2018). arXiv:1810.05281 http://arxiv.org/abs/1810.05281
  10. COCO: A platform for comparing continuous optimizers in a black-box setting. Optim. Methods Softw. 36, 1 (2021), 114–144.
  11. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633/document
  12. Automated Machine Learning - Methods, Systems, Challenges. Springer. https://doi.org/10.1007/978-3-030-05318-5
  13. FAIR Principles: Interpretations and Implementation Considerations. Data Intell. 2, 1-2 (2020), 10–29. https://doi.org/10.1162/dint_r_00024
  14. Automated Algorithm Selection: Survey and Perspectives. Evol. Comput. 27, 1 (2019), 3–45. https://doi.org/10.1162/evco_a_00242
  15. Per-run Algorithm Selection with Warm-starting using Trajectory-based Features. In Proc. of Parallel Problem Solving from Nature (PPSN’22) (LNCS, Vol. 13398). Springer, 46–60. https://doi.org/10.1007/978-3-031-14714-2_4 Free version available at https://arxiv.org/abs/2204.09483.
  16. OPTION: OPTImization Algorithm Benchmarking ONtology. IEEE Trans. Evol. Comput. (2022). https://doi.org/10.1109/TEVC.2022.3232844 To appear. Free version available at https://arxiv.org/abs/2211.11332.
  17. Benjamin Lacroix and John McCall. 2019. Limitations of Benchmark Sets and Landscape Features for Algorithm Selection and Performance Prediction. In Proc. of Genetic and Evolutionary Computation (GECCO’19) (Prague, Czech Republic). ACM, New York, NY, USA, 261–262. https://doi.org/10.1145/3319619.3322051
  18. Evolving test instances of the Hamiltonian completion problem. Comput. Oper. Res. 149 (2023), 106019. https://doi.org/10.1016/j.cor.2022.106019
  19. Learning the characteristics of engineering optimization problems with applications in automotive crash. In Proc. of Genetic and Evolutionary Computation (GECCO’22), Jonathan E. Fieldsend and Markus Wagner (Eds.). ACM, 1227–1236. https://doi.org/10.1145/3512290.3528712
  20. BBOB Instance Analysis: Landscape Properties and Algorithm Performance across Problem Instances. CoRR abs/2211.16318 (2022). https://doi.org/10.48550/arXiv.2211.16318 arXiv:2211.16318
  21. A Novelty-Search Approach to Filling an Instance-Space with Diverse and Discriminatory Instances for the Knapsack Problem. In Proc. of Parallel Problem Solving from Nature (PPSN’22) (LNCS, Vol. 13398). Springer, 223–236. https://doi.org/10.1007/978-3-031-14714-2_16
  22. Exploratory landscape analysis. In Proc. of Genetic and Evolutionary Computation (GECCO’11). ACM, 829–836.
  23. Mario A. Muñoz and Kate Smith-Miles. 2020. Generating New Space-Filling Test Instances for Continuous Black-Box Optimization. Evol. Comput. 28, 3 (2020), 379–404. https://doi.org/10.1162/evco_a_00262
  24. An Instance Space Analysis of Regression Problems. ACM Trans. Knowl. Discov. Data 15, 2 (2021), 28:1–28:25. https://doi.org/10.1145/3436893
  25. RF+ clust for Leave-One-Problem-Out Performance Prediction. In Proc. of Applications of Evolutionary Computation (Evo Applications’23). Springer, 285–301.
  26. Raphael Patrick Prager. 2022. pFlacco. https://pypi.org/project/pflacco/.
  27. Raphael Patrick Prager and Heike Trautmann. 2023. Nullifying the Inherent Bias of Non-invariant Exploratory Landscape Analysis Features. In Proc. of Applications of Evolutionary Computation (Evo Applications’23). Springer, 411–425.
  28. Jérémy Rapin and Olivier Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad.
  29. Expressiveness and Robustness of Landscape Features. In Proc. of Genetic and Evolutionary Computation (GECCO’19) (Prague, Czech Republic). ACM, 2048–2051. https://doi.org/10.1145/3319619.3326913
  30. Learning Step-Size Adaptation in CMA-ES. In Proc. of Parallel Problem Solving from Nature (PPSN’20) (LNCS, Vol. 12269). Springer, 691–706. https://doi.org/10.1007/978-3-030-58112-1_48
  31. A Recommender System for Metaheuristic Algorithms for Continuous Optimization Based on Deep Recurrent Neural Networks. IEEE Trans. Artif. Intell. 1, 1 (2020), 5–18. https://doi.org/10.1109/TAI.2020.3022339
  32. Modular Differential Evolution. In Proc. of Genetic and Evolutionary Computation (GECCO’23). ACM. https://doi.org/10.1145/3583131.3590417 To appear. Code available at https://github.com/Dvermetten/ModDE.
  33. Reproducibility files and additional figures. Code repository: https://github.com/Dvermetten/Many-affine-BBOB Data and figure repository: https://doi.org/10.5281/zenodo.7826036.
  34. Using Affine Combinations of BBOB Problems for Performance Assessment. In Proc. of Genetic and Evolutionary Computation Conference (GECCO’23), Vol. abs/2303.04573. ACM. https://doi.org/10.1145/3583131.3590412
  35. IOHanalyzer: Detailed Performance Analysis for Iterative Optimization Heuristic. ACM Trans. Evol. Learn. Optim. 2, 1 (2022), 3:1–3:29. https://doi.org/10.1145/3510426 IOHanalyzer is available at CRAN, on GitHub, and as web-based GUI, see https://iohprofiler.github.io/IOHanalyzer/ for links.
  36. Informing Multiobjective Optimization Benchmark Construction Through Instance Space Analysis. IEEE Trans. Evol. Comput. 26, 6 (2022), 1246–1260. https://doi.org/10.1109/TEVC.2022.3205165
  37. Martin Zaefferer and Frederik Rehbach. 2020. Continuous Optimization Benchmarks by Simulation. In Proc. of Parallel Problem Solving from Nature (PPSN’20) (LNCS, Vol. 12269), Thomas Bäck, Mike Preuss, André H. Deutz, Hao Wang, Carola Doerr, Michael T. M. Emmerich, and Heike Trautmann (Eds.). Springer, 273–286. https://doi.org/10.1007/978-3-030-58112-1_19
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now