Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Explainable Benchmarking for Iterative Optimization Heuristics (2401.17842v2)

Published 31 Jan 2024 in cs.NE and cs.AI

Abstract: Benchmarking heuristic algorithms is vital to understand under which conditions and on what kind of problems certain algorithms perform well. In most current research into heuristic optimization algorithms, only a very limited number of scenarios, algorithm configurations and hyper-parameter settings are explored, leading to incomplete and often biased insights and results. This paper presents a novel approach we call explainable benchmarking. Introducing the IOH-Xplainer software framework, for analyzing and understanding the performance of various optimization algorithms and the impact of their different components and hyper-parameters. We showcase the framework in the context of two modular optimization frameworks. Through this framework, we examine the impact of different algorithmic components and configurations, offering insights into their performance across diverse scenarios. We provide a systematic method for evaluating and interpreting the behaviour and efficiency of iterative optimization heuristics in a more transparent and comprehensible manner, allowing for better benchmarking and algorithm design.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. A restart cma evolution strategy with increasing population size. In 2005 IEEE congress on evolutionary computation (2005), vol. 2, IEEE, pp. 1769–1776.
  2. Simplify your covariance matrix adaptation evolution strategy. IEEE Transactions on Evolutionary Computation 21, 5 (2017), 746–759.
  3. Quantifying the impact of boundary constraint handling methods on differential evolution. In Proceedings of the 2021 Genetic and Evolutionary Computation Conference Companion (New York, NY, USA, July 2021), GECCO ’21 Companion, Association for Computing Machinery, pp. 1199––1207.
  4. Evolutionary algorithms for parameter optimization—thirty years later. Evolutionary Computation 31, 2 (06 2023), 81–122.
  5. PSO-X: A component-based framework for the automatic design of particle swarm optimization algorithms. IEEE Trans. Evol. Comput. 26, 3 (2022), 402–416.
  6. A study on rotation invariance in differential evolution. Swarm and Evolutionary Computation 50 (2019), 100436.
  7. Tuning as a means of assessing the benefits of new ideas in interplay with existing algorithmic modules. In Proc. of Genetic and Evolutionary Computation Conference (GECCO’21, Companion material) (2021), ACM, pp. 1375–1384.
  8. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation 6, 2 (2002), 182–197.
  9. Recent advances in selection hyper-heuristics. European Journal of Operational Research 285, 2 (2020), 405–428.
  10. Paradiseo: from a modular framework for evolutionary computation to the automated design of metaheuristics: 22 years of paradiseo. In GECCO ’21: Genetic and Evolutionary Computation Conference, Companion Volume, Lille, France, July 10-14, 2021 (2021), K. Krawiec, Ed., ACM, pp. 1522–1530.
  11. Comparison of three versions of whale optimization algorithm (woa) on the bbob test suite. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation (New York, NY, USA, 2023), GECCO ’23 Companion, Association for Computing Machinery, p. 1595–1602.
  12. What’s inside the black-box? a genetic programming method for interpreting complex machine learning models. In Proceedings of the genetic and evolutionary computation conference (2019), pp. 1012–1020.
  13. Applying genetic programming to improve interpretability in machine learning models. In 2020 IEEE congress on evolutionary computation (CEC) (2020), IEEE, pp. 1–8.
  14. Covariance matrix adaptation map-annealing. In Proceedings of the Genetic and Evolutionary Computation Conference (New York, NY, USA, 2023), GECCO ’23, Association for Computing Machinery, p. 456–465.
  15. Towards explainable metaheuristics: Pca for trajectory mining in evolutionary algorithms. In Artificial Intelligence XXXVIII (Cham, 2021), M. Bramer and R. Ellis, Eds., Springer International Publishing, pp. 89–102.
  16. Multi-granularity competition-cooperation optimization algorithm with adaptive parameter configuration. Applied Intelligence 52 (2022), 13132–13161.
  17. Short-term combined economic and emission hydrothermal optimization by surrogate differential evolution. Applied Energy 141 (2015), 42–56.
  18. Hansen, N. Benchmarking a bi-population cma-es on the bbob-2009 function testbed. In Proceedings of the 11th annual conference companion on genetic and evolutionary computation conference: late breaking papers (2009), pp. 2389–2396.
  19. Anytime performance assessment in blackbox optimization benchmarking. IEEE Transactions on Evolutionary Computation 26, 6 (2022), 1293–1305.
  20. Coco: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software 36, 1 (2021), 114–144.
  21. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Tech. Rep. RR-6829, INRIA, 2009.
  22. Impacts of invariance in search: When cma-es and pso face ill-conditioned and non-separable problems. Applied Soft Computing 11, 8 (2011), 5755–5769.
  23. An efficient approach for assessing hyperparameter importance. In International conference on machine learning (2014), PMLR, pp. 754–762.
  24. Using genetic algorithms to solve np-complete problems. In Proceedings of the 3rd International Conference on Genetic Algorithms, George Mason University, Fairfax, Virginia, USA, June 1989 (1989), J. D. Schaffer, Ed., Morgan Kaufmann, pp. 124–132.
  25. Water evaporation optimization: A novel physically inspired optimization algorithm. Computers & Structures 167 (2016), 69–85.
  26. Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. Evolutionary computation 27, 1 (2019), 99–127.
  27. Proposal of benchmark problem based on real-world car structure design optimization. In Proceedings of the Genetic and Evolutionary Computation Conference Companion (New York, NY, USA, 2018), GECCO ’18, Association for Computing Machinery, p. 183–184.
  28. Structural bias in population-based algorithms. Information Sciences 298 (2015), 468–490.
  29. The importance of being constrained: dealing with infeasible solutions in differential evolution and beyond. Evolutionary Computation (11 2023), 1–46.
  30. The importance of landscape features for performance prediction of modular cma-es variants. In Proceedings of the Genetic and Evolutionary Computation Conference (2022), pp. 648–656.
  31. Using knowledge graphs for performance prediction of modular optimization algorithms. In Applications of Evolutionary Computation - 26th European Conference, EvoApplications 2023, Held as Part of EvoStar 2023, Brno, Czech Republic, April 12-14, 2023, Proceedings (2023), J. Correia, S. L. Smith, and R. Qaddoura, Eds., vol. 13989 of Lecture Notes in Computer Science, Springer, pp. 253–268.
  32. Unveiling the role of modules: Assessing importance and classifying modules in modcma-es and modde via algorithmic behavior analysis.
  33. Genetic programming for evolving a front of interpretable models for data visualization. IEEE transactions on cybernetics 51, 11 (2020), 5468–5482.
  34. Smac3: A versatile bayesian optimization package for hyperparameter optimization. The Journal of Machine Learning Research 23, 1 (2022), 2475–2483.
  35. Boah: A tool suite for multi-fidelity bayesian optimization & analysis of hyperparameters. arXiv:1908.06756 [cs.LG].
  36. Learning the characteristics of engineering optimization problems with applications in automotive crash. In Proceedings of the Genetic and Evolutionary Computation Conference (2022), pp. 1227–1236.
  37. Bbob instance analysis: Landscape properties and algorithm performance across problem instances. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar) (2023), Springer, pp. 380–395.
  38. Automatic configuration of multi-objective aco algorithms. In International Conference on Swarm Intelligence (2010), Springer, pp. 95–106.
  39. The automatic design of multiobjective ant colony optimization algorithms. IEEE Transactions on Evolutionary Computation 16, 6 (2012), 861–875.
  40. From local explanations to global understanding with explainable ai for trees. Nature Machine Intelligence 2, 1 (2020), 2522–5839.
  41. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc., 2017, pp. 4765–4774.
  42. Malan, K. M. A Survey of Advances in Landscape Analysis for Optimisation. Algorithms 14, 2 (Jan. 2021), 40.
  43. Exploratory landscape analysis. In Proceedings of the 13th annual conference on Genetic and evolutionary computation (2011), pp. 829–836.
  44. Patterns of convergence and bound constraint violation in differential evolution on sbox-cost benchmarking suite. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation (2023), GECCO ’23 Companion, Association for Computing Machinery, p. 2337–2345.
  45. Algorithm selection for black-box continuous optimization problems: A survey on methods and challenges. Information Sciences 317 (2015), 224–245.
  46. Search trajectory networks: A tool for analysing and visualising the behaviour of metaheuristics. Applied Soft Computing 109 (2021), 107492.
  47. Coyote optimization algorithm: A new metaheuristic for global optimization problems. In 2018 IEEE Congress on Evolutionary Computation (CEC) (2018).
  48. Per-instance configuration of the modularized cma-es by means of classifier chains and exploratory landscape analysis. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI) (2020), IEEE, pp. 996–1003.
  49. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad, 2018.
  50. ” why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (2016), pp. 1135–1144.
  51. A survey of methods for automated algorithm configuration. Journal of Artificial Intelligence Research 75 (2022), 425–487.
  52. Sobol’, I. Sensitivity estimates for nonlinear mathematical models. Math. Model. Comput. Exp. 1 (1993), 407.
  53. A comparison of global sensitivity analysis methods for explainable ai with an application in genomic prediction. IEEE Access 10 (2022), 103364–103381.
  54. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing 89 (2020), 106078.
  55. Optimizing cma-es with cma-es. In Proceedings of the 15th International Joint Conference on Computational Intelligence - ECTA (2023), INSTICC, SciTePress, pp. 214–221.
  56. Explainable landscape-aware optimization performance prediction. In Symposium Series on Computational Intelligence (New York, NY, USA, 2021), IEEE, pp. 01–08.
  57. Explainable landscape analysis in automated algorithm performance prediction, 2022.
  58. Improving nevergrad’s algorithm selection wizard ngopt through automated algorithm configuration. In Parallel Problem Solving from Nature - PPSN XVII - 17th International Conference, PPSN 2022, Dortmund, Germany, September 10-14, 2022, Proceedings, Part I (2022), G. Rudolph, A. V. Kononova, H. E. Aguirre, P. Kerschke, G. Ochoa, and T. Tusar, Eds., vol. 13398 of Lecture Notes in Computer Science, Springer, pp. 18–31.
  59. Improving nevergrad’s algorithm selection wizard ngopt through automated algorithm configuration. In Parallel Problem Solving from Nature – PPSN XVII (Cham, 2022), G. Rudolph, A. V. Kononova, H. Aguirre, P. Kerschke, G. Ochoa, and T. Tušar, Eds., Springer International Publishing, pp. 18–31.
  60. Identifying Properties of Real-World Optimisation Problems Through a Questionnaire. Springer International Publishing, Cham, 2023, pp. 59–80.
  61. Doe2vec: Deep-learning based features for exploratory landscape analysis. arXiv preprint arXiv:2304.01219 (2023).
  62. Gsareport: Easy to use global sensitivity reporting. Journal of Open Source Software 7, 78 (2022), 4721.
  63. Deep bias: Detecting structural bias using explainable ai. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation (New York, NY, USA, 2023), GECCO ’23 Companion, Association for Computing Machinery, p. 455–458.
  64. Ioh-xplainer: v0.9 arxiv release, Jan. 2024.
  65. Modular differential evolution. In Proceedings of the Genetic and Evolutionary Computation Conference (New York, NY, USA, 2023), GECCO ’23, Association for Computing Machinery, p. 864–872.
  66. Bias: A toolbox for benchmarking structural bias in the continuous domain. IEEE Transactions on Evolutionary Computation 26, 6 (2022), 1380–1393.
  67. Iohanalyzer: Detailed performance analyses for iterative optimization heuristics. ACM Transactions on Evolutionary Learning and Optimization 2, 1 (2022), 1–29.
  68. No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1 (1997), 67–82.
  69. Problem definitions and evaluation criteria for the cec 2017 competition on constrained real-parameter optimization. National University of Defense Technology, Changsha, Hunan, PR China and Kyungpook National University, Daegu, South Korea and Nanyang Technological University, Singapore, Technical Report (2017).
  70. Success history applied to expert system for underwater glider path planning using differential evolution. Expert Systems with Applications 119 (2019), 155–170.
  71. Evolutionary Approaches to Explainable Machine Learning. Springer Nature Singapore, Singapore, 2024, pp. 487–506.
Citations (9)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now