Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

R2 Indicator and Deep Reinforcement Learning Enhanced Adaptive Multi-Objective Evolutionary Algorithm (2404.08161v2)

Published 11 Apr 2024 in cs.NE and cs.AI

Abstract: Choosing an appropriate optimization algorithm is essential to achieving success in optimization challenges. Here we present a new evolutionary algorithm structure that utilizes a reinforcement learning-based agent aimed at addressing these issues. The agent employs a double deep q-network to choose a specific evolutionary operator based on feedback it receives from the environment during optimization. The algorithm's structure contains five single-objective evolutionary algorithm operators. This single-objective structure is transformed into a multi-objective one using the R2 indicator. This indicator serves two purposes within our structure: first, it renders the algorithm multi-objective, and second, provides a means to evaluate each algorithm's performance in each generation to facilitate constructing the reinforcement learning-based reward function. The proposed R2-reinforcement learning multi-objective evolutionary algorithm (R2-RLMOEA) is compared with six other multi-objective algorithms that are based on R2 indicators. These six algorithms include the operators used in R2-RLMOEA as well as an R2 indicator-based algorithm that randomly selects operators during optimization. We benchmark performance using the CEC09 functions, with performance measured by inverted generational distance and spacing. The R2-RLMOEA algorithm outperforms all other algorithms with strong statistical significance (p<0.001) when compared with the average spacing metric across all ten benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. A. Mukhopadhyay, U. Maulik, S. Bandyopadhyay, and C. A. C. Coello, “A survey of multiobjective evolutionary algorithms for data mining: Part i,” IEEE Transactions on Evolutionary Computation, vol. 18, no. 1, pp. 4–19, 2013.
  2. E. Zitzler and S. Künzli, “Indicator-based selection in multiobjective search,” in International conference on parallel problem solving from nature, pp. 832–842, Springer, 2004.
  3. D. Brockhoff, T. Wagner, and H. Trautmann, “On the properties of the R2 indicator,” in Proceedings of the 14th annual conference on genetic and evolutionary computation, pp. 465–472, 2012.
  4. O. Schutze, X. Esquivel, A. Lara, and C. A. C. Coello, “Using the averaged Hausdorff distance as a performance measure in evolutionary multiobjective optimization,” IEEE Transactions on Evolutionary Computation, vol. 16, no. 4, pp. 504–522, 2012.
  5. J. R. Schott, “Fault tolerant design using single and multicriteria genetic algorithm optimization,” tech. rep., 1995.
  6. C. A. C. Coello and M. R. Sierra, “A study of the parallelization of a coevolutionary multi-objective evolutionary algorithm,” in Mexican international conference on artificial intelligence, pp. 688–697, Springer, 2004.
  7. M. G. P. de Lacerda, L. F. de Araujo Pessoa, F. B. de Lima Neto, T. B. Ludermir, and H. Kuchen, “A systematic literature review on general parameter control for evolutionary and swarm-based algorithms,” Swarm and Evolutionary Computation, vol. 60, p. 100777, 2021.
  8. G. Karafotias, M. Hoogendoorn, and Á. E. Eiben, “Parameter control in evolutionary algorithms: Trends and challenges,” IEEE Transactions on Evolutionary Computation, vol. 19, no. 2, pp. 167–187, 2014.
  9. A. Aleti and I. Moser, “Predictive parameter control,” in Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 561–568, 2011.
  10. A. Aleti, I. Moser, I. Meedeniya, and L. Grunske, “Choosing the appropriate forecasting model for predictive parameter control,” Evolutionary Computation, vol. 22, no. 2, pp. 319–349, 2014.
  11. R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT Press, 2018.
  12. M. M. Drugan, “Reinforcement learning versus evolutionary computation: A survey on hybrid algorithms,” Swarm and Evolutionary Computation, vol. 44, pp. 228–246, 2019.
  13. J. E. Pettinger and R. M. Everson, “Controlling genetic algorithms with reinforcement learning,” in Proceedings of the 4th annual conference on genetic and evolutionary computation, pp. 692–692, 2002.
  14. F. Chen, Y. Gao, Z.-q. Chen, and S.-f. Chen, “SCGA: Controlling genetic algorithms with sarsa (0),” in International Conference on Computational Intelligence for Modelling, Control and Automation and International Conference on Intelligent Agents, Web Technologies and Internet Commerce (CIMCA-IAWTIC’06), vol. 1, pp. 1177–1183, IEEE, 2005.
  15. A. Eiben, M. Horvath, W. Kowalczyk, and M. C. Schut, “Reinforcement learning for online control of evolutionary algorithms,” in ESOA, pp. 151–160, Springer, 2006.
  16. K. M. Sallam, S. M. Elsayed, R. K. Chakrabortty, and M. J. Ryan, “Evolutionary framework with reinforcement learning-based mutation adaptation,” IEEE Access, vol. 8, pp. 194045–194071, 2020.
  17. W. Ning, B. Guo, X. Guo, C. Li, and Y. Yan, “Reinforcement learning aided parameter control in multi-objective evolutionary algorithm based on decomposition,” Progress in Artificial Intelligence, vol. 7, pp. 385–398, 2018.
  18. T. Visutarrom, T.-C. Chiang, A. Konak, and S. Kulturel-Konak, “Reinforcement learning-based differential evolution for solving economic dispatch problems,” in 2020 IEEE International Conference on Industrial Engineering and Engineering Management (IEEM), pp. 913–917, IEEE, 2020.
  19. Z. Tan, Y. Tang, K. Li, H. Huang, and S. Luo, “Differential evolution with hybrid parameters and mutation strategies based on reinforcement learning,” Swarm and Evolutionary Computation, vol. 75, p. 101194, 2022.
  20. MIT press, 1992.
  21. H.-G. Beyer and H.-P. Schwefel, “Evolution strategies–a comprehensive introduction,” Natural computing, vol. 1, pp. 3–52, 2002.
  22. R. V. Rao, V. J. Savsani, and D. Vakharia, “Teaching–learning-based optimization: A novel method for constrained mechanical design optimization problems,” Computer-aided design, vol. 43, no. 3, pp. 303–315, 2011.
  23. S. Mirjalili and A. Lewis, “The whale optimization algorithm,” Advances in engineering software, vol. 95, pp. 51–67, 2016.
  24. A. Faramarzi, M. Heidarinejad, B. Stephens, and S. Mirjalili, “Equilibrium optimizer: A novel optimization algorithm,” Knowledge-Based Systems, vol. 191, p. 105190, 2020.
  25. M. A. Al-Betar, I. Abu Doush, S. N. Makhadmeh, G. Al-Naymat, O. A. Alomari, and M. A. Awadallah, “Equilibrium optimizer: a comprehensive survey,” Multimedia Tools and Applications, pp. 1–50, 2023.
  26. N. Rana, M. S. A. Latiff, S. M. Abdulhamid, and H. Chiroma, “Whale optimization algorithm: a systematic review of contemporary applications, modifications and developments,” Neural Computing and Applications, vol. 32, pp. 16245–16277, 2020.
  27. F. Tahernezhad-Javazm, D. Rankin, and D. Coyle, “R2-HMEWO: Hybrid multi-objective evolutionary algorithm based on the Equilibrium Optimizer and Whale Optimization Algorithm,” in 2022 IEEE Congress on Evolutionary Computation (CEC), pp. 1–8, IEEE, 2022.
  28. N. Hansen and A. Ostermeier, “Completely derandomized self-adaptation in evolution strategies,” Evolutionary computation, vol. 9, no. 2, pp. 159–195, 2001.
  29. N. Hansen, D. V. Arnold, and A. Auger, “Evolution strategies,” Springer handbook of computational intelligence, pp. 871–898, 2015.
  30. F. Zou, D. Chen, and Q. Xu, “A survey of teaching–learning-based optimization,” Neurocomputing, vol. 335, pp. 366–383, 2019.
  31. F. Tahernezhad-Javazm, D. Rankin, and D. Coyle, “A Hybrid Multi-Objective Teaching Learning-Based Optimization Using Reference Points and R2 Indicator,” in 2022 6th International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, pp. 19–23, 2022.
  32. F. Tahernezhad-Javazm, V. Azimirad, and M. Shoaran, “A review and experimental study on the application of classifiers and evolutionary algorithms in EEG-based brain–machine interface systems,” Journal of neural engineering, vol. 15, no. 2, p. 021007, 2018.
  33. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.
  34. Q. Zhang and H. Li, “MOEA/D: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on Evolutionary Computation, vol. 11, no. 6, pp. 712–731, 2007.
  35. K. Li, A. Fialho, S. Kwong, and Q. Zhang, “Adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on Evolutionary Computation, vol. 18, no. 1, pp. 114–130, 2013.
  36. E. Zitzler and L. Thiele, “Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach,” IEEE Transactions on Evolutionary Computation, vol. 3, no. 4, pp. 257–271, 1999.
  37. K. M. Malan and A. P. Engelbrecht, “A survey of techniques for characterising fitness landscapes and some possible ways forward,” Information Sciences, vol. 241, pp. 148–163, 2013.
  38. Y. Huang, W. Li, F. Tian, and X. Meng, “A fitness landscape ruggedness multiobjective differential evolution algorithm with a reinforcement learning strategy,” Applied Soft Computing, vol. 96, p. 106693, 2020.
  39. J. Brest, M. S. Maučec, and B. Bošković, “iL-SHADE: Improved L-SHADE algorithm for single objective real-parameter optimization,” in 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 1188–1195, IEEE, 2016.
  40. J. Brest, M. S. Maučec, and B. Bošković, “Single objective real-parameter optimization: Algorithm jSO,” in 2017 IEEE Congress on Evolutionary Computation (CEC), pp. 1311–1318, IEEE, 2017.
  41. K. V. Price, “Eliminating drift bias from the differential evolution algorithm,” Advances in differential evolution, pp. 33–88, 2008.
  42. A. K. Qin and P. N. Suganthan, “Self-adaptive differential evolution algorithm for numerical optimization,” in 2005 IEEE congress on evolutionary computation, vol. 2, pp. 1785–1791, IEEE, 2005.
  43. J. Ilonen, J.-K. Kamarainen, and J. Lampinen, “Differential evolution training algorithm for feed-forward neural networks,” Neural Processing Letters, vol. 17, pp. 93–105, 2003.
  44. J. Sun, X. Liu, T. Bäck, and Z. Xu, “Learning adaptive differential evolution algorithm from optimization experiences by policy gradient,” IEEE Transactions on Evolutionary Computation, vol. 25, no. 4, pp. 666–680, 2021.
  45. G. Zhang and Y. Shi, “Hybrid sampling evolution strategy for solving single objective bound constrained problems,” in 2018 IEEE Congress on Evolutionary Computation (CEC), pp. 1–7, IEEE, 2018.
  46. L. Tao, Y. Dong, W. Chen, Y. Yang, L. Su, Q. Guo, and G. Wang, “A differential evolution with reinforcement learning for multi-objective assembly line feeding problem,” Computers & Industrial Engineering, vol. 174, p. 108714, 2022.
  47. K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints,” IEEE Transactions on Evolutionary Computation, vol. 18, no. 4, pp. 577–601, 2013.
  48. A. J. Nebro, J. J. Durillo, J. Garcia-Nieto, C. C. Coello, F. Luna, and E. Alba, “SMPSO: A new PSO-based metaheuristic for multi-objective optimization,” in 2009 IEEE Symposium on computational intelligence in multi-criteria decision-making (MCDM), pp. 66–73, IEEE, 2009.
  49. M. Tessari and G. Iacca, “Reinforcement learning based adaptive metaheuristics,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1854–1861, 2022.
  50. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.
  51. G. Shala, A. Biedenkapp, N. Awad, S. Adriaensen, M. Lindauer, and F. Hutter, “Learning step-size adaptation in CMA-ES,” in Parallel Problem Solving from Nature–PPSN XVI: 16th International Conference, PPSN 2020, Leiden, The Netherlands, September 5-9, 2020, Proceedings, Part I 16, pp. 691–706, Springer, 2020.
  52. M. G. P. de Lacerda, F. B. de Lima Neto, H. d. A. A. Neto, H. Kuchen, and T. B. Ludermir, “On the learning properties of dueling DDQN in parameter control for evolutionary and swarm-based algorithms,” in 2019 IEEE Latin American Conference on Computational Intelligence (LA-CCI), pp. 1–6, IEEE, 2019.
  53. J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceedings of ICNN’95-international conference on neural networks, vol. 4, pp. 1942–1948, IEEE, 1995.
  54. Y. Sakurai, K. Takada, T. Kawabe, and S. Tsuruta, “A method to control parameters of evolutionary algorithms by using reinforcement learning,” in 2010 sixth international conference on signal-image technology and internet based systems, pp. 74–79, IEEE, 2010.
  55. A. Dantas and A. Pozo, “Online Selection of Heuristic Operators with Deep Q-Network: A Study on the HyFlex Framework,” in Intelligent Systems: 10th Brazilian Conference, BRACIS 2021, Virtual Event, November 29–December 3, 2021, Proceedings, Part I 10, pp. 280–294, Springer, 2021.
  56. L. Gambardella and A.-Q. DM, “A Reinforcement Learning Approach to the Traveling Salesman Problem,” in of ML-95 Twelfth Intern Conf on Machining, pp. 252–260, 1994.
  57. S. D. Handoko, D. T. Nguyen, Z. Yuan, and H. C. Lau, “Reinforcement learning for adaptive operator selection in memetic search applied to quadratic assignment problem,” in Proceedings of the companion publication of the 2014 annual conference on genetic and evolutionary computation, pp. 193–194, 2014.
  58. G. Karafotias, M. Hoogendoorn, and B. Weel, “Comparing generic parameter controllers for EAs,” in 2014 IEEE Symposium on Foundations of Computational Intelligence (FOCI), pp. 46–53, IEEE, 2014.
  59. E. Alba and B. Dorronsoro, “Introduction to cellular genetic algorithms,” in Cellular Genetic Algorithms, pp. 3–20, Springer, 2008.
  60. S. M. Elsayed, R. A. Sarker, and D. L. Essam, “GA with a new multi-parent crossover for solving IEEE-CEC2011 competition problems,” in 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 1034–1040, IEEE, 2011.
  61. T. Liao and T. Stützle, “Bounding the population size of IPOP-CMA-ES on the noiseless BBOB testbed,” in Proceedings of the 15th annual conference companion on Genetic and evolutionary computation, pp. 1161–1168, 2013.
  62. E. Emary, H. M. Zawbaa, and C. Grosan, “Experienced gray wolf optimization through reinforcement learning and neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 3, pp. 681–694, 2017.
  63. S. Mirjalili, S. M. Mirjalili, and A. Lewis, “Grey wolf optimizer,” Advances in engineering software, vol. 69, pp. 46–61, 2014.
  64. A. K. Sadhu, A. Konar, T. Bhattacharjee, and S. Das, “Synergism of firefly algorithm and Q-learning for robot arm path planning,” Swarm and Evolutionary Computation, vol. 43, pp. 50–68, 2018.
  65. X.-S. Yang, Nature-inspired metaheuristic algorithms. Luniver press, 2010.
  66. M. Sharma, A. Komninos, M. López-Ibáñez, and D. Kazakov, “Deep reinforcement learning based parameter control in differential evolution,” in Proceedings of the Genetic and Evolutionary Computation Conference, pp. 709–717, 2019.
  67. Z. Li, L. Shi, C. Yue, Z. Shang, and B. Qu, “Differential evolution based on reinforcement learning with fitness ranking for solving multimodal multiobjective problems,” Swarm and Evolutionary Computation, vol. 49, pp. 234–244, 2019.
  68. F. Zou, G. G. Yen, L. Tang, and C. Wang, “A reinforcement learning approach for dynamic multi-objective optimization,” Information Sciences, vol. 546, pp. 815–834, 2021.
  69. M. P. Hansen and A. Jaszkiewicz, Evaluating the quality of approximations to the non-dominated set. Citeseer, 1994.
  70. J. G. Falcón-Cardona and C. A. C. Coello, “Indicator-based multi-objective evolutionary algorithms: A comprehensive survey,” ACM Computing Surveys (CSUR), vol. 53, no. 2, pp. 1–35, 2020.
  71. R. Hernández Gómez and C. A. Coello Coello, “Improved metaheuristic based on the R2 indicator for many-objective optimization,” in Proceedings of the 2015 annual conference on genetic and evolutionary computation, pp. 679–686, 2015.
  72. B. Jang, M. Kim, G. Harerimana, and J. W. Kim, “Q-learning algorithms: A comprehensive classification and applications,” IEEE access, vol. 7, pp. 133653–133667, 2019.
  73. V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, 2013.
  74. L.-J. Lin, “Self-improving reactive agents based on reinforcement learning, planning and teaching,” Machine learning, vol. 8, pp. 293–321, 1992.
  75. H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 30, 2016.
  76. W. Yang, L. Chen, Y. Wang, and M. Zhang, “Multi/Many-Objective Particle Swarm Optimization Algorithm Based on Competition Mechanism,” Computational intelligence and neuroscience, vol. 2020, 2020.
  77. C. A. C. Coello and N. C. Cortés, “Solving multiobjective optimization problems using an artificial immune system,” Genetic programming and evolvable machines, vol. 6, pp. 163–190, 2005.
  78. D. Hadka, “Platypus-multiobjective optimization in python,” Revision 0552483b, 2015.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Farajollah Tahernezhad-Javazm (2 papers)
  2. Naomi Du Bois (1 paper)
  3. Alice E. Smith (1 paper)
  4. Damien Coyle (6 papers)
  5. Debbie Rankin (2 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now