Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (2401.02051v3)

Published 4 Jan 2024 in cs.NE and cs.AI

Abstract: Heuristics are widely used for dealing with complex search and optimization problems. However, manual design of heuristics can be often very labour extensive and requires rich working experience and knowledge. This paper proposes Evolution of Heuristic (EoH), a novel evolutionary paradigm that leverages both LLMs and Evolutionary Computation (EC) methods for Automatic Heuristic Design (AHD). EoH represents the ideas of heuristics in natural language, termed thoughts. They are then translated into executable codes by LLMs. The evolution of both thoughts and codes in an evolutionary search framework makes it very effective and efficient for generating high-performance heuristics. Experiments on three widely studied combinatorial optimization benchmark problems demonstrate that EoH outperforms commonly used handcrafted heuristics and other recent AHD methods including FunSearch. Particularly, the heuristic produced by EoH with a low computational budget (in terms of the number of queries to LLMs) significantly outperforms widely-used human hand-crafted baseline algorithms for the online bin packing problem.

PDF HTML Abstract

Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using LLMs

The paper examines the novel approach termed Evolution of Heuristics (EoH), which synergistically combines LLMs and Evolutionary Computation (EC) to advance Automatic Heuristic Design (AHD). EoH aims to reduce the complexity and computational cost associated with heuristic development for combinatorial optimization problems. This method is particularly pertinent given the tedious nature of manual heuristic design, which necessitates considerable human intuition and experience.

EoH introduces a paradigm where heuristic ideas, or "thoughts," are represented through linguistic descriptions generated by LLMs, then translated into executable code. The coevolution of these thoughts and codes offers a dynamic framework that surpasses existing AHD methods. Five evolution operators are integrated into EoH to efficiently search for novel heuristics, optimizing the heuristic development process akin to human designers.

The paper presents comprehensive experimental evaluations on three benchmarks: the online bin packing problem, the traveling salesman problem (TSP), and the flow shop scheduling problem (FSSP). Results showcase EoH's superior ability to design competitive heuristics using approximately 0.1% of the computational budget required by previous methods. These heuristics outperform many traditional metaheuristics, achieving optimal or near-optimal solutions, particularly noted by a 0% gap for TSP20 and TSP50 and a 0.23% gap for FSSP instances.

Further analysis compares hand-crafted heuristics, such as first-fit and best-fit, and results from concurrent frameworks like FunSearch, underscoring EoH’s efficiency. Notably, EoH identifies superior heuristics on the online bin packing problem with significantly fewer LLM queries compared to FunSearch.

The implications of EoH extend beyond practical algorithm design efficiency, offering a robust framework adaptable across diverse domains without necessitating extensive computational resources or domain expertise. The theoretical underpinnings of EoH suggest possible refinements in heuristic research and evolution, fostering advancements in AI-driven optimization tasks.

In conclusion, EoH demonstrates a promising shift in automated algorithm design, fostering reproducibility and accessibility by publishing its source code. The approach not only mitigates the challenges associated with heuristic design but extends the possibilities for AI's role in efficiently solving complex optimization problems. Future developments may focus on refining prompt strategies and exploring broader applications of EoH across different optimization paradigms, thus contributing significantly to the evolution of metaheuristic methodologies.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  2623–2631, 2019.
  2. Guided local search., 2018.
  3. A gender-based genetic algorithm for the automatic configuration of algorithms. In International Conference on Principles and Practice of Constraint Programming, pp.  142–157. Springer, 2009.
  4. Concorde tsp solver, 2006.
  5. Knowledge-guided local search for the vehicle routing problem. Computers & Operations Research, 105:32–46, 2019.
  6. Guided local search with shifting bottleneck for job shop scheduling. Management science, 44(2):262–275, 1998.
  7. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016.
  8. Machine learning for combinatorial optimization: a methodological tour d’horizon. European Journal of Operational Research, 290(2):405–421, 2021.
  9. A guided local search heuristic for the capacitated arc routing problem. European Journal of Operational Research, 147(3):629–643, 2003.
  10. Chip-chat: Challenges and opportunities in conversational hardware design. arXiv preprint arXiv:2305.13243, 2023.
  11. Mo-paramils: A multi-objective automatic algorithm configuration framework. In Learning and Intelligent Optimization: 10th International Conference, LION 10, Ischia, Italy, May 29–June 1, 2016, Revised Selected Papers 10, pp.  32–47. Springer, 2016.
  12. Enhancing genetic improvement mutations using large language models. arXiv preprint arXiv:2310.19813, 2023.
  13. Hyper-heuristics: A survey of the state of the art. Journal of the Operational Research Society, 64:1695–1724, 2013.
  14. A classification of hyper-heuristic approaches: revisited. Handbook of metaheuristics, pp.  453–477, 2019.
  15. Evoprompting: Language models for code-level neural architecture search. arXiv preprint arXiv:2302.14838, 2023.
  16. Learning to optimize: A primer and a benchmark. The Journal of Machine Learning Research, 23(1):8562–8620, 2022.
  17. Exploring the potential of gpt-4 in biomedical engineering: the dawn of a new era. Annals of Biomedical Engineering, pp.  1–9, 2023.
  18. Croes, G. A. A method for solving traveling-salesman problems. Operations research, 6(6):791–812, 1958.
  19. Deb, K. Optimization for engineering design: Algorithms and examples. PHI Learning Pvt. Ltd., 2012.
  20. Learning heuristics for the tsp by policy gradient. In International conference on the integration of constraint programming, artificial intelligence, and operations research, pp.  170–181. Springer, 2018.
  21. A supervised learning and control method to improve particle swarm optimization algorithms. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(7):1135–1148, 2016.
  22. Bq-nco: Bisimulation quotienting for generalizable neural combinatorial optimization. arXiv preprint arXiv:2301.03313, 2023.
  23. Guided local search for the three-dimensional bin-packing problem. Informs journal on computing, 15(3):267–283, 2003.
  24. Generalize a small pre-trained model to arbitrarily large tsp instances. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp.  7474–7482, 2021.
  25. Handbook of metaheuristics, volume 57. Springer Science & Business Media, 2006.
  26. Towards optimizing with large language models. arXiv preprint arXiv:2310.05204, 2023.
  27. Automl: A survey of the state-of-the-art. Knowledge-Based Systems, 212:106622, 2021.
  28. Chateda: A large language model powered autonomous agent for eda. arXiv preprint arXiv:2308.10204, 2023.
  29. Helsgaun, K. An extension of the lin-kernighan-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems. Roskilde: Roskilde University, 12, 2017.
  30. Graph neural network guided local search for the traveling salesperson problem. arXiv preprint arXiv:2110.05291, 2021.
  31. Paramils: an automatic algorithm configuration framework. Journal of artificial intelligence research, 36:267–306, 2009.
  32. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5, pp.  507–523. Springer, 2011.
  33. Is gpt-3 all you need for low-data discovery in chemistry? 2023.
  34. Llm performance predictors are good initializers for architecture search. arXiv preprint arXiv:2310.16712, 2023.
  35. Learning heuristics with different representations for stochastic routing. IEEE Transactions on Cybernetics, 2022.
  36. An efficient graph convolutional network technique for the travelling salesman problem. arXiv preprint arXiv:1906.01227, 2019.
  37. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
  38. Attention, learn to solve routing problems! arXiv preprint arXiv:1803.08475, 2018.
  39. Deep policy dynamic programming for vehicle routing problems. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 19th International Conference, CPAIOR 2022, Los Angeles, CA, USA, June 20-23, 2022, Proceedings, pp.  190–213. Springer, 2022.
  40. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances in Neural Information Processing Systems, 33:21188–21198, 2020.
  41. Matrix encoding networks for neural combinatorial optimization. In Advances in Neural Information Processing Systems, 2021.
  42. Discovering evolution strategies via meta-black-box optimization. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp.  29–30, 2023.
  43. Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine. New England Journal of Medicine, 388(13):1233–1239, 2023.
  44. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp.  331–366. Springer, 2023.
  45. Evolution Through Large Models, pp.  331–366. Springer Nature Singapore, Singapore, 2024.
  46. Survey on evolutionary deep learning: Principles, algorithms, applications, and open issues. ACM Computing Surveys, 56(2):1–34, 2023.
  47. Large language model for multi-objective evolutionary optimization. arXiv preprint arXiv:2310.12541, 2023a.
  48. Algorithm evolution using large language model. arXiv preprint arXiv:2311.15249, 2023b.
  49. Prediction guided meta-learning for multi-objective reinforcement learning. In 2021 IEEE Congress on Evolutionary Computation (CEC), pp.  2171–2178. IEEE, 2021.
  50. Large language models as evolutionary optimizers. arXiv preprint arXiv:2310.19046, 2023c.
  51. Hybridization of evolutionary algorithm and deep reinforcement learning for multi-objective orienteering optimization. IEEE Transactions on Evolutionary Computation, 2022.
  52. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43–58, 2016.
  53. Iterated Local Search: Framework and Applications, pp.  129–168. Springer International Publishing, Cham, 2019.
  54. Neural combinatorial optimization with heavy decoder: Toward large scale generalization. arXiv preprint arXiv:2310.07985, 2023.
  55. Learning to optimize: reference vector reinforcement learning adaption to constrained many-objective optimization of industrial copper burdening system. IEEE Transactions on Cybernetics, 2021.
  56. Explainable artificial intelligence by genetic programming: A survey. IEEE Transactions on Evolutionary Computation, 2022.
  57. Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302.12170, 2023.
  58. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 2021.
  59. Llmatic: Neural architecture search via large language models and quality-diversity optimization. arXiv preprint arXiv:2306.01102, 2023.
  60. Numerical optimization. Springer, 1999.
  61. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023.
  62. Dimes: A differentiable meta solver for combinatorial optimization problems. arXiv preprint arXiv:2210.04123, 2022.
  63. Mathematical discoveries from program search with large language models. Nature, pp.  1–3, 2023.
  64. An analysis of several heuristics for the traveling salesman problem. SIAM journal on computing, 6(3):563–581, 1977.
  65. Adaptive solution prediction for combinatorial optimization. European Journal of Operational Research, 309(3):1392–1408, 2023.
  66. Eb-gls: an improved guided local search based on the big valley structure. Memetic computing, 10:333–350, 2018.
  67. Automated design of metaheuristic algorithms. Handbook of metaheuristics, pp.  541–579, 2019.
  68. Neuralgls: learning to guide local search with graph convolutional network for the traveling salesman problem. Neural Computing and Applications, pp.  1–20, 2023.
  69. Learning to generate columns with application to vertex coloring. In The Eleventh International Conference on Learning Representations, 2022a.
  70. Boosting ant colony optimization via solution prediction and machine learning. Computers & Operations Research, 143:105769, 2022b.
  71. Evolutionary transfer optimization-a new frontier in evolutionary computation research. IEEE Computational Intelligence Magazine, 16(1):22–33, 2021.
  72. Is chatgpt the ultimate programming assistant–how far is it? arXiv preprint arXiv:2304.11938, 2023.
  73. Deep reinforcement learning based adaptive operator selection for evolutionary multi-objective optimization. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.
  74. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  75. Pointer networks. Advances in neural information processing systems, 28, 2015.
  76. Guided local search and its application to the traveling salesman problem. European journal of operational research, 113(2):469–499, 1999.
  77. Guided local search. In Handbook of metaheuristics, pp.  321–361. Springer, 2010.
  78. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023.
  79. Learning improvement heuristics for solving routing problems. IEEE transactions on neural networks and learning systems, 33(9):5057–5069, 2021.
  80. Step-wise deep learning models for solving routing problems. IEEE Transactions on Industrial Informatics, 17(7):4861–4871, 2020.
  81. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023.
  82. Gpt-nas: Neural architecture search with the generative pre-trained model. arXiv preprint arXiv:2305.05351, 2023.
  83. Automl-gpt: Automatic machine learning with gpt. arXiv preprint arXiv:2305.02499, 2023a.
  84. Meta-learning-based deep reinforcement learning for multiobjective optimization problems. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  85. Reinforcement learning-based multiobjective evolutionary algorithm for mixed-model multimanned assembly line balancing under uncertain demand. IEEE Transactions on Cybernetics, 2023b.
  86. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023a.
  87. Large language models as commonsense knowledge for large-scale task planning. arXiv preprint arXiv:2305.14078, 2023b.
  88. Can gpt-4 perform neural architecture search? arXiv preprint arXiv:2304.10970, 2023.
  89. Toward adaptive knowledge transfer in multifactorial evolutionary computation. IEEE transactions on cybernetics, 51(5):2563–2576, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Fei Liu (232 papers)
  2. Xialiang Tong (14 papers)
  3. Mingxuan Yuan (81 papers)
  4. Xi Lin (135 papers)
  5. Fu Luo (8 papers)
  6. Zhenkun Wang (34 papers)
  7. Zhichao Lu (52 papers)
  8. Qingfu Zhang (78 papers)
Citations (31)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - FeiLiu36/EoH: Evolution of Heuristics (74 stars)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets