Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pointer Networks Trained Better via Evolutionary Algorithms (2312.01150v4)

Published 2 Dec 2023 in cs.NE

Abstract: Pointer Network (PtrNet) is a specific neural network for solving Combinatorial Optimization Problems (COPs). While PtrNets offer real-time feed-forward inference for complex COPs instances, its quality of the results tends to be less satisfactory. One possible reason is that such issue suffers from the lack of global search ability of the gradient descent, which is frequently employed in traditional PtrNet training methods including both supervised learning and reinforcement learning. To improve the performance of PtrNet, this paper delves deeply into the advantages of training PtrNet with Evolutionary Algorithms (EAs), which have been widely acknowledged for not easily getting trapped by local optima. Extensive empirical studies based on the Travelling Salesman Problem (TSP) have been conducted. Results demonstrate that PtrNet trained with EA can consistently perform much better inference results than eight state-of-the-art methods on various problem scales. Compared with gradient descent based PtrNet training methods, EA achieves up to 30.21\% improvement in quality of the solution with the same computational time. With this advantage, this paper is able to at the first time report the results of solving 1000-dimensional TSPs by training a PtrNet on the same dimensionality, which strongly suggests that scaling up the training instances is in need to improve the performance of PtrNet on solving higher-dimensional COPs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Pointer Networks. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  2. Stage-Wise Magnitude-Based Pruning for Recurrent Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2022.
  3. Solving Traveling Salesman Problem with Time Windows Using Hybrid Pointer Networks with Time Features. Sustainability, 13(22):12906, January 2021.
  4. Moshe Sniedovich. Dynamic Programming: Foundations and Principles. CRC press, 2010.
  5. P. Festa. A brief introduction to exact, approximation, and heuristic algorithms for solving hard combinatorial optimization problems. In 2014 16th International Conference on Transparent Optical Networks (ICTON), pages 1–20, July 2014.
  6. Energy management in hybrid microgrid with considering multiple power market and real time demand response. Energy, 174:10–23, May 2019.
  7. Joint resource allocation and trajectory design for multi-uav systems with moving users: Pointer network and unfolding. IEEE Transactions on Wireless Communications, 22(5):3310–3323, 2023.
  8. HierRL: Hierarchical Reinforcement Learning for Task Scheduling in Distributed Systems. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8, July 2022.
  9. Chenxu Shi. Pointer Network Solution Pool : Combining Pointer Networks and Heuristics to Solve TSP Problems. In 2022 3rd International Conference on Computer Vision, Image and Deep Learning & International Conference on Computer Engineering and Applications (CVIDL & ICCEA), pages 1236–1242, May 2022.
  10. Deep Reinforcement Learning for the Electric Vehicle Routing Problem With Time Windows. IEEE Transactions on Intelligent Transportation Systems, 23(8):11528–11538, August 2022.
  11. Learning to Perform Local Rewriting for Combinatorial Optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  12. A Pointer Network Based Deep Learning Algorithm for the Max-Cut Problem. In Long Cheng, Andrew Chi Sing Leung, and Seiichi Ozawa, editors, Neural Information Processing, Lecture Notes in Computer Science, pages 238–248, Cham, 2018. Springer International Publishing.
  13. How Good Is Neural Combinatorial Optimization? A Systematic Evaluation on the Traveling Salesman Problem, April 2023.
  14. Keld Helsgaun. An effective implementation of the Lin–Kernighan traveling salesman heuristic. European Journal of Operational Research, 126(1):106–130, October 2000.
  15. Combinatorial optimization by graph pointer networks and hierarchical reinforcement learning. arXiv preprint arXiv:1911.04936, 2019.
  16. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793–2803. PMLR, 2021.
  17. POMO: Policy Optimization with Multiple Optima for Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 33, pages 21188–21198. Curran Associates, Inc., 2020.
  18. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016.
  19. Attention, Learn to Solve Routing Problems!, February 2019.
  20. Heterogeneous Pointer Network for Travelling Officer Problem. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8, July 2022.
  21. Reinforcement Learning for Solving the Vehicle Routing Problem. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  22. A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 33(9):4861–4875, September 2022.
  23. X. Zhao and S. B. Holden. Towards a Competitive 3-Player Mahjong AI using Deep Reinforcement Learning. In 2022 IEEE Conference on Games (CoG), pages 524–527, 21.
  24. Hierarchical Neural Architecture Search for Deep Stereo Matching. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22158–22169. Curran Associates, Inc., 2020.
  25. A Survey on Evolutionary Construction of Deep Neural Networks. IEEE Transactions on Evolutionary Computation, 25(5):894–912, October 2021.
  26. Natural evolution strategies. The Journal of Machine Learning Research, 15(1):949–980, 2014.
  27. Negatively correlated search. IEEE Journal on Selected Areas in Communications, 34(3):542–550, 2016.
  28. Parallel exploration via negatively correlated search. Frontiers of Computer Science, 15(5):155333, July 2021.
  29. A Scalable Approach to Capacitated Arc Routing Problems Based on Hierarchical Decomposition. IEEE Transactions on Cybernetics, 47(11):3928–3940, November 2017.
  30. Evaluating Curriculum Learning Strategies in Neural Combinatorial Optimization, November 2020.
  31. Kenji Kawaguchi. Deep Learning without Poor Local Minima. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  32. “Neural” computation of decisions in optimization problems. Biological Cybernetics, 52(3):141–152, July 1985.
  33. J. Wang. A recurrent neural network for solving the shortest path problem. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 43(6):482–486, June 1996.
  34. A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6):1291–1307, November 2012.
  35. Deep Reinforcement Learning for Multiobjective Optimization. IEEE Transactions on Cybernetics, 51(6):3103–3114, June 2021.
  36. Xin Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423–1447, September 1999.
  37. Search Heuristics for the Optimization of DBN for Time Series Forecasting. In Hitoshi Iba and Nasimul Noman, editors, Deep Neural Evolution: Deep Learning with Evolutionary Computation, pages 131–152. Springer Singapore, Singapore, 2020.
  38. Discovering Gated Recurrent Neural Network Architectures. In Hitoshi Iba and Nasimul Noman, editors, Deep Neural Evolution: Deep Learning with Evolutionary Computation, pages 233–251. Springer Singapore, Singapore, 2020.
  39. Evolution Strategies as a Scalable Alternative to Reinforcement Learning, September 2017.
  40. Evolutionary reinforcement learning via cooperative coevolutionary negatively correlated search. Swarm and Evolutionary Computation, 68:100974, 2022.
  41. Evolutionary job scheduling with optimized population by deep reinforcement learning. Engineering Optimization, 55(3):494–509, March 2023.
  42. Accelerating the Genetic Algorithm for Large-scale Traveling Salesman Problems by Cooperative Coevolutionary Pointer Network with Reinforcement Learning. arXiv preprint arXiv:2209.13077, 2022.
  43. Multi-Objective Neural Evolutionary Algorithm for Combinatorial Optimization Problems. IEEE Transactions on Neural Networks and Learning Systems, 34(4):2133–2143, April 2023.
  44. Evolutionary programming made faster. IEEE Transactions on Evolutionary Computation, 3(2):82–102, July 1999.
  45. T. Kailath. The Divergence and Bhattacharyya Distance Measures in Signal Selection. IEEE Transactions on Communication Technology, 15(1):52–60, February 1967.
  46. Evolution strategies – A comprehensive introduction. Natural Computing, 1(1):3–52, March 2002.
  47. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research, 134:105400, 2021.
  48. Learning Combinatorial Optimization Algorithms over Graphs. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  49. Combining Reinforcement Learning with Lin-Kernighan-Helsgaun Algorithm for the Traveling Salesman Problem. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12445–12452, May 2021.
  50. Local search in combinatorial optimization. Artificial Neural Networks: An Introduction to ANN Theory and Practice, pages 157–174, 2005.
  51. Rafał Skinderowicz. Improving Ant Colony Optimization efficiency for solving large TSP instances. Applied Soft Computing, 120:108653, May 2022.

Summary

  • The paper demonstrates that training pointer networks with evolutionary algorithms avoids local optima and boosts performance.
  • It applies a Negatively Correlated Search method to handle large-scale instances, achieving a 30.21% improvement over gradient descent.
  • The study highlights that increasing EA population size enhances solution diversity and scalability for complex combinatorial problems.

Introduction

Pointer Networks (PtrNets) are specialized neural networks designed to address NP-hard Combinatorial Optimization Problems (COPs). They offer real-time inference making them advantageous over traditional methods. However, PtrNets are often trained using gradient descent methods, which come with a limitation: they easily get trapped in local optima. This paper proposes using Evolutionary Algorithms (EAs) to train PtrNets, exploiting their global search capability and propensity to avoid local optima.

Evolutionary Training Advantages

EAs differ significantly from gradient descent approaches. EAs perform searches using populations of solutions, applying operators like mutation and crossover to evolve these solutions over time. This method has two key advantages in training neural networks:

  • Global Search Capability: EAs can potentially find better overall solutions by exploring various regions of the solution space simultaneously, rather than focusing on a single gradient path.
  • Diverse Search: Because EAs use populations, they naturally maintain a diversity of possible solutions, which can prevent premature convergence to suboptimal points.

Empirical Evaluation

To assess the benefits of EAs, the researchers conducted extensive experiments with PtrNet trained using an EA called Negatively Correlated Search (NCS) to solve the Traveling Salesman Problem (TSP). They explored scenarios with dimensionality up to 1000-nodes, which is challenging for PtrNets traditionally trained with much smaller problems. The experimental results revealed:

  • PtrNet trained with EA (PtrNet-EA) achieved up to 30.21% improvement over gradient descent-based methods.
  • Training PtrNet on the same dimensionality as the problem instances (in this case, 1000-dimensional TSPs) led to significant performance gains.
  • Increasing the population size within the EA improved the performance further, showcasing the scalability and efficiency benefits of parallelized EAs.

Conclusions and Implications

By demonstrating that EAs can effectively train PtrNets to outperform gradient descent techniques on larger-scale problems, this paper paves the way for more robust neural solvers in the field of optimization. It suggests that future applications of PtrNets and other similar neural models can benefit from training via EAs, especially for large and complex COPs where traditional methods struggle. The results also indicate that for problems like the TSP, using a training set of the same scale as the test cases is crucial for reaching high-quality solutions.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown