Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Backpropagation-Free 4D Continuous Ant-Based Neural Topology Search (2305.06715v3)

Published 11 May 2023 in cs.NE

Abstract: Continuous Ant-based Topology Search (CANTS) is a previously introduced novel nature-inspired neural architecture search (NAS) algorithm that is based on ant colony optimization (ACO). CANTS utilizes a continuous search space to indirectly-encode a neural architecture search space. Synthetic ant agents explore CANTS' continuous search space based on the density and distribution of pheromones, strongly inspired by how ants move in the real world. This continuous search space allows CANTS to automate the design of artificial neural networks (ANNs) of any size, removing a key limitation inherent to many current NAS algorithms that must operate within structures of a size that is predetermined by the user. This work expands CANTS by adding a fourth dimension to its search space representing potential neural synaptic weights. Adding this extra dimension allows CANTS agents to optimize both the architecture as well as the weights of an ANN without applying backpropagation (BP), which leads to a significant reduction in the time consumed in the optimization process: at least an average of 96% less time consumption with very competitive optimization performance, if not better. The experiments of this study - using real-world data - demonstrate that the BP-Free CANTS algorithm exhibits highly competitive performance compared to both CANTS and ANTS while requiring significantly less operation time.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Optimizing convolutional neural network hyperparameters by enhanced swarm intelligence metaheuristics. Algorithms 13, 3 (2020).
  2. Choosing optimal network structure. Springer Netherlands, Dordrecht, 1990, pp. 890–893.
  3. An ant colony optimization approach to the probabilistic traveling salesman problem. In International Conference on Parallel Problem Solving from Nature (2002), Springer, pp. 883–892.
  4. The ant colony metaphor for searching continuous design spaces. In AISB workshop on evolutionary computing (1995), Springer, pp. 25–39.
  5. A survey of swarm and evolutionary computing approaches for deep learning. Artificial Intelligence Review 53, 3 (2020), 1767–1812.
  6. Evolving deep recurrent neural networks using ant colony optimization. In Evolutionary Computation in Combinatorial Optimization (Cham, 2015), G. Ochoa and F. Chicano, Eds., Springer International Publishing, pp. 86–98.
  7. Dorigo, M. Optimization, learning and natural algorithms. PhD Thesis, Politecnico di Milano (1992).
  8. Ant colony optimization. IEEE computational intelligence magazine 1, 4 (2006), 28–39.
  9. Ant colony system: a cooperative learning approach to the traveling salesman problem. IEEE Transactions on evolutionary computation 1, 1 (1997), 53–66.
  10. Ant system: optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 26, 1 (1996), 29–41.
  11. Optimizing long short-term memory recurrent neural networks using ant colony optimization to predict turbine engine vibration. Applied Soft Computing 73 (2018), 969–991.
  12. Continuous ant-based neural topology search. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar) (2021), Springer, pp. 291–306.
  13. Ant-based neural topology search (ants) for optimizing recurrent networks. In International Conference on the Applications of Evolutionary Computation (Part of EvoStar) (2020), Springer, pp. 626–641.
  14. Neural architecture search: A survey. arXiv preprint arXiv:1808.05377 (2018).
  15. Impact of small-world topology on the performance of a feed-forward artificial neural network based on 2 different real-life problems. Turkish Journal of Electrical Engineering & Computer Sciences 22, 3 (2014), 708–718.
  16. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd (1996), vol. 96, pp. 226–231.
  17. Gordon, D. M. Ant encounters: interaction networks and colony behavior, vol. 1. Princeton University Press, 2010.
  18. Transistor size optimization in digital circuits using ant colony optimization for continuous domain. International Journal of Circuit Theory and Applications 42, 6 (2014), 642–658.
  19. Horng, M.-H. Fine-tuning parameters of deep belief networks using artificial bee colony algorithm. DEStech Transactions on Computer Science and Engineering (2017).
  20. Kuhn, L. D. Ant colony optimization for continuous spaces. Computer Science and Computer Engineering Undergraduate Honors Theses (35) (2002).
  21. A deep learning-cuckoo search method for missing data estimation in high-dimensional datasets. In International Conference on Swarm Intelligence (2017), Springer, pp. 561–572.
  22. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018).
  23. A survey on evolutionary neural architecture search. IEEE transactions on neural networks and learning systems (2021).
  24. Neural architecture optimization. In Advances in neural information processing systems (2018), pp. 7816–7827.
  25. M. Dorigo and L. M. Gambardella. Ant colonies for the travelling sales man problem. BioSystems, 43(2):73–81 (1997).
  26. Parallel ant colony optimization for the traveling salesman problem. In International Workshop on Ant Colony Optimization and Swarm Intelligence (2006), Springer, pp. 224–234.
  27. Evolving neural networks using ant colony optimization with pheromone trail limits. In Computational Intelligence (UKCI), 2013 13th UK Workshop on (2013), IEEE, pp. 16–23.
  28. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing. Elsevier, 2019, pp. 293–312.
  29. Cmac: An associative neural network alternative to backpropagation. Proceedings of the IEEE 78, 10 (1990), 1561–1567.
  30. Discrete event, continuous time rnns. arXiv preprint arXiv:1710.04110 (2017).
  31. Investigating recurrent neural network memory structures using neuro-evolution. In Proceedings of the Genetic and Evolutionary Computation Conference (New York, NY, USA, 2019), GECCO ’19, ACM, pp. 446–455.
  32. A detailed comparison of backpropagation neural network and maximum-likelihood classifiers for urban land use classification. IEEE Transactions on Geoscience and Remote Sensing 33, 4 (1995), 981–996.
  33. Scaling techniques for parallel ant colony optimization on large problem instances. In Proceedings of the Genetic and Evolutionary Computation Conference (2019), pp. 47–54.
  34. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268 (2018).
  35. Ant colony optimization for continuous domains. European journal of operational research 185, 3 (2008), 1155–1173.
  36. Designing neural networks through neuroevolution. Nature Machine Intelligence 1, 1 (2019), 24–35.
  37. Evolving neural networks through augmenting topologies. Evolutionary computation 10, 2 (2002), 99–127.
  38. A hybrid ant colony optimization for continuous domains. Expert Systems with Applications 38, 9 (2011), 11072–11077.
  39. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926 (2018).
  40. Yang, X.-S. Nature-inspired metaheuristic algorithms. Luniver press, 2010.
  41. Yang, X.-S. A new metaheuristic bat-inspired algorithm. In Nature inspired cooperative strategies for optimization (NICSO 2010). Springer, 2010, pp. 65–74.
  42. A backpropagation algorithm with adaptive learning rate and momentum coefficient. In Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN’02 (Cat. No.02CH37290) (2002), vol. 2, pp. 1218–1223 vol.2.
  43. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016).
  44. Comparing performances of backpropagation and genetic algorithms in the data classification. Expert Systems with Applications 38, 4 (2011), 3703–3709.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now