Papers
Topics
Authors
Recent
Search
2000 character limit reached

Exploring the Intersection between Neural Architecture Search and Continual Learning

Published 11 Jun 2022 in cs.AI, cs.CV, and cs.NE | (2206.05625v2)

Abstract: Despite the significant advances achieved in Artificial Neural Networks (ANNs), their design process remains notoriously tedious, depending primarily on intuition, experience and trial-and-error. This human-dependent process is often time-consuming and prone to errors. Furthermore, the models are generally bound to their training contexts, with no considerations to their surrounding environments. Continual adaptiveness and automation of neural networks is of paramount importance to several domains where model accessibility is limited after deployment (e.g IoT devices, self-driving vehicles, etc.). Additionally, even accessible models require frequent maintenance post-deployment to overcome issues such as Concept/Data Drift, which can be cumbersome and restrictive. By leveraging and combining approaches from Neural Architecture Search (NAS) and Continual Learning (CL), more robust and adaptive agents can be developed. This study conducts the first extensive review on the intersection between NAS and CL, formalizing the prospective Continually-Adaptive Neural Networks (CANNs) paradigm and outlining research directions for lifelong autonomous ANNs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (154)
  1. A. J. Otoole, P. J. Phillips, F. Jiang, J. Ayyad, N. Penard, and H. Abdi, “Face recognition algorithms surpass humans matching faces over changes in illumination,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 9, p. 1642–1646, 2007.
  2. D. Hassabis, “Artificial intelligence: Chess match of the century,” Nature, vol. 544, no. 7651, p. 413–414, 2017.
  3. K. Darlington, “How good are ai reading comprehension systems?” IQ: The RIM Quarterly, vol. 37, no. 4, pp. 40–43, 2021.
  4. M. Negrão and P. Domingues, “Speechtotext: An open-source software for automatic detection and transcription of voice recordings in digital forensics,” Forensic Science International: Digital Investigation, vol. 38, p. 301223, 2021.
  5. Y. Sun, D. Liang, X. Wang, and X. Tang, “Deepid3: Face recognition with very deep neural networks,” arXiv preprint arXiv:1502.00873, 2015.
  6. Q. Yu, Y. Yang, F. Liu, Y.-Z. Song, T. Xiang, and T. M. Hospedales, “Sketch-a-net: A deep neural network that beats humans,” International Journal of Computer Vision, vol. 122, no. 3, p. 411–425, 2016.
  7. H. Y. Khaw, F. C. Soon, J. H. Chuah, and C.-O. Chow, “Image noise types recognition using convolutional neural network with principal components analysis,” IET Image Processing, vol. 11, no. 12, pp. 1238–1245, 2017.
  8. A. Blair and A. Saffidine, “Ai surpasses humans at six-player poker,” Science, vol. 365, no. 6456, p. 864–865, 2019.
  9. A. M. Zador, “A critique of pure learning and what artificial neural networks can learn from animal brains,” Nature Communications, vol. 10, no. 1, 2019.
  10. K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” Evolutionary computation, vol. 10, no. 2, pp. 99–127, 2002.
  11. Y. Liu, Y. Sun, B. Xue, M. Zhang, G. G. Yen, and K. C. Tan, “A survey on evolutionary neural architecture search,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–21, 2021.
  12. B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” in ICLR, 2017.
  13. M. Mccloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” Psychology of Learning and Motivation, p. 109–165, 1989.
  14. J. L. Mcclelland, B. L. Mcnaughton, and R. C. Oreilly, “Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.” Psychological Review, vol. 102, no. 3, p. 419–457, 1995.
  15. A. Soltoggio, K. O. Stanley, and S. Risi, “Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks,” Neural Networks, vol. 108, p. 48–67, 2018.
  16. G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” Neural Networks, vol. 113, p. 54–71, 2019.
  17. G. Anthes, “Lifelong learning in artificial neural networks,” Communications of the ACM, vol. 62, no. 6, p. 13–15, 2019.
  18. M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 1–1, 2021.
  19. J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, and C.-C. J. Kuo, “Class-incremental learning via deep model consolidation,” 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020.
  20. Y. Luo, L. Yin, W. Bai, and K. Mao, “An appraisal of incremental learning methods,” Entropy, vol. 22, no. 11, p. 1190, 2020.
  21. G. M. van de Ven, T. Tuytelaars, and A. S. Tolias, “Three types of incremental learning,” Nature Machine Intelligence, pp. 1–13, 2022.
  22. A. Robins, “Sequential learning in neural networks: A review and a discussion of pseudorehearsal based methods,” Intelligent Data Analysis, vol. 8, no. 3, p. 301–322, 2004.
  23. J. Zevin and M. Seidenberg, “Simulating consistency effects and individual differences in nonword naming: A comparison of current models,” Journal of Memory and Language, vol. 54, no. 2, p. 145–160, 2006.
  24. J. Chen, Y. C. Leong, C. J. Honey, C. H. Yong, K. A. Norman, and U. Hasson, “Shared memories reveal shared structure in neural activity across individuals,” Nature Neuroscience, vol. 20, no. 1, p. 115–125, 2016.
  25. A. Cleeremans, “The radical plasticity thesis: How the brain learns to be conscious,” Frontiers in Psychology, vol. 2, 2011.
  26. M. S. Bennett, “What behavioral abilities emerged at key milestones in human brain evolution? 13 hypotheses on the 600-million-year phylogenetic history of human intelligence,” Frontiers in Psychology, vol. 12, 2021.
  27. F. L. Coolidge, “The evolution of learning and memory systems,” Evolutionary Neuropsychology, p. 27–53, 2020.
  28. T. Takeuchi, A. J. Duszkiewicz, and R. G. M. Morris, “The synaptic plasticity and memory hypothesis: encoding, storage and persistence,” Philosophical Transactions of the Royal Society B: Biological Sciences, vol. 369, no. 1633, p. 20130288, 2014.
  29. D. E. Feldman, “Spike timing–dependent plasticity,” Neural Circuit and Cognitive Development, p. 127–141, 2020.
  30. M. A. Lynch, “Long-term potentiation and memory,” Physiological Reviews, vol. 84, no. 1, p. 87–136, 2004.
  31. H. C. Dringenberg, “The history of long‐term potentiation as a memory mechanism: Controversies, confirmation, and some lessons to remember,” Hippocampus, vol. 30, no. 9, p. 987–1012, 2020.
  32. T. Tsumoto, “Long-term depression in cerebral cortex: a possible substrate of “forgetting” that should not be forgotten,” Neuroscience Research, vol. 16, no. 4, p. 263–270, 1993.
  33. M. Escobar and B. Derrick, “Long-term potentiation and depression as putative mechanisms for memory formation,” Neural Plasticity and Memory Frontiers in Neuroscience, p. 15–46, 2007.
  34. E. Tulving, “Organization of memory: Quo vadis,” in The Cognitive Neurosciences, M. S. Gazzaniga, Ed.   MIT Press, 1995, pp. 839–847.
  35. R. C. O’Reilly, R. Bhattacharyya, M. D. Howard, and N. Ketz, “Complementary learning systems,” Cognitive Science, vol. 38, no. 6, p. 1229–1248, 2011.
  36. S. Blakeman and D. Mareschal, “A complementary learning systems approach to temporal difference learning,” Neural Networks, vol. 122, p. 218–230, 2020.
  37. R. Kemker and C. Kanan, “Fearnet: Brain-inspired model for incremental learning,” in International Conference on Learning Representations, 2018.
  38. R. Shin, C. Packer, and D. Song, “Differentiable neural network architecture search,” in ICLR, 2018.
  39. J. M. Pierre, “Incremental lifelong deep learning for autonomous vehicles,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018, pp. 3949–3954.
  40. Z. Sun, Y. Wang, and R. Laganière, “Online model adaptation for uav tracking with convolutional neural network,” in 2018 15th Conference on Computer and Robot Vision (CRV), 2018, pp. 329–336.
  41. B. Wehbe, M. Hildebrandt, and F. Kirchner, “A framework for on-line learning of underwater vehicles dynamic models,” 2019 International Conference on Robotics and Automation (ICRA), pp. 7969–7975, 2019.
  42. Y. Nose, A. Kojima, H. Kawabata, and T. Hironaka, “A study on a lane keeping system using cnn for online learning of steering control from real time images,” in 2019 34th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC), 2019, pp. 1–4.
  43. W. Si, T. Wei, and C. Liu, “Agen: Adaptable generative prediction networks for autonomous driving,” in 2019 IEEE Intelligent Vehicles Symposium (IV), 2019, pp. 281–286.
  44. M. M. Ferdaus, M. Pratama, S. G. Anavatti, and M. A. Garratt, “Online identification of a rotary wing unmanned aerial vehicle from data streams,” Applied Soft Computing, vol. 76, p. 313–325, 2019.
  45. K. Shaheen, M. A. Hanif, O. Hasan, and M. Shafique, “Continual learning for real-world autonomous systems: Algorithms, challenges and frameworks,” CoRR, vol. abs/2105.12374, 2021.
  46. S. Thrun and T. M. Mitchell, “Lifelong robot learning,” Robotics and Autonomous Systems, vol. 15, no. 1-2, p. 25–46, 1995.
  47. T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” Information Fusion, vol. 58, p. 52–68, 2020.
  48. S. Chen and J. T. Wen, “Adaptive neural trajectory tracking control for flexible-joint robots with online learning,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 2358–2364.
  49. W. Zheng, H. Liu, and F. Sun, “Lifelong visual-tactile cross-modal learning for robotic material perception,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 3, pp. 1192–1203, 2021.
  50. A. Xie and C. Finn, “Lifelong robotic reinforcement learning by retaining experiences,” in Advances in Neural Information Processing Systems, 2021.
  51. G. Kahn, P. Abbeel, and S. Levine, “Badgr: An autonomous self-supervised learning-based navigation system,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 1312–1319, 2021.
  52. H.-C. Cheng and W.-W. Liao, “Establishing an lifelong learning environment using iot and learning analytics,” in 2012 14th International Conference on Advanced Communication Technology (ICACT), 2012, pp. 1178–1183.
  53. M. Song, K. Zhong, J. Zhang, Y. Hu, D. Liu, W. Zhang, J. Wang, and T. Li, “In-situ ai: Towards autonomous and incremental deep learning for iot systems,” in 2018 IEEE International Symposium on High Performance Computer Architecture (HPCA).   IEEE, 2018, pp. 92–103.
  54. Y. Bao and W. Chen, “Il4iot: Incremental learning for internet-of-things devices,” Lecture Notes in Computer Science, p. 92–107, 2019.
  55. A. K. Chathoth, A. Jagannatha, and S. Lee, “Federated intrusion detection for iot with heterogeneous cohort privacy,” 2021. [Online]. Available: https://arxiv.org/abs/2101.09878
  56. M. Rasouli, R. Ghosh, W. W. Lee, N. V. Thakor, and S. Kukreja, “Stable force-myographic control of a prosthetic hand using incremental learning,” in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).   IEEE, 2015, pp. 4828–4831.
  57. A. Patra and J. A. Noble, “Hierarchical class incremental learning of anatomical structures in fetal echocardiography videos,” IEEE journal of biomedical and health informatics, vol. 24, no. 4, pp. 1046–1058, 2020.
  58. A. Elskhawy, A. Lisowska, M. Keicher, J. Henry, P. Thomson, and N. Navab, “Continual class incremental learning for ct thoracic segmentation,” in Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning.   Springer, 2020, pp. 106–116.
  59. J. Chauhan, Y. D. Kwon, P. Hui, and C. Mascolo, “Contauth,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 4, no. 4, p. 1–23, 2020.
  60. S. Srivastava, M. Yaqub, K. Nandakumar, Z. Ge, and D. Mahapatra, “Continual domain incremental learning for chest x-ray classification in low-resource clinical settings,” in Domain Adaptation and Representation Transfer, and Affordable Healthcare and AI for Resource Diverse Global Health.   Springer, 2021, pp. 226–238.
  61. Y. Liu, J. Wang, J. Li, S. Niu, and H. Song, “Class-incremental learning for wireless device identification in iot,” IEEE Internet of Things Journal, 2021.
  62. J. Park, M. Kang, and B. Han, “Class-incremental learning for action recognition in videos,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 13 698–13 707.
  63. P. Ren, Y. Xiao, X. Chang, P.-y. Huang, Z. Li, X. Chen, and X. Wang, “A comprehensive survey of neural architecture search,” ACM Computing Surveys, vol. 54, no. 4, p. 1–34, 2022.
  64. G. Raghavan and M. Thomson, “Neural networks grown and self-organized by noise,” 2019. [Online]. Available: https://arxiv.org/abs/1906.01039
  65. A. Kaplunovich and Y. Yesha, “Automatic hyperparameter optimization for arbitrary neural networks in serverless aws cloud,” 2021 12th International Conference on Information and Communication Systems (ICICS), 2021.
  66. J.-B. Mouret and P. Tonelli, “Artificial evolution of plastic neural networks: a few key concepts,” Growing Adaptive Machines: combining Development and Learning in Artificial Neural Networks, pp. 251–261, 2014.
  67. T. Miconi, K. Stanley, and J. Clune, “Differentiable plasticity: training plastic neural networks with backpropagation,” in International Conference on Machine Learning.   PMLR, 2018, pp. 3559–3568.
  68. T. Nagarajah and G. Poravi, “A review on automated machine learning (automl) systems,” in 2019 IEEE 5th International Conference for Convergence in Technology (I2CT).   IEEE, 2019, pp. 1–6.
  69. J. Waring, C. Lindvall, and R. Umeton, “Automated machine learning: Review of the state-of-the-art and opportunities for healthcare,” Artificial Intelligence in Medicine, vol. 104, p. 101822, 2020.
  70. X. He, K. Zhao, and X. Chu, “Automl: A survey of the state-of-the-art,” Knowledge-Based Systems, vol. 212, p. 106622, 2021.
  71. T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” Automated Machine Learning, p. 63–77, 2019.
  72. Y. Jaafra, J. L. Laurent, A. Deruyver, and M. S. Naceur, “Reinforcement learning for neural architecture search: A review,” Image and Vision Computing, vol. 89, pp. 57–66, 2019.
  73. M. Wistuba, A. Rawat, and T. Pedapati, “A survey on neural architecture search,” 2019. [Online]. Available: https://arxiv.org/abs/1905.01392
  74. S. Grossberg, “Studies of mind and brain,” Boston Studies in the Philosophy of Science, 1982.
  75. M. Mermillod, A. Bugaiska, and P. Bonin, “The stability-plasticity dilemma: investigating the continuum from catastrophic forgetting to age-limited learning effects,” Frontiers in Psychology, vol. 4, 2013.
  76. D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17.   Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6470–6479.
  77. R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 139–154.
  78. C. X. Ling and T. Bohn, “A deep learning framework for lifelong machine learning,” 2021. [Online]. Available: https://arxiv.org/abs/2105.00157
  79. S. Farquhar and Y. Gal, “Towards robust evaluations of continual learning,” CoRR, vol. abs/1805.09733, 2018.
  80. S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  81. J. Yoon, E. Yang, J. Lee, and S. J. Hwang, “Lifelong learning with dynamically expandable networks,” in International Conference on Learning Representations, 2018.
  82. D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32.   Curran Associates, Inc., 2019.
  83. T. L. Hayes, G. P. Krishnan, M. Bazhenov, H. T. Siegelmann, T. J. Sejnowski, and C. Kanan, “Replay in deep learning: Current approaches and missing biological elements,” Neural Computation, vol. 33, no. 11, pp. 2908–2950, 2021.
  84. G. Hinton, O. Vinyals, J. Dean et al., “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531
  85. A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” 2016. [Online]. Available: https://arxiv.org/abs/1606.04671
  86. R. Aljundi, P. Chakravarty, and T. Tuytelaars, “Expert gate: Lifelong learning with a network of experts,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3366–3375.
  87. Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 12, p. 2935–2947, 2018.
  88. J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017.
  89. F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in International Conference on Machine Learning.   PMLR, 2017, pp. 3987–3995.
  90. N. Kamra, U. Gupta, and Y. Liu, “Deep generative dual memory network for continual learning,” 2017. [Online]. Available: https://arxiv.org/abs/1710.10368
  91. S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang, “Overcoming catastrophic forgetting by incremental moment matching,” Advances in neural information processing systems, vol. 30, 2017.
  92. H. Jung, J. Ju, M. Jung, and J. Kim, “Less-forgetful learning for domain expansion in deep neural networks,” in AAAI, 2018, pp. 3358–3365.
  93. R. Aljundi, M. Rohrbach, and T. Tuytelaars, “Selfless sequential learning,” in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.   OpenReview.net, 2019.
  94. A. New, M. Baker, E. Nguyen, and G. Vallabha, “Lifelong learning metrics,” arXiv preprint arXiv:2201.08278, 2022.
  95. N. Diaz-Rodriguez, V. Lomonaco, D. Filliat, and D. Maltoni, “Don’t forget, there is more than forgetting: new metrics for continual learning,” arXiv preprint arXiv:1810.13166, 2018.
  96. R. Kemker, M. McClure, A. Abitino, T. Hayes, and C. Kanan, “Measuring catastrophic forgetting in neural networks,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018.
  97. T. L. Hayes, R. Kemker, N. D. Cahill, and C. Kanan, “New metrics and experimental paradigms for continual learning,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018.
  98. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  99. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in International Conference on Learning Representations, 2015.
  100. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  101. F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size,” arXiv preprint arXiv:1602.07360, 2016.
  102. I. Guyon, K. Bennett, G. Cawley, H. J. Escalante, S. Escalera, T. K. Ho, N. Macià, B. Ray, M. Saeed, A. Statnikov et al., “Design of the 2015 chalearn automl challenge,” in 2015 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2015, pp. 1–8.
  103. H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, and F. Hutter, “Towards automatically-tuned neural networks,” in Proceedings of the Workshop on Automatic Machine Learning, ser. Proceedings of Machine Learning Research, F. Hutter, L. Kotthoff, and J. Vanschoren, Eds., vol. 64.   New York, New York, USA: PMLR, 24 Jun 2016, pp. 58–65.
  104. H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, M. Urban, M. Burkart, M. Dippel, M. Lindauer, and F. Hutter, “Towards automatically-tuned deep neural networks,” Automated Machine Learning, p. 135–149, 2019.
  105. K. Yu, C. Sciuto, M. Jaggi, C. Musat, and M. Salzmann, “Evaluating the search phase of neural architecture search,” in International Conference on Learning Representations, 2020.
  106. B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10 726–10 734.
  107. H. Hu, J. Langford, R. Caruana, S. Mukherjee, E. J. Horvitz, and D. Dey, “Efficient forward architecture search,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32.   Curran Associates, Inc., 2019.
  108. D. Baymurzina, E. Golikov, and M. Burtsev, “A review of neural architecture search,” Neurocomputing, vol. 474, pp. 82–93, 2022.
  109. B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architectures using reinforcement learning,” CoRR, vol. abs/1611.02167, 2016.
  110. H. Cai, J. Yang, W. Zhang, S. Han, and Y. Yu, “Path-level network transformation for efficient architecture search,” in International Conference on Machine Learning.   PMLR, 2018, pp. 678–687.
  111. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 8697–8710.
  112. X. Gong, S. Chang, Y. Jiang, and Z. Wang, “Autogan: Neural architecture search for generative adversarial networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3224–3234.
  113. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700–4708.
  114. C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 19–34.
  115. Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu, “Practical block-wise neural network architecture generation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2423–2432.
  116. H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” in International Conference on Learning Representations, 2019.
  117. T. Elsken, J. H. Metzen, and F. Hutter, “Efficient multi-objective neural architecture search via lamarckian evolution,” in International Conference on Learning Representations, 2019.
  118. E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in Proceedings of the aaai conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 4780–4789.
  119. H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu, “Hierarchical representations for efficient architecture search,” in International Conference on Learning Representations, 2018.
  120. K. O. Stanley, D. B. D’Ambrosio, and J. Gauci, “A hypercube-based encoding for evolving large-scale neural networks,” Artificial life, vol. 15, no. 2, pp. 185–212, 2009.
  121. L. Wu, B. Liu, P. Stone, and Q. Liu, “Firefly neural architecture descent: a general approach for growing neural networks,” Advances in Neural Information Processing Systems, vol. 33, pp. 22 373–22 383, 2020.
  122. Y. Ci, C. Lin, M. Sun, B. Chen, H. Zhang, and W. Ouyang, “Evolving search space for neural architecture search,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6659–6669.
  123. H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in International Conference on Learning Representations, 2019.
  124. A. Brock, T. Lim, J. Ritchie, and N. Weston, “SMASH: One-shot model architecture search through hypernetworks,” in International Conference on Learning Representations, 2018.
  125. P. Liashchynskyi and P. Liashchynskyi, “Grid search, random search, genetic algorithm: A big comparison for nas,” 2019. [Online]. Available: https://arxiv.org/abs/1912.06059
  126. J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization.” Journal of machine learning research, vol. 13, no. 2, 2012.
  127. R. G. Mantovani, A. L. D. Rossi, J. Vanschoren, B. Bischl, and A. C. P. L. F. de Carvalho, “Effectiveness of random search in svm hyper-parameter tuning,” in 2015 International Joint Conference on Neural Networks (IJCNN), 2015, pp. 1–8.
  128. L. Li and A. Talwalkar, “Random search and reproducibility for neural architecture search,” in Uncertainty in artificial intelligence.   PMLR, 2020, pp. 367–377.
  129. R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3, pp. 229–256, 1992.
  130. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017. [Online]. Available: https://arxiv.org/abs/1707.06347
  131. G. F. Miller, P. M. Todd, and S. U. Hegde, “Designing neural networks using genetic algorithms.” in ICGA, vol. 89, 1989, pp. 379–384.
  132. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in International Conference on Machine Learning.   PMLR, 2017, pp. 2902–2911.
  133. L. Xie and A. Yuille, “Genetic cnn,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1379–1388.
  134. M. Suganuma, S. Shirakawa, and T. Nagao, “A genetic programming approach to designing convolutional neural network architectures,” in Proceedings of the genetic and evolutionary computation conference, 2017, pp. 497–504.
  135. D. E. Goldberg and K. Deb, “A comparative analysis of selection schemes used in genetic algorithms,” in Foundations of genetic algorithms.   Elsevier, 1991, vol. 1, pp. 69–93.
  136. Y. Chen, G. Meng, Q. Zhang, S. Xiang, C. Huang, L. Mu, and X. Wang, “Renas: Reinforced evolutionary neural architecture search,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4787–4796.
  137. X. Dong and Y. Yang, “Searching for a robust neural architecture in four gpu hours,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 1761–1770.
  138. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” Advances in neural information processing systems, vol. 24, 2011.
  139. J. Bergstra, D. Yamins, and D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in International conference on machine learning.   PMLR, 2013, pp. 115–123.
  140. B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas, “Taking the human out of the loop: A review of bayesian optimization,” Proceedings of the IEEE, vol. 104, no. 1, pp. 148–175, 2015.
  141. H. Zhou, M. Yang, J. Wang, and W. Pan, “Bayesnas: A bayesian approach for neural architecture search,” in International conference on machine learning.   PMLR, 2019, pp. 7603–7613.
  142. C. White, W. Neiswanger, and Y. Savani, “Bananas: Bayesian optimization with neural architectures for neural architecture search,” arXiv preprint arXiv:1910.11858, vol. 1, no. 2, p. 4, 2019.
  143. R. Niu, H. Li, Y. Zhang, and Y. Kang, “Neural architecture search based on particle swarm optimization,” in 2019 3rd International Conference on Data Science and Business Analytics (ICDSBA), 2019, pp. 319–324.
  144. E. Byla and W. Pang, “Deepswarm: Optimising convolutional neural networks using swarm intelligence,” in UK Workshop on Computational Intelligence.   Springer, 2019, pp. 119–130.
  145. S. Lankford and D. Grimes, “Neural architecture search using particle swarm and ant colony optimization.” in AICS, 2020, pp. 229–240.
  146. E. Suda and H. Iba, “Neural architecture search and weight adjustment by means of ant colony optimization,” in 2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS).   IEEE, 2020, pp. 184–191.
  147. R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” Advances in neural information processing systems, vol. 31, 2018.
  148. M. Zhang, H. Li, S. Pan, X. Chang, and S. Su, “Overcoming multi-model forgetting in one-shot nas with diversity maximization,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 7806–7815.
  149. K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” Advances in neural information processing systems, vol. 31, 2018.
  150. H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in International conference on machine learning.   PMLR, 2018, pp. 4095–4104.
  151. Z. Ding, Y. Chen, N. Li, D. Zhao, Z. Sun, and C. L. P. Chen, “Bnas: Efficient neural architecture search using broad scalable architecture,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2021.
  152. Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, “Nsga-net: Neural architecture search using multi-objective genetic algorithm,” in Proceedings of the Genetic and Evolutionary Computation Conference, ser. GECCO ’19.   New York, NY, USA: Association for Computing Machinery, 2019, p. 419–427. [Online]. Available: https://doi.org/10.1145/3321707.3321729
  153. Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” in European Conference on Computer Vision.   Springer, 2020, pp. 544–560.
  154. Z. Yang, Y. Wang, X. Chen, B. Shi, C. Xu, C. Xu, Q. Tian, and C. Xu, “Cars: Continuous evolution for efficient neural architecture search,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1829–1838.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up