Papers
Topics
Authors
Recent
Search
2000 character limit reached

ECToNAS: Evolutionary Cross-Topology Neural Architecture Search

Published 8 Mar 2024 in cs.LG, cs.CV, and cs.NE | (2403.05123v1)

Abstract: We present ECToNAS, a cost-efficient evolutionary cross-topology neural architecture search algorithm that does not require any pre-trained meta controllers. Our framework is able to select suitable network architectures for different tasks and hyperparameter settings, independently performing cross-topology optimisation where required. It is a hybrid approach that fuses training and topology optimisation together into one lightweight, resource-friendly process. We demonstrate the validity and power of this approach with six standard data sets (CIFAR-10, CIFAR-100, EuroSAT, Fashion MNIST, MNIST, SVHN), showcasing the algorithm's ability to not only optimise the topology within an architectural type, but also to dynamically add and remove convolutional cells when and where required, thus crossing boundaries between different network types. This enables researchers without a background in machine learning to make use of appropriate model types and topologies and to apply machine learning methods in their domains, with a computationally cheap, easy-to-use cross-topology neural architecture search framework that fully encapsulates the topology optimisation within the training process.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Neural network surgery: Combining training with topology optimization. Neural Networks, 144:384–393, 2021. 10.1016/j.neunet.2021.08.034.
  2. Net2Net: Accelerating Learning via Knowledge Transfer. In International Conference on Learning Representations, 2016. 10.48550/arXiv.1511.05641.
  3. R. Liu, et al. NFP: A No Fine-tuning Pruning Approach for Convolutional Neural Network Compression. In 2020 3rd International Conference on Artificial Intelligence and Big Data (ICAIBD), pages 74–77. IEEE, 2020. 10.1109/ICAIBD49809.2020.9137429.
  4. M. Tenorio and W.-T. Lee. Self Organizing Neural Networks for the Identification Problem. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmann, 1988. https://proceedings.neurips.cc/paper_files/paper/1988/file/f2217062e9a397a1dca429e7d70bc6ca-Paper.pdf.
  5. B. Zoph and Q. Le. Neural Architecture Search with Reinforcement Learning. In International Conference on Learning Representations, 2017. https://openreview.net/forum?id=r1Ue8Hcxg.
  6. DARTS: Differentiable Architecture Search. In International Conference on Learning Representations, 2019. https://openreview.net/forum?id=S1eYHoC5FX.
  7. P. Ren, et al. A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions. ACM Computing Surveys, 54(4), May 2021. 10.1145/3447582.
  8. Efficient Neural Architecture Search via Parameters Sharing. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4095–4104. PMLR, July 2018. https://proceedings.mlr.press/v80/pham18a.html.
  9. X. Dong and Y. Yang. Searching for a Robust Neural Architecture in Four GPU Hours. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1761–1770, 2019. 10.1109/CVPR.2019.00186.
  10. G. Bender, et al. Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 10.1109/CVPR42600.2020.01433.
  11. Evaluating The Search Phase of Neural Architecture Search. In International Conference on Learning Representations, 2020. https://openreview.net/forum?id=H1loF2NFwr.
  12. Y. Xu, et al. PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search. In International Conference on Learning Representations, 2020. https://openreview.net/forum?id=BJlS634tPr.
  13. L. Xie, et al. Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap. ACM Computing Surveys, 54(9), October 2021. 10.1145/3473330.
  14. Geometry-Aware Gradient Algorithms for Neural Architecture Search. In International Conference on Learning Representations, 2021. https://openreview.net/forum?id=MuSYkd1hxRP.
  15. A Survey on Evolutionary Construction of Deep Neural Networks. IEEE Transactions on Evolutionary Computation, 25(5):894–912, 2021. 10.1109/TEVC.2021.3079985.
  16. A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks. IEEE Computational Intelligence Magazine, 16(3):67–78, 2021. 10.1109/MCI.2021.3084435.
  17. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 10.1109/CVPR.2016.90.
  18. Densely Connected Convolutional Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261–2269, 2017. 10.1109/CVPR.2017.243.
  19. ReCNAS: Resource-Constrained Neural Architecture Search Based on Differentiable Annealing and Dynamic Pruning. IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2022. 10.1109/TNNLS.2022.3192169.
  20. What is the State of Neural Network Pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze, editors, Proceedings of Machine Learning and Systems, volume 2, pages 129–146, 2020. https://proceedings.mlsys.org/paper/2020/file/d2ddea18f00665ce8623e36bd4e3c7c5-Paper.pdf.
  21. Network Pruning That Matters: A Case Study on Retraining Variants. In International Conference on Learning Representations, 2021. https://openreview.net/forum?id=Cb54AMqHQFP.
  22. E. M. Mirkes. Artificial Neural Network Pruning to Extract Knowledge. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2020. 10.1109/IJCNN48605.2020.9206861.
  23. D. Gurevin, et al. Enabling Retrain-free Deep Neural Network Pruning Using Surrogate Lagrangian Relaxation. In Z.-H. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2497–2504. International Joint Conferences on Artificial Intelligence Organization, August 2021. 10.24963/ijcai.2021/344. Main Track.
  24. Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing. IEEE Signal Processing Letters, 28:344–348, 2021. 10.1109/LSP.2021.3054315.
  25. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019. https://openreview.net/forum?id=B1VZqjAcYX.
  26. Picking Winning Tickets Before Training by Preserving Gradient Flow. In International Conference on Learning Representations, 2020. https://openreview.net/forum?id=SkgsACVKPH.
  27. Pruning neural networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6377–6389. Curran Associates, Inc., 2020. https://proceedings.neurips.cc/paper/2020/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf.
  28. Single-Shot Pruning for Offline Reinforcement Learning, 2021. 10.48550/ARXIV.2112.15579.
  29. T. Chen, et al. Only Train Once: A One-Shot Neural Network Training And Pruning Framework. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 19637–19651. Curran Associates, Inc., 2021. https://proceedings.neurips.cc/paper/2021/file/a376033f78e144f494bfc743c0be3330-Paper.pdf.
  30. Dynamic Pruning of a Neural Network via Gradient Signal-to-Noise Ratio. In 8th ICML Workshop on Automated Machine Learning (AutoML), 2021. https://openreview.net/forum?id=34awaeWZgya.
  31. L. Miao, et al. Learning Pruning-Friendly Networks via Frank-Wolfe: One-Shot, Any-Sparsity, And No Retraining. In International Conference on Learning Representations, 2022. https://openreview.net/forum?id=O1DEtITim__.
  32. Z. Liu, et al. Learning Efficient Convolutional Networks through Network Slimming. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2755–2763, 2017. 10.1109/ICCV.2017.298.
  33. Y. Idelbayev and M. A. Carreira-Perpiñán. Optimal Selection of Matrix Shape and Decomposition Scheme for Neural Network Compression. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3250–3254, 2021. 10.1109/ICASSP39728.2021.9414224.
  34. Learning Structured Sparsity in Deep Neural Networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. https://proceedings.neurips.cc/paper/2016/file/41bfd20a38bb1b0bec75acf0845530a7-Paper.pdf.
  35. Training Structured Neural Networks Through Manifold Identification and Variance Reduction. In International Conference on Learning Representations, 2022. https://openreview.net/forum?id=mdUYT5QV0O.
  36. Learning Efficient Image Super-Resolution Networks via Structure-Regularized Pruning. In International Conference on Learning Representations, 2022. https://openreview.net/forum?id=AjGC97Aofee.
  37. Zero-Cost Proxies for Lightweight NAS. In International Conference on Learning Representations, 2021. https://openreview.net/forum?id=0cmMMy8J5q.
  38. B. Ru, et al. Speedy Performance Estimation for Neural Architecture Search. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. https://openreview.net/forum?id=8V2hZW0d2aS.
  39. How many degrees of freedom do we need to train deep networks: a loss landscape perspective. In International Conference on Learning Representations, 2022. https://openreview.net/forum?id=ChMLTGRjFcU.
  40. A. LeNail. NN-SVG: Publication-Ready Neural Network Architecture Schematics. Journal of Open Source Software, 4(33):747, 2019. 10.21105/joss.00747.
  41. Efficient Architecture Search by Network Transformation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018. 10.1609/aaai.v32i1.11709.
  42. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
  43. Introducing Eurosat: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. In IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, pages 204–207, 2018. 10.1109/IGARSS.2018.8519248.
  44. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 10.1109/JSTARS.2019.2918242.
  45. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms, 2017. 10.48550/ARXIV.1708.07747.
  46. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 10.1109/5.726791.
  47. Y. Netzer, et al. Reading Digits in Natural Images with Unsupervised Feature Learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
  48. R. Kohavi. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, volume 96, pages 202–207, 1996.
  49. D. Dua and C. Graff. The UCI Machine Learning Repository, 2017. http://archive.ics.uci.edu/.
  50. M. Lindauer and F. Hutter. Best Practices for Scientific Research on Neural Architecture Search. Journal of Machine Learning Research, 21(243):1–18, 2020. http://jmlr.org/papers/v21/20-056.html;https://www.automl.org/nas_checklist.pdf.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up