Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search (2402.13204v1)

Published 20 Feb 2024 in cs.NE and cs.LG

Abstract: Recent advancements in AI, driven by Neural Networks (NN), demand innovative neural architecture designs, particularly within the constrained environments of Internet of Things (IoT) systems, to balance performance and efficiency. HW-aware Neural Architecture Search (HW-aware NAS) emerges as an attractive strategy to automate the design of NN using multi-objective optimization approaches, such as evolutionary algorithms. However, the intricate relationship between NN design parameters and HW-aware NAS optimization objectives remains an underexplored research area, overlooking opportunities to effectively leverage this knowledge to guide the search process accordingly. Furthermore, the large amount of evaluation data produced during the search holds untapped potential for refining the optimization strategy and improving the approximation of the Pareto front. Addressing these issues, we propose SONATA, a self-adaptive evolutionary algorithm for HW-aware NAS. Our method leverages adaptive evolutionary operators guided by the learned importance of NN design parameters. Specifically, through tree-based surrogate models and a Reinforcement Learning agent, we aspire to gather knowledge on 'How' and 'When' to evolve NN architectures. Comprehensive evaluations across various NAS search spaces and hardware devices on the ImageNet-1k dataset have shown the merit of SONATA with up to 0.25% improvement in accuracy and up to 2.42x gains in latency and energy. Our SONATA has seen up to sim$93.6% Pareto dominance over the native NSGA-II, further stipulating the importance of self-adaptive evolution operators in HW-aware NAS.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. K. Yu, C. Sciuto, M. Jaggi, C. Musat, and M. Salzmann, “Evaluating the search phase of neural architecture search,” in International Conference on Learning Representations, 2019.
  2. H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,” arXiv preprint arXiv:1812.00332, 2018.
  3. H. Benmeziane, K. E. Maghraoui, H. Ouarnoughi, S. Niar, M. Wistuba, and N. Wang, “A comprehensive survey on hardware-aware neural architecture search,” CoRR, vol. abs/2101.09336, 2021. [Online]. Available: https://arxiv.org/abs/2101.09336
  4. L.-C. Chen, M. Collins, Y. Zhu, G. Papandreou, B. Zoph, F. Schroff, H. Adam, and J. Shlens, “Searching for efficient multi-scale architectures for dense image prediction,” Advances in neural information processing systems, vol. 31, 2018.
  5. Z. Lu, K. Deb, E. Goodman, W. Banzhaf, and V. N. Boddeti, “Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16.   Springer, 2020, pp. 35–51.
  6. S. Liu, H. Zhang, and Y. Jin, “A survey on computationally efficient neural architecture search,” Journal of Automation and Intelligence, vol. 1, no. 1, p. 100002, 2022.
  7. B. Greenwood and T. McDonnell, “Surrogate-assisted neuroevolution,” in Proceedings of the Genetic and Evolutionary Computation Conference, 2022, pp. 1048–1056.
  8. T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” The Journal of Machine Learning Research, vol. 20, no. 1, pp. 1997–2017, 2019.
  9. C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “Nas-bench-101: Towards reproducible neural architecture search,” in International conference on machine learning.   PMLR, 2019, pp. 7105–7114.
  10. X. Dong and Y. Yang, “Nas-bench-201: Extending the scope of reproducible neural architecture search,” in International Conference on Learning Representations, 2019.
  11. H. Cai et al., “Once-for-all: Train one network and specialize it for efficient deployment,” in International Conference on Learning Representations (ICLR), 2019.
  12. B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 734–10 742.
  13. P. R. Lorenzo, J. Nalepa, M. Kawulok, L. S. Ramos, and J. R. Pastor, “Particle swarm optimization for hyper-parameter selection in deep neural networks,” in Proceedings of the genetic and evolutionary computation conference, 2017, pp. 481–488.
  14. Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, “Nsga-net: neural architecture search using multi-objective genetic algorithm,” in Proceedings of the genetic and evolutionary computation conference, 2019, pp. 419–427.
  15. C.-H. Hsu, S.-H. Chang, J.-H. Liang, H.-P. Chou, C.-H. Liu, S.-C. Chang, J.-Y. Pan, Y.-T. Chen, W. Wei, and D.-C. Juan, “Monas: Multi-objective neural architecture search using reinforcement learning,” arXiv preprint arXiv:1806.10332, 2018.
  16. C. White, W. Neiswanger, and Y. Savani, “Bananas: Bayesian optimization with neural architectures for neural architecture search,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, 2021, pp. 10 293–10 301.
  17. Y. Shen, Y. Li, J. Zheng, W. Zhang, P. Yao, J. Li, S. Yang, J. Liu, and B. Cui, “Proxybo: Accelerating neural architecture search via bayesian optimization with zero-cost proxies,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, 2023, pp. 9792–9801.
  18. J. N. Siems, L. Zimmer, A. Zela, J. Lukasik, M. Keuper, and F. Hutter, “Nas-bench-301 and the case for surrogate benchmarks for neural architecture search,” 2020.
  19. C. White, A. Zela, R. Ru, Y. Liu, and F. Hutter, “How powerful are performance predictors in neural architecture search?” Advances in Neural Information Processing Systems, vol. 34, pp. 28 454–28 469, 2021.
  20. H. Bouzidi, H. Ouarnoughi, S. Niar, and A. A. E. Cadi, “Performance modeling of computer vision-based cnn on edge gpus,” ACM Transactions on Embedded Computing Systems (TECS), vol. 21, no. 5, pp. 1–33, 2022.
  21. C. Li, Z. Yu, Y. Fu, Y. Zhang, Y. Zhao, H. You, Q. Yu, Y. Wang, C. Hao, and Y. Lin, “Hw-nas-bench: Hardware-aware neural architecture search benchmark,” in International Conference on Learning Representations, 2020.
  22. L. Chen and H. Xu, “Mfenas: multifactorial evolution for neural architecture search,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2022, pp. 631–634.
  23. I. Trofimov, N. Klyuchnikov, M. Salnikov, A. Filippov, and E. Burnaev, “Multi-fidelity neural architecture search with knowledge distillation,” IEEE Access, 2023.
  24. H. Cai, T. Chen, W. Zhang, Y. Yu, and J. Wang, “Efficient architecture search by network transformation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.
  25. H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” in International Conference on Learning Representations, 2018.
  26. H. Wang, R. Yang, D. Huang, and Y. Wang, “idarts: Improving darts by node normalization and decorrelation discretization,” IEEE Transactions on Neural Networks and Learning Systems, 2021.
  27. H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameters sharing,” in International conference on machine learning.   PMLR, 2018, pp. 4095–4104.
  28. M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, and R. Jin, “Zen-nas: A zero-shot nas for high-performance image recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 347–356.
  29. C. Yan, X. Chang, Z. Li, W. Guan, Z. Ge, L. Zhu, and Q. Zheng, “Zeronas: Differentiable generative adversarial networks search for zero-shot learning,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 12, pp. 9733–9740, 2021.
  30. Y. Liu, Y. Sun, B. Xue, M. Zhang, G. G. Yen, and K. C. Tan, “A survey on evolutionary neural architecture search,” IEEE transactions on neural networks and learning systems, 2021.
  31. C. Pan and X. Yao, “Neural architecture search based on evolutionary algorithms with fitness approximation,” in 2021 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2021, pp. 1–8.
  32. C. White, W. Neiswanger, S. Nolen, and Y. Savani, “A study on encodings for neural architecture search,” Advances in neural information processing systems, vol. 33, pp. 20 309–20 319, 2020.
  33. H. Shi, R. Pi, H. Xu, Z. Li, J. Kwok, and T. Zhang, “Bridging the gap between sample-based and one-shot neural architecture search with bonas,” Advances in Neural Information Processing Systems, vol. 33, pp. 1808–1819, 2020.
  34. N. Sinha and K.-W. Chen, “Novelty driven evolutionary neural architecture search,” in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2022, pp. 671–674.
  35. K. Deb, S. Agrawal, A. Pratap, and T. Meyarivan, “A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii,” in Parallel Problem Solving from Nature PPSN VI: 6th International Conference Paris, France, September 18–20, 2000 Proceedings 6.   Springer, 2000, pp. 849–858.
  36. L. Fan and H. Wang, “Surrogate-assisted evolutionary neural architecture search with network embedding,” Complex & Intelligent Systems, vol. 9, no. 3, pp. 3313–3331, 2023.
  37. Y. Peng, A. Song, V. Ciesielski, H. M. Fayek, and X. Chang, “Pre-nas: predictor-assisted evolutionary neural architecture search,” in Proceedings of the Genetic and Evolutionary Computation Conference, 2022, pp. 1066–1074.
  38. C. Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, and J. Liang, “Npenas: Neural predictor guided evolution for neural architecture search,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  39. E. C. Garrido-Merchán and D. Hernández-Lobato, “Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes,” Neurocomputing, vol. 380, pp. 20–35, 2020.
  40. M. B. Calisto and S. K. Lai-Yuen, “Emonas-net: Efficient multiobjective neural architecture search using surrogate-assisted evolutionary algorithm for 3d medical image segmentation,” Artificial Intelligence in Medicine, vol. 119, p. 102154, 2021.
  41. H. Cho, J. Shin, and W. Rhee, “B2ea: An evolutionary algorithm assisted by two bayesian optimization modules for neural architecture search,” arXiv preprint arXiv:2202.03005, 2022.
  42. J. Wu, X. Dai, D. Chen, Y. Chen, M. Liu, Y. Yu, Z. Wang, Z. Liu, M. Chen, and L. Yuan, “Stronger nas with weaker predictors,” Advances in Neural Information Processing Systems, vol. 34, pp. 28 904–28 918, 2021.
  43. L. Wang, S. Xie, T. Li, R. Fonseca, and Y. Tian, “Sample-efficient neural architecture search by learning actions for monte carlo tree search,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, pp. 5503–5515, 2021.
  44. K. R. Traoré, A. Camero, and X. X. Zhu, “A data-driven approach to neural architecture search initialization,” Annals of Mathematics and Artificial Intelligence, pp. 1–28, 2023.
  45. Y. Chen, G. Meng, Q. Zhang, S. Xiang, C. Huang, L. Mu, and X. Wang, “Renas: Reinforced evolutionary neural architecture search,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4787–4796.
  46. Y. Xue, Y. Wang, J. Liang, and A. Slowik, “A self-adaptive mutation neural architecture search algorithm based on blocks,” IEEE Computational Intelligence Magazine, vol. 16, no. 3, pp. 67–78, 2021.
  47. Z. Qiu, W. Bi, D. Xu, H. Guo, H. Ge, Y. Liang, H. P. Lee, and C. Wu, “Efficient self-learning evolutionary neural architecture search,” Applied Soft Computing, vol. 146, p. 110671, 2023.
  48. P. A. Vikhar, “Evolutionary algorithms: A critical review and its future prospects,” in 2016 International conference on global trends in signal processing, information computing and communication (ICGTSPICC).   IEEE, 2016, pp. 261–265.
  49. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,” IEEE transactions on evolutionary computation, vol. 6, no. 2, pp. 182–197, 2002.
  50. C. Moussa, Y. J. Patel, V. Dunjko, T. Bäck, and J. N. van Rijn, “Hyperparameter importance and optimization of quantum neural networks across small datasets,” Machine Learning, pp. 1–26, 2023.
  51. F. Hutter, H. Hoos, and K. Leyton-Brown, “An efficient approach for assessing hyperparameter importance,” in International conference on machine learning.   PMLR, 2014, pp. 754–762.
  52. S. Watanabe, A. Bansal, and F. Hutter, “Ped-anova: Efficiently quantifying hyperparameter importance in arbitrary subspaces,” arXiv preprint arXiv:2304.10255, 2023.
  53. A. Biedenkapp, J. Marben, M. Lindauer, and F. Hutter, “Cave: Configuration assessment, visualization and evaluation,” in Learning and Intelligent Optimization: 12th International Conference, LION 12, Kalamata, Greece, June 10–15, 2018, Revised Selected Papers 12.   Springer, 2019, pp. 115–130.
  54. A. Biedenkapp, M. Lindauer, K. Eggensperger, F. Hutter, C. Fawcett, and H. Hoos, “Efficient parameter importance analysis via ablation with surrogates,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1, 2017.
  55. X. Zheng, R. Ji, Q. Wang, Q. Ye, Z. Li, Y. Tian, and Q. Tian, “Rethinking performance estimation in neural architecture search,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11 356–11 365.
  56. H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, “Once-for-all: Train one network and specialize it for efficient deployment,” in International Conference on Learning Representations.
  57. D. Wang, C. Gong, M. Li, Q. Liu, and V. Chandra, “Alphanet: Improved training of supernets with alpha-divergence,” in International Conference on Machine Learning.   PMLR, 2021, pp. 10 760–10 771.
  58. C. Gong, D. Wang, M. Li, X. Chen, Z. Yan, Y. Tian, V. Chandra et al., “Nasvit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training,” in International Conference on Learning Representations, 2021.
  59. H. Bouzidi, M. Odema, H. Ouarnoughi, M. A. Al Faruque, and S. Niar, “Hadas: Hardware-aware dynamic neural architecture search for edge performance scaling,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE).   IEEE, 2023, pp. 1–6.
  60. M. Odema, H. Bouzidi, H. Ouarnoughi, S. Niar, and M. A. Al Faruque, “Magnas: A mapping-aware graph neural architecture search framework for heterogeneous mpsoc deployment,” ACM Transactions on Embedded Computing Systems, vol. 22, no. 5s, pp. 1–26, 2023.
  61. M. I. E. Ghebriout, H. Bouzidi, S. Niar, and H. Ouarnoughi, “Harmonic-nas: Hardware-aware multimodal neural architecture search on resource-constrained devices,” arXiv preprint arXiv:2309.06612, 2023.
  62. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4510–4520.
  63. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2020.
  64. T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” ser. KDD ’16.   New York, NY, USA: Association for Computing Machinery, 2016, p. 785–794. [Online]. Available: https://doi.org/10.1145/2939672.2939785
  65. A. Liaw, M. Wiener et al., “Classification and regression by randomforest,” R news, vol. 2, no. 3, pp. 18–22, 2002.
  66. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8697–8710.
  67. A. Ashok, N. Rhinehart, F. Beainy, and K. M. Kitani, “N2n learning: Network to network compression via policy gradient reinforcement learning,” in International Conference on Learning Representations, 2018. [Online]. Available: https://openreview.net/forum?id=B1hcZZ-AW
  68. A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al., “Searching for mobilenetv3,” in Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 1314–1324.
  69. H. Vanholder. (2016, Jun) Efficient inference with tensorrt. [Online]. Available: https://developer.nvidia.com/tensorrt
  70. R. Lange, T. Schaul, Y. Chen, C. Lu, T. Zahavy, V. Dalibard, and S. Flennerhag, “Discovering attention-based genetic algorithms via meta-black-box optimization,” in Proceedings of the Genetic and Evolutionary Computation Conference, 2023, pp. 929–937.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Halima Bouzidi (9 papers)
  2. Smail Niar (15 papers)
  3. Hamza Ouarnoughi (12 papers)
  4. El-Ghazali Talbi (21 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets