Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FR-NAS: Forward-and-Reverse Graph Predictor for Efficient Neural Architecture Search (2404.15622v1)

Published 24 Apr 2024 in cs.LG

Abstract: Neural Architecture Search (NAS) has emerged as a key tool in identifying optimal configurations of deep neural networks tailored to specific tasks. However, training and assessing numerous architectures introduces considerable computational overhead. One method to mitigating this is through performance predictors, which offer a means to estimate the potential of an architecture without exhaustive training. Given that neural architectures fundamentally resemble Directed Acyclic Graphs (DAGs), Graph Neural Networks (GNNs) become an apparent choice for such predictive tasks. Nevertheless, the scarcity of training data can impact the precision of GNN-based predictors. To address this, we introduce a novel GNN predictor for NAS. This predictor renders neural architectures into vector representations by combining both the conventional and inverse graph views. Additionally, we incorporate a customized training loss within the GNN predictor to ensure efficient utilization of both types of representations. We subsequently assessed our method through experiments on benchmark datasets including NAS-Bench-101, NAS-Bench-201, and the DARTS search space, with a training dataset ranging from 50 to 400 samples. Benchmarked against leading GNN predictors, the experimental results showcase a significant improvement in prediction accuracy, with a 3%--16% increase in Kendall-tau correlation. Source codes are available at https://github.com/EMI-Group/fr-nas.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le, “MnasNet: Platform-aware neural architecture search for mobile,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  2. K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” in Advances in Neural Information Processing Systems, vol. 31.   Curran Associates, Inc., 2018.
  3. C. White, W. Neiswanger, and Y. Savani, “BANANAS: Bayesian optimization with neural architectures for neural architecture search,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 10 293–10 301, May 2021.
  4. Z. Lu, I. Whalen, V. Boddeti, Y. Dhebar, K. Deb, E. Goodman, and W. Banzhaf, “Nsga-net: neural architecture search using multi-objective genetic algorithm,” in Proceedings of the genetic and evolutionary computation conference, 2019, pp. 419–427.
  5. Y. Sun, B. Xue, M. Zhang, and G. G. Yen, “Evolving deep convolutional neural networks for image classification,” IEEE Transactions on Evolutionary Computation, vol. 24, no. 2, pp. 394–407, 2020.
  6. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  7. B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer, “FBNet: Hardware-Aware efficient ConvNet design via differentiable neural architecture search,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  8. C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, “NAS-bench-101: Towards reproducible neural architecture search,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97.   PMLR, 09–15 Jun 2019, pp. 7105–7114.
  9. H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, “Once for all: Train one network and specialize it for efficient deployment,” in International Conference on Learning Representations, 2020.
  10. X. Chu, B. Zhang, and R. Xu, “Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 239–12 248.
  11. H. Tan, R. Cheng, S. Huang, C. He, C. Qiu, F. Yang, and P. Luo, “RelativeNAS: Relative neural architecture search via slow-fast learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 1, pp. 475–489, 2023.
  12. C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in Computer Vision – ECCV 2018.   Springer International Publishing, 2018, pp. 19–35.
  13. W. Wen, H. Liu, Y. Chen, H. Li, G. Bender, and P.-J. Kindermans, “Neural predictor for neural architecture search,” in Computer Vision – ECCV 2020.   Cham: Springer International Publishing, 2020, pp. 660–676.
  14. C. Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, and J. Liang, “NPENAS: Neural predictor guided evolution for neural architecture search,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–15, 2022.
  15. Y. Sun, H. Wang, B. Xue, Y. Jin, G. G. Yen, and M. Zhang, “Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor,” IEEE Transactions on Evolutionary Computation, vol. 24, no. 2, pp. 350–364, 2020.
  16. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations, 2017.
  17. K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in International Conference on Learning Representations, 2019.
  18. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.
  19. Z. Chen, Y. Zhan, B. Yu, M. Gong, and B. Du, “Not all operations contribute equally: Hierarchical operation-adaptive predictor for neural architecture search,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 10 508–10 517.
  20. R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” in Advances in Neural Information Processing Systems, vol. 31.   Curran Associates, Inc., 2018.
  21. Z. Lu, R. Cheng, S. Huang, H. Zhang, C. Qiu, and F. Yang, “Surrogate-assisted multiobjective neural architecture search for real-time semantic segmentation,” IEEE Transactions on Artificial Intelligence, vol. 4, no. 6, pp. 1602–1615, 2023.
  22. H. Shi, R. Pi, H. Xu, Z. Li, J. Kwok, and T. Zhang, “Bridging the gap between sample-based and one-shot neural architecture search with BONAS,” in Advances in Neural Information Processing Systems, vol. 33.   Curran Associates, Inc., 2020, pp. 1808–1819.
  23. Y. Liu, Y. Tang, and Y. Sun, “Homogeneous architecture augmentation for neural predictor,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 12 249–12 258.
  24. C. White, A. Zela, R. Ru, Y. Liu, and F. Hutter, “How powerful are performance predictors in neural architecture search?” in Advances in Neural Information Processing Systems, vol. 34.   Curran Associates, Inc., 2021, pp. 28 454–28 469.
  25. X. Dai, A. Wan, P. Zhang, B. Wu, Z. He, Z. Wei, K. Chen, Y. Tian, M. Yu, P. Vajda, and J. E. Gonzalez, “FBNetV3: Joint Architecture-Recipe search using predictor pretraining,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 16 276–16 285.
  26. K. Jing, J. Xu, and P. Li, “Graph masked autoencoder enhanced predictor for neural architecture search,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22.   International Joint Conferences on Artificial Intelligence Organization, 7 2022, pp. 3114–3120, main Track.
  27. M. Zhang, S. Jiang, Z. Cui, R. Garnett, and Y. Chen, “D-VAE: A variational autoencoder for directed acyclic graphs,” in Advances in Neural Information Processing Systems, vol. 32.   Curran Associates, Inc., 2019.
  28. J. Lukasik, D. Friede, A. Zela, F. Hutter, and M. Keuper, “Smooth variational graph embeddings for efficient neural architecture search,” in 2021 International Joint Conference on Neural Networks (IJCNN), 2021, pp. 1–8.
  29. Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” in Advances in Neural Information Processing Systems, vol. 33.   Curran Associates, Inc., 2020, pp. 5812–5823.
  30. Y. Chen, Y. Guo, Q. Chen, M. Li, W. Zeng, Y. Wang, and M. Tan, “Contrastive neural architecture search with neural architecture comparators,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 9502–9511.
  31. D. Hesslow and I. Poli, “Contrastive embeddings for neural architectures,” arXiv preprint arXiv:2102.04208, 2021.
  32. Y. Liu, J. Cao, B. Li, C. Yuan, W. Hu, Y. Li, and Y. Duan, “Knowledge distillation via instance relationship graph,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  33. X. Dong and Y. Yang, “NAS-bench-201: Extending the scope of reproducible neural architecture search,” in International Conference on Learning Representations, 2020.
  34. H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” in International Conference on Learning Representations, 2019.
  35. A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” University of Toronto, Toronto, Ontario, Tech. Rep. 0, 2009.
  36. A. Zela, J. N. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter, “Surrogate NAS benchmarks: going beyond the limited search spaces of tabular NAS benchmarks,” in The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.   OpenReview.net, 2022.
  37. Z. Lu, R. Cheng, Y. Jin, K. C. Tan, and K. Deb, “Neural architecture search as multiobjective optimization benchmarks: Problem formulation and performance assessment,” IEEE Transactions on Evolutionary Computation, pp. 1–1, 2022.
  38. Y. Mehta, C. White, A. Zela, A. Krishnakumar, G. Zabergja, S. Moradian, M. Safari, K. Yu, and F. Hutter, “NAS-bench-suite: NAS evaluation is (now) surprisingly easy,” in International Conference on Learning Representations, 2022.
  39. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in 3rd International Conference on Learning Representations, ICLR, 2015.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Haoming Zhang (15 papers)
  2. Ran Cheng (130 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets