Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Encodings for Prediction-based Neural Architecture Search (2403.02484v1)

Published 4 Mar 2024 in cs.LG, cs.AI, cs.CV, and cs.NE

Abstract: Predictor-based methods have substantially enhanced Neural Architecture Search (NAS) optimization. The efficacy of these predictors is largely influenced by the method of encoding neural network architectures. While traditional encodings used an adjacency matrix describing the graph structure of a neural network, novel encodings embrace a variety of approaches from unsupervised pretraining of latent representations to vectors of zero-cost proxies. In this paper, we categorize and investigate neural encodings from three main types: structural, learned, and score-based. Furthermore, we extend these encodings and introduce \textit{unified encodings}, that extend NAS predictors to multiple search spaces. Our analysis draws from experiments conducted on over 1.5 million neural network architectures on NAS spaces such as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and TransNASBench-101. Building on our study, we present our predictor \textbf{FLAN}: \textbf{Fl}ow \textbf{A}ttention for \textbf{N}AS. FLAN integrates critical insights on predictor design, transfer learning, and \textit{unified encodings} to enable more than an order of magnitude cost reduction for training NAS accuracy predictors. Our implementation and encodings for all neural networks are open-sourced at \href{https://github.com/abdelfattah-lab/flan_nas}{https://github.com/abdelfattah-lab/flan\_nas}.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.08134, 2021.
  2. Multi-predict: Few shot predictors for efficient neural architecture search, 2023.
  3. BLOX: Macro neural architecture search benchmark and algorithms. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=IIbJ9m5G73t.
  4. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020. URL https://openreview.net/forum?id=HJxyZkBKDr.
  5. Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5251–5260, 2021.
  6. Brp-nas: Prediction-based nas using gcns. volume 33, pp.  10480–10490, 2020.
  7. NAT: Neural Architecture Transformer for Accurate and Compact Architectures. Curran Associates Inc., Red Hook, NY, USA, 2019.
  8. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.
  9. Nas-bench-suite-zero: Accelerating research on zero cost proxies, 2022a.
  10. Nas-bench-suite-zero: Accelerating research on zero cost proxies. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022b.
  11. Help: Hardware-adaptive efficient latency prediction for nas via meta-learning. In 35th Conference on Neural Information Processing Systems (NeurIPS) 2021. Conference on Neural Information Processing Systems (NeurIPS), 2021.
  12. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations.
  13. Random search and reproducibility for neural architecture search. arXiv preprint arXiv:1902.07638, 2019.
  14. Best practices for scientific research on neural architecture search. arXiv preprint arXiv:1909.02453, 2019.
  15. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pp.  19–34, 2018.
  16. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=S1eYHoC5FX.
  17. Bridge the gap between architecture spaces via a cross-domain predictor. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=nE6vnoHz9--.
  18. {NAS}-bench-{asr}: Reproducible neural architecture search for speech recognition. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=CU0APx9LMaL.
  19. Nas-bench-suite: Nas evaluation is (now) surprisingly easy, 2022.
  20. Neural architecture search without training. In International Conference on Machine Learning, pp.  7588–7598. PMLR, 2021.
  21. Gennape: Towards generalized neural architecture performance estimators, 2022.
  22. Simple and deep graph convolutional networks. 2020.
  23. TA-GATES: An encoding scheme for neural network architectures. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=74fJwNrBlPI.
  24. A generic graph-based neural architecture encoding scheme with multifaceted information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):7955–7969, 2023. doi: 10.1109/TPAMI.2022.3228604.
  25. Efficient neural architecture search via parameters sharing. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.  4095–4104. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/pham18a.html.
  26. On network design spaces for visual recognition. In ICCV, 2019.
  27. Regularized evolution for image classifier architecture search. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, pp.  4780–4789, 2019.
  28. Bridging the gap between sample-based and one-shot neural architecture search with bonas. volume 33, pp.  1808–1819, 2020.
  29. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377–6389, 2020.
  30. Graph attention networks. In International Conference on Learning Representations, 2018.
  31. A study on encodings for neural architecture search. In Advances in Neural Information Processing Systems, 2020.
  32. How powerful are performance predictors in neural architecture search?, 2021.
  33. Does unsupervised architecture representation learning help neural architecture search? In NeurIPS, 2020.
  34. Cate: Computation-aware neural architecture encoding with transformers. In ICML, 2021.
  35. Nas evaluation is frustratingly hard. 2020.
  36. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp.  7105–7114. PMLR, 2019.
  37. Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks, 2020. URL https://arxiv.org/abs/2008.09777.
  38. Econas: Finding proxies for economical neural architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  11393–11401, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10.1109/CVPR42600.2020.01141. URL https://doi.ieeecomputersociety.org/10.1109/CVPR42600.2020.01141.
  39. Deep subdomain adaptation network for image classification. IEEE Transactions on Neural Networks and Learning Systems, 32(4):1713–1722, 2021. doi: 10.1109/TNNLS.2020.2988928.
  40. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=r1Ue8Hcxg.
  41. Learning transferable architectures for scalable image recognition. pp.  8697–8710, 06 2018. doi: 10.1109/CVPR.2018.00907.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Yash Akhauri (19 papers)
  2. Mohamed S. Abdelfattah (37 papers)
Citations (1)
View on Semantic Scholar