Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

How Much Is Hidden in the NAS Benchmarks? Few-Shot Adaptation of a NAS Predictor (2311.18451v1)

Published 30 Nov 2023 in cs.LG

Abstract: Neural architecture search has proven to be a powerful approach to designing and refining neural networks, often boosting their performance and efficiency over manually-designed variations, but comes with computational overhead. While there has been a considerable amount of research focused on lowering the cost of NAS for mainstream tasks, such as image classification, a lot of those improvements stem from the fact that those tasks are well-studied in the broader context. Consequently, applicability of NAS to emerging and under-represented domains is still associated with a relatively high cost and/or uncertainty about the achievable gains. To address this issue, we turn our focus towards the recent growth of publicly available NAS benchmarks in an attempt to extract general NAS knowledge, transferable across different tasks and search spaces. We borrow from the rich field of meta-learning for few-shot adaptation and carefully study applicability of those methods to NAS, with a special focus on the relationship between task-level correlation (domain shift) and predictor transferability; which we deem critical for improving NAS on diverse tasks. In our experiments, we use 6 NAS benchmarks in conjunction, spanning in total 16 NAS settings -- our meta-learning approach not only shows superior (or matching) performance in the cross-validation experiments but also successful extrapolation to a new search space and tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. NAS-bench-360: Benchmarking neural architecture search on diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  2. Rethinking neural operations for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  3. Efficient architecture search for diverse tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  4. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning (ICML), 2017.
  5. Transnas-bench-101: Improving transferability and generalizability of cross-task neural architecture search. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  6. Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search. In International Conference on Learning Representations (ICLR), 2020.
  7. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning (ICML), 2019.
  8. Nas-bench-asr: Reproducible neural architecture search for speech recognition. In International Conference on Learning Representations (ICLR), 2020.
  9. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
  10. Learning transferable architectures for scalable image recognition. In IEEE conference on computer vision and pattern recognition (CVPR), 2018.
  11. MnasNet: Platform-aware neural architecture search for mobile. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  12. Regularized evolution for image classifier architecture search. In AAAI Conference on Artificial Intelligence (AAAI), 2019.
  13. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision (ECCV), 2020.
  14. Blockwisely supervised neural architecture search with knowledge distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  15. Darts: Differentiable architecture search. In International Conference on Machine Learning (ICLR), 2019.
  16. Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  17. ProxylessNAS: Direct neural architecture search on target task and hardware. In International Conference on Learning Representations (ICLR), 2019.
  18. Sgas: Sequential greedy architecture search. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  19. Rethinking architecture selection in differentiable NAS. In International Conference on Learning Representations (ICLR), 2021.
  20. Bridging the gap between sample-based and one-shot neural architecture search with bonas. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  21. Bananas: Bayesian optimization with neural architectures for neural architecture search. In AAAI Conference on Artificial Intelligence (AAAI), 2020.
  22. Neural predictor for neural architecture search. In European Conference on Computer Vision (ECCV), 2020.
  23. BRP-NAS: Prediction-based NAS using GCNs. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  24. Stronger nas with weaker predictors. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  25. Zero-cost proxies for lightweight NAS. In International Conference on Learning Representations (ICLR), 2021.
  26. Neural architecture search without training. In International Conference on Machine Learning (ICLR), 2021.
  27. Zen-nas: A zero-shot nas for high-performance deep image recognition. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  28. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations (ICLR), 2021.
  29. Semi-supervised neural architecture search. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  30. A semi-supervised assessor of neural architectures. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  31. Are labels necessary for neural architecture search? In European Conference on Computer Vision (ECCV), 2020.
  32. Generic neural architecture search via regression. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  33. Dynamic ensemble of low-fidelity experts: Mitigating nas "cold-start". In AAAI Conference on Artificial Intelligence (AAAI), 2023.
  34. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations (ICLR), 2020.
  35. Contrastive neural architecture search with neural architecture comparators. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  36. A deeper look at zero-cost proxies for lightweight nas. In ICLR Blog Track, 2022.
  37. Densely connected search space for more flexible neural architecture search. In IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020.
  38. Boil: Towards representation change for few-shot learning. In International Conference on Machine Learning (ICML), 2021.
  39. Meta-learning of neural architectures for few-shot learning. In IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020.
  40. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2018.
  41. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations (ICLR), 2019.
  42. Meta architecture search. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
  43. How to train your maml. In International Conference on Learning Representations (ICLR), 2019.
  44. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations (ICLR), 2021.
  45. Help: Hardware-adaptive efficient latency prediction for nas via meta-learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  46. How powerful are performance predictors in neural architecture search? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  47. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Machine Learning (ICML), 2020.
  48. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  49. Learning multiple layers of features from tiny images, 2009.
  50. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017.
  51. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations (ICLR), 2021.
  52. Building the ninapro database: A resource for the biorobotics community. In IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob), 2012.
  53. Spherical CNNs. In International Conference on Learning Representations (ICLR), 2018.
  54. Af classification from a short single lead ecg recording: The physionet/computing in cardiology challenge 2017. In Computing in Cardiology (CinC), pages 1–4. IEEE, 2017.
  55. Satellite image time series analysis under time warping. IEEE transactions on geoscience and remote sensing, 2012.
  56. BLOX: Macro neural architecture search benchmark and algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  57. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777, 2020.
  58. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
  59. sEMG gesture recognition with a simple model of attention. In Machine Learning for Health, pages 126–138. PMLR, 2020.
  60. Holmes: health online model ensemble serving for deep learning models in intensive care units. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1614–1624, 2020.
  61. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5):1454–1495, 2020.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Hrushikesh Loya (4 papers)
  2. Łukasz Dudziak (41 papers)
  3. Abhinav Mehrotra (16 papers)
  4. Royson Lee (19 papers)
  5. Javier Fernandez-Marques (19 papers)
  6. Nicholas D. Lane (97 papers)
  7. Hongkai Wen (29 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets