Papers
Topics
Authors
Recent
Search
2000 character limit reached

LISSNAS: Locality-based Iterative Search Space Shrinkage for Neural Architecture Search

Published 6 Jul 2023 in cs.CV | (2307.03110v1)

Abstract: Search spaces hallmark the advancement of Neural Architecture Search (NAS). Large and complex search spaces with versatile building operators and structures provide more opportunities to brew promising architectures, yet pose severe challenges on efficient exploration and exploitation. Subsequently, several search space shrinkage methods optimize by selecting a single sub-region that contains some well-performing networks. Small performance and efficiency gains are observed with these methods but such techniques leave room for significantly improved search performance and are ineffective at retaining architectural diversity. We propose LISSNAS, an automated algorithm that shrinks a large space into a diverse, small search space with SOTA search performance. Our approach leverages locality, the relationship between structural and performance similarity, to efficiently extract many pockets of well-performing networks. We showcase our method on an array of search spaces spanning various sizes and datasets. We accentuate the effectiveness of our shrunk spaces when used in one-shot search by achieving the best Top-1 accuracy in two different search spaces. Our method achieves a SOTA Top-1 accuracy of 77.6\% in ImageNet under mobile constraints, best-in-class Kendal-Tau, architectural diversity, and search space size.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 115–123, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
  2. SMASH: One-shot model architecture search through hypernetworks. In International Conference on Learning Representations, 2018.
  3. Once for all: Train one network and specialize it for efficient deployment. CoRR, abs/1908.09791, 2019.
  4. XGBoost. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, aug 2016.
  5. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. CoRR, abs/1907.01845, 2019.
  6. Evolving search space for neural architecture search. CoRR, abs/2011.10904, 2020.
  7. Single path one-shot neural architecture search with uniform sampling. CoRR, abs/1904.00420, 2019.
  8. Searching for mobilenetv3. CoRR, abs/1905.02244, 2019.
  9. Angle-based search space shrinking for neural architecture search. CoRR, abs/2004.13431, 2020.
  10. Greedynasv2: Greedier search with a greedy path filter. CoRR, abs/2111.12609, 2021.
  11. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
  12. DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
  13. Balanced one-shot neural architecture optimization. arXiv preprint arXiv:1909.10815, 2019.
  14. Understanding and improving one-shot neural architecture optimization. CoRR, abs/1909.10815, 2019.
  15. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  16. Towards automatically-tuned neural networks. In Workshop on automatic machine learning, pages 58–65. PMLR, 2016.
  17. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095–4104. PMLR, 2018.
  18. On network design spaces for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  19. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020.
  20. Large-scale evolution of image classifiers. In International Conference on Machine Learning, pages 2902–2911. PMLR, 2017.
  21. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789, 2019.
  22. Evaluating the search phase of neural architecture search. CoRR, abs/1902.08142, 2019.
  23. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. CoRR, abs/2008.09777, 2020.
  24. K-shot NAS: learnable weight-sharing for NAS with k-shot supernets. CoRR, abs/2106.06442, 2021.
  25. Mnasnet: Platform-aware neural architecture search for mobile. CoRR, abs/1807.11626, 2018.
  26. NAS-navigator: Visual steering for explainable one-shot deep neural network synthesis. IEEE Transactions on Visualization and Computer Graphics, pages 1–11, 2022.
  27. Attentivenas: Improving neural architecture search via attentive sampling. arXiv preprint arXiv:2011.09011, 2020.
  28. Alphanet: Improved training of supernets with alpha-divergence. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10760–10771. PMLR, 18–24 Jul 2021.
  29. Npenas: Neural predictor guided evolution for neural architecture search. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  30. Neural predictor for neural architecture search. In European Conference on Computer Vision, pages 660–676. Springer, 2020.
  31. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 34:28454–28469, 2021.
  32. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. CoRR, abs/1812.03443, 2018.
  33. HNAS: hierarchical neural architecture search on mobile devices. CoRR, abs/2005.07564, 2020.
  34. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018.
  35. A multi-objective evolutionary approach based on graph-in-graph for neural architecture search of convolutional neural networks. International Journal of Neural Systems, 31(09):2150035, 2021.
  36. Nas-bench-101: Towards reproducible neural architecture search. CoRR, abs/1902.09635, 2019.
  37. Greedynas: Towards fast one-shot nas with greedy supernet. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1996–2005, 2020.
  38. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  39. You only search once: Single shot neural architecture search via direct sparse optimization. CoRR, abs/1811.01567, 2018.
  40. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9):2891–2904, 2020.
  41. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
  42. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697–8710, 2018.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up