Zero-Cost Proxies for Lightweight NAS (2101.08134v2)

Abstract: Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between multiple models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4$\times$ quicker than the best previous result. Our code is made public at: https://github.com/mohsaied/zero-cost-nas.

Zero-Cost Proxies for Lightweight NAS: An In-Depth Evaluation

In the domain of automated neural architecture discovery, the paper titled "Zero-Cost Proxies for Lightweight NAS" proposes a significant advancement in the efficiency of Neural Architecture Search (NAS). This paper explores innovative proxy methods that forego the traditional and computationally expensive full-training model evaluations, instead introducing zero-cost proxies inspired by pruning-at-initialization techniques.

Core Contributions

The paper puts forward the concept of zero-cost proxies, which are metrics derived from recent pruning literature. These metrics are computed using a single minibatch of training data and a forward/backward pass, significantly reducing computational expense. The key contributions of the paper can be summarized as follows:

1. Introduction of Zero-Cost Proxies: The authors adapt pruning-at-initialization metrics—namely SNIP, GRASP, Synaptic Flow (SynFlow), Fisher, and Jacobian Covariance—used at a per-parameter level, to score entire neural networks for NAS, thus repurposing them for efficient architecture search.
2. Empirical Evaluation: A comprehensive empirical analysis on several benchmarks (NAS-Bench-201, NAS-Bench-101, NAS-Bench-ASR, and NAS-Bench-NLP) is provided, evaluating both the rank consistency of zero-cost proxies and their practicality in real-world NAS scenarios.
3. Enhancements to NAS Algorithms: The zero-cost proxies are integrated into existing NAS search methodologies, such as random search, reinforcement learning, evolutionary search, and predictor-based search, demonstrating substantial improvements in search efficiency.

Performance and Comparisons

The paper provides strong numerical results, demonstrating that zero-cost proxies are capable of matching, and in some cases, outperforming conventional reduced-training proxies. Notably, the Spearman's rank correlation coefficient between final validation accuracy and their best zero-cost proxy (SynFlow) on NAS-Bench-201 is 0.82, outperforming the 0.61 achieved by EcoNAS, a widely recognized reduced-training proxy.

Practical and Theoretical Implications

The implications of this work are twofold:

• Practical: By drastically reducing the computational requirements for model evaluation, this approach democratizes access to NAS, making it feasible for resource-constrained settings or real-time applications, potentially catalyzing faster adoption of efficient model deployment in diverse environments.
• Theoretical: This research enriches the intersection of pruning literature and NAS, hinting at the underlying potential of initialization-based saliency metrics to identify promising architectures without exhaustive training. This opens avenues for further explorations into the theoretical underpinnings of network trainability and architectural saliency.

Future Directions

The paper invites further research into the analytical insights of why certain zero-cost proxies, particularly SynFlow, work effectively across various benchmarks. Understanding the mechanisms that enable successful zero-cost predictions could lead to the development of new proxies, further improving NAS efficiency. Additionally, integrating zero-cost proxies into other machine learning paradigms beyond NAS could present a fertile ground for future exploration.

In essence, this paper lays the groundwork for more efficient, accessible approaches to NAS, propelling the field towards more resource-conscious methodologies while maintaining performance integrity. The publication makes a clear case for the broader adoption of initialization-based proxies, presenting a versatile toolkit for future NAS research and applications.