NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy (2201.13396v2)

Abstract: The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench-201, has significantly lowered the computational overhead for conducting scientific research in neural architecture search (NAS). Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classification. Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing. However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks. In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, finding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks. To help remedy this problem, we introduce NAS-Bench-Suite, a comprehensive and extensible collection of NAS benchmarks, accessible through a unified interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research. Our code is available at https://github.com/automl/naslib.

Summary of "NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy"

The paper presents "NAS-Bench-Suite", an extensive benchmarking suite designed to evaluate Neural Architecture Search (NAS) algorithms across a wide variety of search spaces and tasks. The development of this benchmarking suite addresses the limitations and inconsistencies presented by existing tabular benchmarks like NAS-Bench-101 and NAS-Bench-201, which are constrained to small search spaces and specific tasks such as image classification.

Key Contributions

1. Comprehensive Benchmarking Suite: The authors introduce NAS-Bench-Suite, encompassing 25 combinations of search spaces and datasets, which offers a more generalizable and extensible framework for NAS research.
2. Diverse Range of Tasks: The suite covers a broad spectrum, including tasks such as object detection, speech recognition, and natural language processing, beyond classic image classification. This diversity helps in understanding the generalizability of NAS algorithms across different tasks.
3. Standardization Across Benchmarks: NAS-Bench-Suite provides a unified interface for NAS benchmarks, facilitating reproducible and rapid NAS research. This standardization is critical in comparing NAS algorithms without the overhead of incompatible libraries and differing benchmark abstractions.
4. Analysis of NAS Algorithms: The paper presents in-depth empirical analysis across various NAS algorithms, highlighting that conclusions from a limited subset of benchmarks may not be generalizable. It was demonstrated that superior performance on benchmarks like NAS-Bench-101 and NAS-Bench-201 does not necessarily translate to other search spaces.
5. Hyperparameter Tuning: Findings suggest that NAS algorithms do not possess robust default hyperparameters, indicating the necessity for tuning specific to the search space. The transferability of hyperparameters across tasks was also scrutinized, showing potential deterioration in performance if improperly transferred.

Implications and Future Directions

The NAS-Bench-Suite has significant implications for the field of neural architecture search:

• Practical Use: Researchers can significantly reduce computational overhead and achieve more statistically sound comparisons using the standardized search spaces and unified interface provided by NAS-Bench-Suite.
• Theoretical Insights: By identifying the lack of generalization and the challenges in hyperparameter transferability, the work underscores the nuanced understanding required for effectively deploying NAS algorithms.
• Future Work: Opportunities for future research include integrating even more diverse tasks and extending the benchmark to incorporate distributed and hardware-efficient NAS methods.

NAS-Bench-Suite is a step forward in codifying the evaluation of NAS algorithms, pushing the community towards reproducibility, comprehensive evaluation, and ultimately, a better understanding of the capabilities and limitations inherent in neural architecture search.