Random Search and Reproducibility for Neural Architecture Search (1902.07638v3)

Abstract: Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks---PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results. We note the lack of source material needed to exactly reproduce these results, and further discuss the robustness of published results given the various sources of variability in NAS experimental setups. Relatedly, we provide all information (code, random seeds, documentation) needed to exactly reproduce our results, and report our random search with weight-sharing results for each benchmark on multiple runs.

Overview of Random Search and Reproducibility for Neural Architecture Search

The paper "Random Search and Reproducibility for Neural Architecture Search" by Liam Li and Ameet Talwalkar addresses critical aspects of Neural Architecture Search (NAS) by highlighting new baselines and reproducibility challenges. The authors propose a methodology that leverages random search, particularly focusing on weight-sharing, to establish competitive baselines in NAS. They evaluate these methods using standard benchmarks: Penn Treebank (PTB) and CIFAR-10.

Key Contributions

• NAS as Hyperparameter Optimization: The authors frame NAS as a specialized case of hyperparameter optimization, stating that random search can be a competitive baseline. This perspective shifts the focus toward evaluating NAS methodologies against established hyperparameter optimization techniques.
• Evaluation Schemes: They propose evaluating two methods: random search with early-stopping (ASHA) and random search with weight-sharing, showcasing their effectiveness on PTB and CIFAR-10 benchmarks. These methods compete well with previous NAS approaches, such as ENAS and DARTS, thus underlining the necessity of robust baselines.
• Reproducibility and Experimental Setup: Li and Talwalkar address the reproducibility crisis in NAS experiments by open-sourcing their code, random seeds, and documentation. Their transparency provides a pathway to exact and broad reproducibility, which is often lacking in NAS research.

Strong Numerical Results

• PTB Benchmark: The authors demonstrate that random search with weight-sharing reaches a test perplexity of 55.5, outperforming ENAS, and aligns closely with other state-of-the-art NAS methods. ASHA also showed promising results, achieving a test perplexity of 56.4.
• CIFAR-10 Benchmark: On CIFAR-10, random search with weight-sharing achieved an average test error of 2.85%, which is competitive with DARTS and SNAS. ASHA reported an average test error of 3.03%, confirming random search's capability as a baseline.

Critique of Existing NAS Methods

The paper scrutinizes existing methods, revealing inadequacies in comparisons and the complexity introduced by advanced techniques like RL-based and gradient-based searches. They advocate for simplified methods that present clear and reproducible advantages over these techniques, questioning the necessity of intricate components.

Implications for NAS Research

• Practical Impacts: By presenting simpler and cost-effective methods, they reduce the computational burden and enhance accessibility to NAS for a broader range of researchers and practitioners.
• Theoretical Insights: Recasting NAS under hyperparameter optimization allows for an integration of theoretical understandings from both fields, enhancing the development of future NAS strategies.

Future Directions

Li and Talwalkar suggest that broader community efforts are required to establish reproducibility standards within NAS. This could be achieved through community benchmarks and incentives for open-source releases. Additionally, they highlight potential improvements using more advanced weight-sharing techniques and meta-hyperparameter tuning.

Conclusion

This paper contributes substantively to both the theoretical framing of NAS and practical implementations by providing competitive baselines and highlighting reproducibility issues. It calls for simpler and more robust methodologies that ensure the reliability and accessibility of NAS research. The work sets an imperative for transparency and reproducibility in the field, potentially guiding future developments towards more efficient and understandable NAS practices.