Regularized Evolution for Image Classifier Architecture Search (1802.01548v7)

Abstract: The effort devoted to hand-crafting neural network image classifiers has motivated the use of architecture search to discover them automatically. Although evolutionary algorithms have been repeatedly applied to neural network topologies, the image classifiers thus discovered have remained inferior to human-crafted ones. Here, we evolve an image classifier---AmoebaNet-A---that surpasses hand-designs for the first time. To do this, we modify the tournament selection evolutionary algorithm by introducing an age property to favor the younger genotypes. Matching size, AmoebaNet-A has comparable accuracy to current state-of-the-art ImageNet models discovered with more complex architecture-search methods. Scaled to larger size, AmoebaNet-A sets a new state-of-the-art 83.9% / 96.6% top-5 ImageNet accuracy. In a controlled comparison against a well known reinforcement learning algorithm, we give evidence that evolution can obtain results faster with the same hardware, especially at the earlier stages of the search. This is relevant when fewer compute resources are available. Evolution is, thus, a simple method to effectively discover high-quality architectures.

• The paper introduces a regularized evolution algorithm that promotes diversity by selecting younger genotypes to explore neural architecture search spaces.
• It presents AmoebaNet-A, which achieves top-1 accuracy of 83.9% and top-5 accuracy of 96.6% on ImageNet, outperforming manually designed models.
• Controlled comparisons show that the evolutionary method outperforms reinforcement learning and random search, offering faster and more efficient architecture discovery.

Analyzing Regularized Evolution for Image Classifier Architecture Search

The paper "Regularized Evolution for Image Classifier Architecture Search" by Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le introduces an evolutionary algorithm-based method for discovering neural network architectures that outperforms manually crafted models and other existing architecture search techniques. The primary contributions include the development of the AmoebaNet-A model, the introduction of a regularized evolution algorithm, and controlled comparisons with alternative methods to highlight the efficacy of this approach.

Overview of Contributions

Regularized Evolution Algorithm

The authors propose a modified version of the tournament selection algorithm, termed "aging evolution" or "regularized evolution." This variation introduces an age property to the genotypes, favoring the selection of younger genotypes during the evolutionary process. This form of regularization mitigates premature convergence and enhances the exploration of the search space, ensuring a broader diversity of architectures. The evolutionary search is conducted within the NASNet search space, defined by convolutional neural network architectures represented as directed acyclic graphs with defined mutation rules.

AmoebaNet-A Architecture

AmoebaNet-A is the most notable outcome of this evolutionary strategy, achieving a state-of-the-art top-1 accuracy of 83.9% and top-5 accuracy of 96.6% on the ImageNet dataset. When scaled to larger sizes, AmoebaNet-A significantly surpasses the performance of manually designed models and those discovered using reinforcement learning (RL) techniques. This demonstrates the potential of evolutionary algorithms to automatically discover high-performing architectures with fewer computational resources.

Controlled Comparisons with RL and Random Search

In a rigorous comparison against RL and random search (RS) methods, the regularized evolution algorithm exhibited superior performance, particularly in the early stages of the search process. Evolution attained higher accuracy faster, emphasizing its suitability for scenarios with limited computational resources. The evolutionary approach provided substantial improvements in terms of both search efficiency and final model quality.

Experimental Findings and Numerical Results

AmoebaNet-A's performance is validated through extensive experiments on CIFAR-10 and ImageNet datasets. The scaled model, trained with advanced techniques on ImageNet, achieves a top-1 accuracy of 83.9% and top-5 accuracy of 96.6%, setting a new benchmark at the time of release. The paper also emphasizes the simplicity and effectiveness of the regularized evolution algorithm, which, despite its simplicity, outperformed more complex methods.

Implications and Future Work

The implications of this research are profound, suggesting that evolutionary algorithms, with appropriate modifications, can rival and even surpass other architecture search methods like RL. The regularized evolution approach provides a robust framework for future research in neural architecture search (NAS), particularly for applications requiring efficient exploration of vast search spaces with limited computational budgets.

Future work could explore the application of this method to other domains and more complex datasets. Additionally, further investigation into the theoretical underpinnings of aging evolution could yield deeper insights into its regularization effects and optimization behaviors in noisy environments. Expanding the scope to include more sophisticated search spaces and hybrid approaches that combine evolutionary strategies with other automated techniques could further enhance the capabilities of NAS algorithms.

Conclusion

"Regularized Evolution for Image Classifier Architecture Search" offers a compelling alternative to traditional architecture design and existing automated search methods. The introduction of regularized evolution, exemplified through the development of the AmoebaNet-A model, highlights the potential of evolutionary algorithms to drive significant advancements in neural network performance. This paper sets a precedent for future research and application in the domain of automated neural architecture search.