Progressive Neural Architecture Search (1712.00559v3)

Abstract: We propose a new method for learning the structure of convolutional neural networks (CNNs) that is more efficient than recent state-of-the-art methods based on reinforcement learning and evolutionary algorithms. Our approach uses a sequential model-based optimization (SMBO) strategy, in which we search for structures in order of increasing complexity, while simultaneously learning a surrogate model to guide the search through structure space. Direct comparison under the same search space shows that our method is up to 5 times more efficient than the RL method of Zoph et al. (2018) in terms of number of models evaluated, and 8 times faster in terms of total compute. The structures we discover in this way achieve state of the art classification accuracies on CIFAR-10 and ImageNet.

• The paper introduces a sequential model-based optimization strategy that incrementally builds CNN architectures, achieving up to 5 times speedup and 8 times compute savings over RL methods.
• It leverages a structured search space and heuristic pruning to explore cell architectures, resulting in state-of-the-art performance on CIFAR-10 and ImageNet.
• The study highlights practical benefits in resource efficiency and transferability, paving the way for adaptive, cost-effective neural architecture search in automated machine learning.

Progressive Neural Architecture Search: An Efficient Approach to Learning CNN Structures

Progressive Neural Architecture Search (PNAS) represents an innovative methodology for optimizing the structure of Convolutional Neural Networks (CNNs). The authors, hailing from reputable institutions such as Johns Hopkins University, Google AI, and Stanford University, have presented a method that claims improvements in efficiency over traditional methods like Reinforcement Learning (RL) and Evolutionary Algorithms (EA). This paper elucidates a sequential model-based optimization (SMBO) strategy that incrementally searches for neural architectures while concurrently training a surrogate model to guide the search process.

Methodology

PNAS fundamentally diverges from existing techniques by progressively searching through increasingly complex architectures. At each stage, simpler models are evaluated first, and their performance data is used to inform the construction and evaluation of more complex models. This is facilitated through a learned surrogate model, which predicts the performance of prospective CNN architectures based on their structural attributes without necessitating full training.

From a practical standpoint, the PNAS framework consists of the following steps:

1. Structured Search Space: The search algorithm focuses on discovering efficient "cell" structures, smaller building blocks of CNNs, and stacks these cells to form the full network.
2. Heuristic Search: Beginning with simple models, the algorithm progressively explores more complex architectures while pruning less promising ones at each iteration.
3. Surrogate Model: To predict the performance of proposed architectures without full training, the method employs either an ensemble of Multi-Layer Perceptrons (MLP) or Recurrent Neural Networks (RNN).

The efficacy of this approach is validated by outperforming state-of-the-art RL-based methods in both efficiency and accuracy on benchmark datasets like CIFAR-10 and ImageNet.

Numerical Results

The results presented in the paper highlight significant improvements in search efficiency and accuracy:

• Efficiency: Under comparable conditions, PNAS demonstrates a speedup factor of up to 5 times in terms of models evaluated and up to 8 times in total compute compared to the RL method previously proposed by Zoph et al. (2018).
• Accuracy: PNAS achieves state-of-the-art classification accuracies on CIFAR-10 and ImageNet. Specifically, the PNASNet-5 model, which emerged from this search, achieves notable performance metrics: 3.41% top-1 error on CIFAR-10, 74.2% top-1 accuracy on ImageNet in the mobile setting, and 82.9% top-1 accuracy in the large setting.

These results suggest that PNAS is highly effective at finding efficient and accurate neural architectures with significantly reduced computational resources.

Theoretical and Practical Implications

Theoretical Implications:

• Enhanced Search Models: PNAS refines the search process by leveraging progressive search strategies supported by surrogate models, highlighting the utility of incrementally increasing architectural complexity.
• Predictive Modelling: The surrogate model's role underscores the potential of predicting model performance to save computational effort, paving the way for more sophisticated prediction techniques in future research.

Practical Implications:

• Resource Efficiency: By demonstrating substantial improvements in efficiency, PNAS makes neural architecture search more accessible and cost-effective, particularly in environments with limited computational resources.
• Transferability: The strong correlation between CIFAR-10 performance and ImageNet performance accentuates the transferability of discovered architectures across datasets, reinforcing the strategy of using smaller datasets as proxies for larger ones.

Future Directions

The research suggests several avenues for future exploration:

• Enhanced Predictive Models: Incorporating advanced models such as Gaussian processes or more complex RNNs to further refine the surrogate prediction capabilities.
• Early Stopping Mechanisms: Developing techniques for early termination of unpromising models during training to further optimize computational resources.
• Adaptive Search Parameters: Investigating mechanisms for dynamically adjusting the search parameters, like the number of models evaluated at each search stage, based on real-time performance feedback.

PNAS sets a precedent for efficient neural architecture search and opens up concrete pathways for more extensive and resource-efficient exploration of deep learning architectures. The combination of progressive search techniques with predictive modeling holds promise for future advances in the field of automated machine learning and neural architecture optimization.