Designing Neural Network Architectures using Reinforcement Learning

Abstract: At present, designing convolutional neural network (CNN) architectures requires both human expertise and labor. New architectures are handcrafted by careful experimentation or modified from a handful of existing networks. We introduce MetaQNN, a meta-modeling algorithm based on reinforcement learning to automatically generate high-performing CNN architectures for a given learning task. The learning agent is trained to sequentially choose CNN layers using $Q$-learning with an $\epsilon$-greedy exploration strategy and experience replay. The agent explores a large but finite space of possible architectures and iteratively discovers designs with improved performance on the learning task. On image classification benchmarks, the agent-designed networks (consisting of only standard convolution, pooling, and fully-connected layers) beat existing networks designed with the same layer types and are competitive against the state-of-the-art methods that use more complex layer types. We also outperform existing meta-modeling approaches for network design on image classification tasks.

• The paper introduces MetaQNN, a reinforcement learning framework that automatically selects CNN layers to optimize performance.
• It models the CNN design process as a Markov Decision Process using ε-greedy exploration and experience replay for efficient learning.
• Experimental results on CIFAR-10, SVHN, and MNIST show that MetaQNN achieves state-of-the-art accuracy and effective transfer learning.

Designing Neural Network Architectures Using Reinforcement Learning

The paper "Designing Neural Network Architectures Using Reinforcement Learning" by Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar addresses the growing necessity of automating the design process of Convolutional Neural Networks (CNNs). Given the complex and labor-intensive nature of manual CNN design, the authors propose an innovative approach termed MetaQNN, which leverages reinforcement learning to autonomously generate high-performing CNN architectures for image classification tasks.

Abstract and Introduction

CNN architecture design traditionally requires extensive human expertise and iterative experimentation. The paper posits that the enormous design space of possible architectures renders an exhaustive manual search infeasible. Their solution, MetaQNN, employs a $Q$-learning agent to sequentially select CNN layers from a discretized and finite set of possible configurations. The agent receives the validation accuracy of the proposed architecture as a reward, enabling it to optimize its design approach iteratively.

Methodology

State and Action Space Definition

The core of MetaQNN's methodology lies in modeling the layer selection process as a Markov Decision Process (MDP). The state space is defined to include all relevant parameters of the CNN layers:

• Convolutional layers characterized by the number of filters, receptive field size, stride, and representation size.
• Pooling layers defined similarly, but excluding consecutive pooling actions to maintain experimental tractability.
• Fully-connected layers constrained to have a maximum of two consecutive layers to limit the number of parameters.
• Termination layers, either global average pooling or softmax.

The actions the agent may take are appropriately restricted to ensure that the state-action graph is Directed Acyclic (DAG).

Reinforcement Learning Framework

MetaQNN employs an $\epsilon$-greedy strategy for exploration and exploitation, gradually reducing $\epsilon$ to transition from random to more deterministic exploration based on learned $Q$-values. Experience replay is utilized to stabilize and expedite the learning process. The learning rate and discount factor for $Q$-learning are set carefully to balance the incorporation of new information and long-term rewards.

Training Procedure

The researchers use a consistent yet aggressive training scheme for all models during the exploration phase to ensure efficiency. For final evaluations, the top performing models identified during exploration are fine-tuned with a more extensive training schedule.

Experimental Results

Across Datasets

MetaQNN is evaluated on three standard image classification datasets: CIFAR-10, SVHN, and MNIST. The results demonstrate that the architectures discovered by MetaQNN outperform existing networks crafted with similar types of layers and compete favorably against state-of-the-art models using more complex layer types. In addition, MetaQNN outperforms previous automated network design techniques significantly.

Numerical Analysis

Statistical metrics indicate the increasing efficacy of model selection as $\epsilon$ decreases:

• On CIFAR-10, the best model designed by MetaQNN achieved a test error of 6.92%, with validation errors consistently reduced through the training iterations.
• SVHN experiments showed mean accuracy improvements from 52.25% at $\epsilon=1$ to 88.02% at $\epsilon=0.1%.
• In the MNIST dataset, the ensemble of ten MetaQNN top models achieved a test error of 0.28%, surpassing existing benchmarks without data augmentation.

Implications and Future Directions

MetaQNN represents a significant stride toward scalable neural network design automation, suitable for a wide array of tasks beyond image classification. Its reinforcement learning framework allows for the adaptation to various optimization constraints, such as model size and inference speed. Furthermore, integrating hyperparameter optimization can augment its efficacy.

The inherent ability of MetaQNN-designed architectures to transfer effectively across different tasks underscores its flexibility. For instance, the best CIFAR-10 model trained directly on SVHN and MNIST demonstrated competitive performance metrics, illustrating its robustness for transfer learning scenarios.

Concluding Remarks

This work establishes a foundational approach to automated neural network design, aligning with the broader goal of making deep learning accessible and efficient for varied applications. While specific to CNNs, the methodology holds potential for adaptation across diverse network architectures and learning paradigms. Future research could focus on expanding the state-action space and integrating this framework with real-time, adaptive training environments.