Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment (2208.04321v2)

Abstract: The ongoing advancements in network architecture design have led to remarkable achievements in deep learning across various challenging computer vision tasks. Meanwhile, the development of neural architecture search (NAS) has provided promising approaches to automating the design of network architectures for lower prediction error. Recently, the emerging application scenarios of deep learning have raised higher demands for network architectures considering multiple design criteria: number of parameters/floating-point operations, and inference latency, among others. From an optimization point of view, the NAS tasks involving multiple design criteria are intrinsically multiobjective optimization problems; hence, it is reasonable to adopt evolutionary multiobjective optimization (EMO) algorithms for tackling them. Nonetheless, there is still a clear gap confining the related research along this pathway: on the one hand, there is a lack of a general problem formulation of NAS tasks from an optimization point of view; on the other hand, there are challenges in conducting benchmark assessments of EMO algorithms on NAS tasks. To bridge the gap: (i) we formulate NAS tasks into general multi-objective optimization problems and analyze the complex characteristics from an optimization point of view; (ii) we present an end-to-end pipeline, dubbed $\texttt{EvoXBench}$, to generate benchmark test problems for EMO algorithms to run efficiently -- without the requirement of GPUs or Pytorch/Tensorflow; (iii) we instantiate two test suites comprehensively covering two datasets, seven search spaces, and three hardware devices, involving up to eight objectives. Based on the above, we validate the proposed test suites using six representative EMO algorithms and provide some empirical analyses. The code of $\texttt{EvoXBench}$ is available from $\href{https://github.com/EMI-Group/EvoXBench}{\rm{here}}$.

• The paper formalizes NAS tasks as multiobjective optimization problems by balancing prediction error, model complexity, and inference latency.
• The paper introduces EvoXBench, an end-to-end benchmark pipeline that uses surrogate models to evaluate EMO algorithms without heavy GPU dependency.
• The paper empirically assesses six EMO algorithms on diverse test suites, highlighting performance gaps in complex neural search spaces.

Neural Architecture Search as Multiobjective Optimization Benchmarks

This paper addresses the complexities of Neural Architecture Search (NAS) when viewed through the lens of multiobjective optimization. The authors provide a comprehensive formulation of NAS as a multiobjective problem, detailing the different objectives that can be considered, such as prediction error, model complexity, and hardware performance. From an optimization standpoint, modern NAS tasks involve balancing these multiple conflicting objectives, which naturally lends itself to the application of Evolutionary Multiobjective Optimization (EMO) algorithms.

The paper identifies several critical gaps that have previously limited the application of EMO to NAS. It emphasizes the lack of a broadly accepted problem formulation and the scarcity of benchmark assessments to evaluate EMO methods effectively. To address these gaps, the paper outlines three main contributions:

1. Problem Formulation: The paper formalizes NAS tasks into multiobjective optimization problems, involving objectives such as prediction error, the number of parameters, and inference latency. It highlights complex characteristics such as multi-modal fitness landscapes, noisy objectives, and degenerate Pareto fronts that arise from the intrinsic properties of NAS tasks.
2. EvoXBench Benchmark: The paper introduces EvoXBench, an end-to-end pipeline designed to generate benchmark test problems specifically for EMO algorithms without the need for GPUs or deep learning frameworks like PyTorch or TensorFlow. EvoXBench simplifies the incorporation of various search spaces, datasets, and hardware by using surrogate models and lightweight evaluations, allowing researchers to focus on algorithm development rather than computational resource constraints.
3. Test Suite Generation and Evaluation: Two sizable benchmark test suites, C-10/MOP and IN-1K/MOP, were developed. These suites encompass several search spaces and objectives tailored for EMO, covering a range of complexities from bi-objective to many-objective problems. The suite includes convolutional and Transformer architectures, highlighting applicability across different neural network designs. The paper documents an empirical paper validating these test suites with six representative EMO algorithms, revealing their strengths and limitations.

The implications of this research are multifaceted. Practically, the introduction of EvoXBench allows for efficient testing of EMO algorithms in NAS, fostering better comparison and advancement. Theoretically, the work pushes forward the boundaries of understanding how multiobjective dynamics interact in NAS scenarios. The procedural focus on surrogate modeling within NAS highlights the trend of employing approximate evaluations to mitigate the traditionally prohibitive computational demands of NAS.

Looking to the future, the authors suggest that EvoXBench could be extended to support additional search spaces and more complex tasks like semantic segmentation. Such extensions could further establish the benchmark's utility and adaptability in various deep learning contexts. Moreover, more advanced and computationally efficient EMO algorithms could be developed leveraging insights from the rich optimization problems presented by NAS tasks, ultimately contributing to the broader AI community's goal of creating more efficient and versatile neural networks.