A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities (2205.10358v1)

Abstract: Recent advances in Neural Architecture Search (NAS) such as one-shot NAS offer the ability to extract specialized hardware-aware sub-network configurations from a task-specific super-network. While considerable effort has been employed towards improving the first stage, namely, the training of the super-network, the search for derivative high-performing sub-networks is still under-explored. Popular methods decouple the super-network training from the sub-network search and use performance predictors to reduce the computational burden of searching on different hardware platforms. We propose a flexible search framework that automatically and efficiently finds optimal sub-networks that are optimized for different performance metrics and hardware configurations. Specifically, we show how evolutionary algorithms can be paired with lightly trained objective predictors in an iterative cycle to accelerate architecture search in a multi-objective setting for various modalities including machine translation and image classification.

• The paper demonstrates that integrating evolutionary algorithms with lightly-trained predictors can significantly accelerate neural architecture search.
• It outlines a framework that optimizes sub-network selection by balancing latency and accuracy with hardware-specific considerations.
• Experimental results highlight notable computational savings and enhanced adaptability, paving the way for scalable hardware-aware NAS systems.

A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities

The paper, titled "A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities," presents a compelling framework targeting the optimization of Neural Architecture Search (NAS) with an emphasis on hardware efficiency. The authors recognize the existing duopoly in NAS methodologies, where substantial enhancement efforts are inclined mainly towards the training of super-networks, while the pertinent phase of searching for optimal sub-network configurations particularly tailored to specific hardware remains inadequately explored.

Methodology Overview

The proposed framework diverges from traditional approaches by integrating evolutionary algorithms with objective predictors. This integration facilitates an efficient search for high-performance architectures optimized for various evaluation metrics and hardware considerations. The novelty is encapsulated in the adaptive search mechanism leveraging lightly-trained predictors in tandem with evolutionary processes. This framework iteratively refines the architecture search, balancing the computational expense against multiple objectives, which include, but are not limited to, latency and accuracy, essential for tasks such as machine translation and image classification.

Experimental Results

The experimental evaluation underscores the robustness and versatility of the proposed framework. The results indicate that the incorporation of evolutionary algorithms significantly enhances the capability of NAS to yield architecture configurations that are not only performance-efficient but also hardware-adaptive. While specific numerical results are not articulated, the methodology promises substantial computational savings and higher adaptability across different hardware settings, a notable advancement in the field.

Implications and Future Directions

The implications of this research are twofold: First, it offers a reduction in computational overhead commonly associated with exhaustive search phases in NAS by using pre-trained predictors and evolutionary strategies. Second, it proposes a potentially scalable approach to hardware-aware NAS, which can inform future research and development of NAS platforms that are chiefly constrained by hardware limitations.

The paper opens avenues for future exploration, particularly in enhancing the fidelity of objective predictors and refining the evolutionary algorithms to assimilate a broader spectrum of hardware metrics. Anticipated developments may include improved synergy between hardware design and NAS, fostering architectures that intrinsically comprehend the constraints and affordances of underlying hardware.

Conclusively, this framework could serve as a baseline for subsequent studies focusing on hardware-constrained neural network deployment, facilitating a more nuanced understanding of the trade-offs in performance, efficiency, and scalability within NAS processes across diverse application domains.