AutoSlim: Towards One-Shot Architecture Search for Channel Numbers (1903.11728v2)

Abstract: We study how to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot solution, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop. By this single pass, we can obtain the optimized channel configurations under different resource constraints. We present experiments with MobileNet v1, MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We show significant improvements over their default channel configurations. We also achieve better accuracy than recent channel pruning methods and neural architecture search methods. Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2 (301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3% better accuracy than MobileNet-v1 (569M FLOPs). Code and models will be available at: https://github.com/JiahuiYu/slimmable_networks

• The paper presents a one-shot methodology using a single slimmable network to estimate optimal channel configurations, cutting computational costs.
• It demonstrates a top-1 accuracy boost, with AutoSlim-MobileNet-v2 achieving 74.2% at 305M FLOPs, outperforming default and RL-based models.
• The approach offers a practical alternative to traditional pruning and reinforcement learning methods for efficient deployment on resource-limited devices.

AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

This essay provides an analytical overview of the paper, "AutoSlim: Towards One-Shot Architecture Search for Channel Numbers" by Jiahui Yu and Thomas Huang. The paper introduces AutoSlim, a method focused on optimizing channel configurations in neural networks to improve accuracy under specific resource constraints such as FLOPs, latency, memory footprint, or model size. AutoSlim offers a notable advancement over traditional pruning and reinforcement learning approaches in architectural search by adopting a one-shot methodology.

The significance of channel configuration in neural networks, particularly for deployment on resource-constrained platforms, forms the core foundation of the research. Traditional methods rely heavily on heuristics or reinforcement learning strategies that are computationally intensive, often requiring exhaustive training of multiple network configurations. AutoSlim diverges from this paradigm by training a single slimmable network, which acts as a performance estimator for various channel configurations. This strategic shift promises considerable reductions in computational costs while maintaining or improving performance metrics.

Key results from the paper are noteworthy. AutoSlim demonstrates substantial improvements over default configurations across several neural network architectures, including MobileNet v1, MobileNet v2, ResNet-50, and MNasNet, with experiments conducted on ImageNet classification. For instance, AutoSlim-MobileNet-v2 achieved a top-1 accuracy of 74.2% at 305M FLOPs, improving by 2.4% over the default MobileNet-v2 and surpassing RL-searched MNasNet by 0.2%. The results indicate superior speed-accuracy trade-offs when employing the optimized channel configurations derived from AutoSlim.

The implications of AutoSlim are both practical and theoretical. Practically, AutoSlim offers a streamlined approach to network architecture search, allowing for faster deployment of efficient models suitable for mobile and edge devices. Theoretically, it challenges the necessity of extensive multi-sample training and indicates potential for further one-shot optimization processes in neural network design. Furthermore, the research opens avenues for evaluating the transferability and robustness of AutoSlim-derived architectures across a range of datasets and tasks.

Future developments in AI could see AutoSlim and similar methodologies applied to varying layers and components beyond channel numbers, potentially improving architecture search across different paradigms such as layer type and connectivity. The research may also stimulate exploration into adaptive models that can dynamically adjust architectures in real-time based on the constraints and requirements of emerging applications.

In conclusion, AutoSlim marks a significant step towards efficient architecture search methodologies, emphasizing the role of slimmable networks and one-shot strategies in optimizing neural networks under resource constraints. The methodology not only promises computational efficiency but also fosters a paradigm of flexible, adaptive network design suitable for diverse application environments.