Neural Architecture Search for Lightweight Non-Local Networks (2004.01961v1)

Abstract: Non-Local (NL) blocks have been widely studied in various vision tasks. However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks. We propose AutoNL to overcome the above two obstacles. Firstly, we propose a Lightweight Non-Local (LightNL) block by squeezing the transformation operations and incorporating compact features. With the novel design choices, the proposed LightNL block is 400x computationally cheaper} than its conventional counterpart without sacrificing the performance. Secondly, by relaxing the structure of the LightNL block to be differentiable during training, we propose an efficient neural architecture search algorithm to learn an optimal configuration of LightNL blocks in an end-to-end manner. Notably, using only 32 GPU hours, the searched AutoNL model achieves 77.7% top-1 accuracy on ImageNet under a typical mobile setting (350M FLOPs), significantly outperforming previous mobile models including MobileNetV2 (+5.7%), FBNet (+2.8%) and MnasNet (+2.1%). Code and models are available at https://github.com/LiYingwei/AutoNL.

• The paper introduces LightNL blocks that reduce computational costs by 400 times compared to traditional non-local operations.
• It employs differentiable neural architecture search to optimally position LightNL blocks within mobile networks using only 32 GPU hours.
• AutoNL models reach 77.7% top-1 accuracy on ImageNet at 350M FLOPs, outperforming benchmarks like MobileNetV2, FBNet, and MnasNet.

Neural Architecture Search for Lightweight Non-Local Networks

The paper "Neural Architecture Search for Lightweight Non-Local Networks" introduces AutoNL, a novel approach that integrates non-local (NL) operations into mobile neural networks through efficient neural architecture search. Historically, NL blocks have been beneficial in various computer vision tasks for modeling long-range dependencies, but their high computational cost has limited their application in mobile settings. The AutoNL framework seeks to address two challenges: the computational inefficiency of traditional NL blocks and the difficulty of optimally embedding these blocks into mobile architectures.

Contributions

The paper makes several key contributions:

1. Lightweight Non-Local Blocks (LightNL): The introduction of LightNL blocks significantly reduces computational demands—making them 400 times more efficient than traditional NL blocks—without compromising performance. This is achieved by compressing transformation operations and utilizing more compact feature representations.
2. Differentiable Neural Architecture Search: To determine the optimal configuration of LightNL blocks in mobile networks, the authors propose a neural architecture search (NAS) method that relaxes the block structure to be differentiable. This allows the search process to be efficiently conducted in an end-to-end manner, making it computationally feasible (32 GPU hours) compared to prior works.
3. Performance Gains: AutoNL models outperform state-of-the-art mobile models. Notably, they achieve 77.7% top-1 accuracy on ImageNet under a mobile setting of 350M FLOPs, surpassing previous models such as MobileNetV2 by 5.7%, FBNet by 2.8%, and MnasNet by 2.1%.

Implications

• Practical Implications: By significantly enhancing performance without a substantial increase in computational cost, AutoNL makes it feasible to deploy advanced NL-based models on resource-constrained devices such as mobile phones and IoT devices. This advancement expands the applicability of NL blocks across various real-world applications ranging from mobile photography to real-time video processing.
• Theoretical Implications: The approach offers a new perspective on how NAS combined with efficient block design can bridge the gap between performance and computational efficiency. The introduction of LightNL contributes to a broader understanding of how attention mechanisms can be scaled down and effectively leveraged in lightweight neural architectures.

Future Directions

This research opens the door to several future directions in both neural architecture search and the deployment of deep learning models on edge devices. Firstly, exploring additional dimensions of differentiability in NAS could lead to even more efficient search processes. Furthermore, extending this framework to other domains like natural language processing or reinforcement learning may yield similar benefits in terms of computational efficiency and performance gains.

In conclusion, the paper presents a methodically sound and computationally efficient way to amalgamate NL blocks into mobile networks via AutoNL. This work not only enhances the efficiency of vision models in resource-constrained environments but also lays the groundwork for future research in advanced NAS methodologies for various deep learning architectures.