ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks (2205.08119v3)

Abstract: Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are capable yet power hungry, impeding their more extensive deployment into resource-constrained devices. As such, multiplication-free networks, which follow a common practice in energy-efficient hardware implementation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free networks usually under-perform their vanilla counterparts in terms of the achieved accuracy. To this end, this work advocates hybrid NNs that consist of both powerful yet costly multiplications and efficient yet less powerful operators for marrying the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that incorporates both multiplication-based and multiplication-free operators for facilitating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different operators that follow heterogeneous distributions (e.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to a largely reduced supernet size and much better searched networks. Extensive experiments and ablation studies on various models, datasets, and tasks consistently validate the efficacy of ShiftAddNAS, e.g., achieving up to a +7.7% higher accuracy or a +4.9 better BLEU score compared to state-of-the-art NN, while leading to up to 93% or 69% energy and latency savings, respectively. Codes and pretrained models are available at https://github.com/RICE-EIC/ShiftAddNAS.

ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks

The continuous advancement in neural networks (NNs), including convolutional neural networks (CNNs) and transformers, has driven significant improvements in computational performance, but at the cost of intensive multiplication operations, which are energy-demanding. The incorporation of these computationally heavy operations in NNs restricts their deployment in resource-constrained environments like edge devices. The paper "ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks" presents a novel approach to mitigate these limitations by leveraging both multiplication-based and multiplication-free operations, thus optimizing accuracy and energy efficiency.

Overview of the ShiftAddNAS Framework

The core innovation presented in the paper is the ShiftAddNAS framework, which focuses on hybrid neural architecture search (NAS) by integrating both multiplication-heavy and multiplication-free operations. The framework automates the search for configurations that are both accurate and resource-efficient. The architecture search space is composed of four building blocks:

1. Multiplication-Based Blocks:
   • Convolution (Conv): Excels at capturing local-context information.
   • Attention (Attn): Specializes in global-context modeling through dot-product computations.
2. Multiplication-Free Blocks:
   • Shift: Implements efficient operations by utilizing bitwise shifts, though limited in expressiveness.
   • Addition (Add): Leveraging simple additions for computation, AdderNets utilize this to minimize power consumption.

To facilitate NAS in this hybrid architecture space, ShiftAddNAS employs a unique weight-sharing strategy. Instead of constraining all blocks to share identical weights—which could lead to biased results due to differing weight distribution characteristics—the framework proposes a heterogeneous weight sharing strategy. This allows each operator to maintain its distinct mathematical distribution (Gaussian for convolutions and Laplacian for adders), thus preserving architecture search integrity.

Empirical Results and Implications

The empirical analyses demonstrate ShiftAddNAS's capability in terms of both NLP (e.g., WMT'14 En-Fr, En-De) and CV tasks (ImageNet classification). The results showcase significant improvements in BLEU scores and Top-1 accuracy over existing NAS methodologies and handcrafted architectures. Notably, ShiftAddNAS achieves up to a 7.7% increase in accuracy with impressive energy savings—93% for NLP tasks and 69% for CV tasks. These statistics verify the framework's efficacy in balancing performance and efficiency.

ShiftAddNAS opens several avenues for future research and practical implementation:

• NAS Adaptation: Its hybrid approach may inspire more nuanced exploration into resource-aware network design, particularly crucial for edge-compute ecosystems where power and latency considerations are paramount.
• Weight Distribution Modeling: The method’s ability to handle heterogeneous weight distributions may lead to further refinements for effectively mapping complex network behaviors in novel settings.
• Expanding Search Spaces: The framework can be adapted to various domains beyond NLP and CV, potentially influencing fields that require precise classification with energy constraints.

Conclusion

ShiftAddNAS presents a comprehensive approach for constructing neural architectures that balance the dual objectives of computational accuracy and energy efficiency. By exploring hardware-inspired alternatives and facilitating efficient NAS in hybrid architectures, the paper advances the trajectory towards scalable and deployable AI solutions in constrained environments. It stands as a pertinent contribution to the ongoing development of neural architecture search and efficient AI design paradigms.