APQ: Joint Search for Network Architecture, Pruning and Quantization Policy (2006.08509v1)

Abstract: We present APQ for efficient deep learning inference on resource-constrained hardware. Unlike previous methods that separately search the neural architecture, pruning policy, and quantization policy, we optimize them in a joint manner. To deal with the larger design space it brings, a promising approach is to train a quantization-aware accuracy predictor to quickly get the accuracy of the quantized model and feed it to the search engine to select the best fit. However, training this quantization-aware accuracy predictor requires collecting a large number of quantized <model, accuracy> pairs, which involves quantization-aware finetuning and thus is highly time-consuming. To tackle this challenge, we propose to transfer the knowledge from a full-precision (i.e., fp32) accuracy predictor to the quantization-aware (i.e., int8) accuracy predictor, which greatly improves the sample efficiency. Besides, collecting the dataset for the fp32 accuracy predictor only requires to evaluate neural networks without any training cost by sampling from a pretrained once-for-all network, which is highly efficient. Extensive experiments on ImageNet demonstrate the benefits of our joint optimization approach. With the same accuracy, APQ reduces the latency/energy by 2x/1.3x over MobileNetV2+HAQ. Compared to the separate optimization approach (ProxylessNAS+AMC+HAQ), APQ achieves 2.3% higher ImageNet accuracy while reducing orders of magnitude GPU hours and CO2 emission, pushing the frontier for green AI that is environmental-friendly. The code and video are publicly available.

• The paper introduces a unified framework that jointly optimizes network architecture, pruning, and quantization to enhance model accuracy and efficiency.
• It employs a quantization-aware accuracy predictor with progressive shrinking, achieving a 2.3% ImageNet accuracy improvement while reducing latency and energy consumption.
• Empirical validation on ImageNet demonstrates that APQ cuts latency by 2x and energy use by 1.3x, supporting sustainable and cost-effective AI deployment.

An Insightful Overview of "APQ: Joint Search for Network Architecture, Pruning and Quantization Policy"

The paper, "APQ: Joint Search for Network Architecture, Pruning and Quantization Policy," introduces a novel methodology that integrates neural architecture search (NAS), pruning, and quantization in a unified framework for efficient deep learning model deployment on resource-constrained hardware. Typical approaches isolate these optimization processes, potentially leading to suboptimal models when applied sequentially due to each stage's inherent peculiarities. This research seeks to concurrently optimize all three facets, permitting an end-to-end approach that jointly refines accuracy, latency, and energy efficiency.

Methodological Innovations

In the field of automated machine learning (AutoML), the idea proposed is to employ a quantization-aware accuracy predictor. This predictor estimates a model's accuracy with varying architectures and quantization policies, significantly curtailing the need for exhaustive trials or evaluations. A noteworthy aspect of this technique is the concept of predictor transfer. The authors suggest initializing a quantization-aware accuracy predictor using a pretrained full-precision predictor, which is then refined using a smaller set of quantization-aware accuracy data. This approach effectively harnesses knowledge from a prior full-precision domain to accurately predict outcomes in the quantized domain, thereby enhancing data efficiency.

The paper details the construction of a once-for-all network, encompassing a vast search space that can effectively answer the joint optimization challenge. Each sub-network is honed to deliver competitive performance without retraining, thanks to a progressive shrinking methodology. Consequently, this infrastructure allows the immediate evaluation of sub-network permutations and quantization strategies without additional computational overhead.

Empirical Validation

Comprehensive experiments on the ImageNet dataset highlight APQ's ability to outperform existing state-of-the-art NAS methods, in terms of both environmental sustainability and resource efficiency. The implementation illustrated in this paper achieves a reduction in both latency and energy consumption by factors of 2 and 1.3, respectively, compared to the MobileNetV2+HAQ system. In contrast, APQ improves ImageNet accuracy by 2.3% against a conventional separation approach, showcasing its superior exploration of the joint design space. Furthermore, it achieves these results with significantly reduced computational resources, marking a step forward for sustainable practices in artificial intelligence.

Implications and Future Directions

This method's ecological implication is particularly significant as it implies a reduction in CO$_2$ emissions during the model design process, aligning well with the emergent concerns surrounding green AI. By minimizing GPU hours and energy consumption, APQ supports sustainability and reduces the financial and ecological cost of deploying efficient deep learning models.

In theoretical terms, APQ's integration of NAS, pruning, and quantization might set a new standard for future research, advocating a shift towards comprehensive search spaces that accommodate multiple optimization routes simultaneously. Such integration could foster the development of even more efficient models where the boundaries between architecture-level and post-processing optimizations blur.

Looking forward, further research may investigate extending this joint optimization approach to other model aspects, beyond pruning and quantization, such as automatic hyperparameter tuning or batch-size optimization. Also, exploration in diversified hardware environments could provide insights into the adaptability and robustness of such integrated strategies.

In conclusion, the contributions of the paper are notable for their methodological advancements and the potential ripple effects they could invoke in the field of efficient deep learning. APQ represents a substantial step toward making AI deployment not only more efficient but also more environmentally conscious. This approach might well be a precursor to more consolidated methodologies in AI pipeline optimization, fostering advancements in both research and practical applications.