Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss (1808.05779v3)

Abstract: Reducing bit-widths of activations and weights of deep networks makes it efficient to compute and store them in memory, which is crucial in their deployments to resource-limited devices, such as mobile phones. However, decreasing bit-widths with quantization generally yields drastically degraded accuracy. To tackle this problem, we propose to learn to quantize activations and weights via a trainable quantizer that transforms and discretizes them. Specifically, we parameterize the quantization intervals and obtain their optimal values by directly minimizing the task loss of the network. This quantization-interval-learning (QIL) allows the quantized networks to maintain the accuracy of the full-precision (32-bit) networks with bit-width as low as 4-bit and minimize the accuracy degeneration with further bit-width reduction (i.e., 3 and 2-bit). Moreover, our quantizer can be trained on a heterogeneous dataset, and thus can be used to quantize pretrained networks without access to their training data. We demonstrate the effectiveness of our trainable quantizer on ImageNet dataset with various network architectures such as ResNet-18, -34 and AlexNet, on which it outperforms existing methods to achieve the state-of-the-art accuracy.

Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss

This paper addresses the pressing issue of computational and memory efficiency in deploying deep neural networks (DNNs) on resource-constrained devices by introducing a novel approach to quantization. Specifically, it presents a method for quantizing both activations and weights in deep networks through a trainable quantizer that optimizes quantization intervals based on task loss. This approach is designed to preserve the accuracy of full-precision networks even at low bit-widths, such as 4-bit, and to minimize the accuracy degradation at further reduced bit-widths (3 and 2-bit).

Key Contributions

The authors make three primary contributions to the field of network quantization:

1. Parameterized Quantization Intervals: They propose a trainable quantizer that parameterizes quantization intervals, allowing simultaneous pruning and clipping of weights and activations. By focusing only on the most significant values, the model efficiently minimizes the quantization error.
2. Joint Optimization with Network Weights: The framework jointly optimizes the quantizers along with the network weights for task-specific loss, all within an end-to-end model training process. This approach contrasts with traditional methods which often minimize the quantization error as a separate step.
3. Empirical Validation of State-of-the-Art Results: Empirical results on the ImageNet dataset with various architectures (ResNet-18, -34, and AlexNet) demonstrate that their quantization method achieves state-of-the-art accuracies at very low bit-width configurations. Importantly, the method can maintain high performance even when trained on a heterogeneous dataset and applied to pretrained networks, which offers practical versatility.

Numerical Results

The experimental results underpin the method's effectiveness. Notably, the 4-bit and 3-bit quantized networks achieved accuracy levels on par with full-precision 32-bit networks, showcasing minimal performance loss. For instance, the 4/4-bit ResNet-34 model achieved a top-1 ImageNet accuracy of 73.7\%, identical to that of the full precision model. Even at a more aggressive quantization level of 2/2-bit, the model maintained strong performance, with only a modest accuracy drop.

Implications and Future Directions

This paper's implications for real-world applications are profound, particularly for deploying deep models in environments where computational resources are scarce, such as mobile and edge devices. The ability to preserve model accuracy while reducing storage and computational demands significantly broadens the horizon for efficient AI applications.

From a theoretical perspective, the adaptive quantization strategy also stimulates further inquiry into how networks can self-optimize different training parameters autonomously and under varying constraints. Future research directions may focus on extending this framework to other model architectures, developing even more granular parameterization of the quantization process, and integrating Bayesian methods for quantization. Additionally, exploring the intersection of quantization with network pruning and structured sparsity could offer further insights into model compression without compromising performance.

Overall, this paper represents a strategic advance in the quantitative handling of DNNs, setting a benchmark for the efficient construction and deployment of deep learning models. As machine learning continues to permeate resource-constrained contexts, methods such as those presented in this paper will become increasingly pivotal.