Towards the Limit of Network Quantization (1612.01543v2)

Abstract: Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively.

Towards the Limit of Network Quantization

The paper "Towards the Limit of Network Quantization" addresses the challenge of compressing deep neural networks through network quantization, a process crucial for deploying such models in resource-constrained environments, such as mobile or edge devices. The authors, Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee, propose advanced quantization schemes that reduce the model size while minimizing the performance loss, by introducing techniques informed by information theory and optimization.

Network quantization is achieved by reducing the diversity of parameter values in deep neural networks, which inherently decreases the storage requirements. The authors aim to optimize the balance between the compression ratio and the performance degradation, introducing Hessian-weighted k-means clustering to measure and minimize quantization errors effectively. The core idea is to employ the Hessian matrix of the loss function — specifically, its diagonal elements — to weight the importance of each parameter's quantization. The Hessian provides a second-order approximation of the loss function that captures parameter sensitivity, allowing more informed clustering during quantization.

When addressing the characteristics of network quantization, the authors explore two main contributions:

1. Hessian-Weighted Distortion Measure: They establish the Hessian-weighted distortion as an objective for optimizing network quantization under a given compression ratio constraint. This approach adjusts for the varying sensitivities of parameters, unlike conventional methods such as k-means clustering, which treat errors uniformly.
2. Entropy-Constrained Scalar Quantization (ECSQ): The paper relates network quantization to ECSQ in information theory, positing that once network parameters are quantized, further compression can be achieved through optimal variable-length coding, like Huffman coding. The authors propose solutions to the ECSQ problem through uniform quantization and an iterative technique similar to Lloyd's algorithm.

In their experiments, conducted on LeNet, ResNet, and AlexNet, the authors demonstrate the efficacy of their methods through significant compression ratios while maintaining accuracy. Notably, they achieve compression ratios of 51.25, 22.17, and 40.65 for LeNet, ResNet, and AlexNet, respectively, using uniform quantization followed by Huffman coding. This indicates that their approach effectively minimizes performance losses, with reduced computational overhead compared to previous benchmark methods. The paper's results particularly highlight the benefit of quantizing all network layers simultaneously using Hessian-weighting, a strategy that addresses inter-layer variance in parameter sensitivity.

Additionally, the authors discuss the practical calculation of the Hessian matrix's diagonal elements, which can be efficiently approximated using existing techniques equivalent in complexity to gradient computation. Alternatively, for models trained with the Adam optimizer, the square root of the second moment estimates of gradients can be used, eliminating additional computational costs for Hessian approximation.

Implications of this paper are substantial for compressing neural networks effectively for deployment in limited-resource environments. It opens avenues for integrating more sophisticated quantization and coding methodologies to achieve higher efficiency. Future work may extend to other model architectures and explore further integration with adaptive optimization techniques, potentially impacting the development of more compact and scalable AI models that maintain robust performance across diverse applications.