Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation (2111.14826v2)

Abstract: The nonuniform quantization strategy for compressing neural networks usually achieves better performance than its counterpart, i.e., uniform strategy, due to its superior representational capacity. However, many nonuniform quantization methods overlook the complicated projection process in implementing the nonuniformly quantized weights/activations, which incurs non-negligible time and space overhead in hardware deployment. In this study, we propose Nonuniform-to-Uniform Quantization (N2UQ), a method that can maintain the strong representation ability of nonuniform methods while being hardware-friendly and efficient as the uniform quantization for model inference. We achieve this through learning the flexible in-equidistant input thresholds to better fit the underlying distribution while quantizing these real-valued inputs into equidistant output levels. To train the quantized network with learnable input thresholds, we introduce a generalized straight-through estimator (G-STE) for intractable backward derivative calculation w.r.t. threshold parameters. Additionally, we consider entropy preserving regularization to further reduce information loss in weight quantization. Even under this adverse constraint of imposing uniformly quantized weights and activations, our N2UQ outperforms state-of-the-art nonuniform quantization methods by 0.5~1.7 on ImageNet, demonstrating the contribution of N2UQ design. Code and models are available at: https://github.com/liuzechun/Nonuniform-to-Uniform-Quantization.

An Analysis of Nonuniform-to-Uniform Quantization with Generalized Straight-Through Estimation

In recent years, the efficiency of Deep Neural Networks (DNNs) has been a pertinent focus in model deployment, particularly for environments with constrained computational resources. The paper "Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation" explores an innovative approach to quantization— a method employed to reduce computational cost and memory usage through bit-reduction of weights and activations in DNNs. This paper introduces Nonuniform-to-Uniform Quantization (N2UQ), addressing the challenges of conventional quantization methods in both performance and practicality.

Technical Contributions

N2UQ is engineered to retain the adeptness of nonuniform quantization while ensuring the simplicity of uniform quantization in hardware applications. The core advancement lies in the quantization strategy: while traditional methods either employ uniform quantization, which fails to adapt effectively to varied data distributions, or nonuniform quantization, which entails costly post-processing steps, N2UQ introduces learnable input thresholds. These thresholds enable flexible adaptation to underlying distributions, thus improving representation accuracy without incurring significant hardware overhead.

Key to N2UQ’s implementation is the Generalized Straight-Through Estimator (G-STE), a novel backward approximation method that enhances training by accurately computing gradients for nonuniform-to-uniform quantizers. This method extends the straightforward estimation (STE) technique commonly used in quantized networks but compensates for its limitations by enabling adaptive gradient calculation regarding the learnable threshold parameters.

Furthermore, the authors propose an entropy-preserving regularization that seeks to maximize the entropy of weight distribution across quantization levels. This regularization ensures that weight distributions fully utilize available quantization levels, thereby minimizing information loss and enhancing model accuracy.

Empirical Evaluation

The empirical evaluation of N2UQ is conducted on the comprehensive ImageNet dataset, utilizing architectures such as ResNet and MobileNet. The authors demonstrate that N2UQ, irrespective of bit widths, achieves superior accuracy compared to both state-of-the-art uniform and nonuniform quantization methods. For instance, a 2-bit quantized ResNet-50 obtained a remarkably close accuracy (76.4% top-1 accuracy) to its full-precision counterpart, narrowing the gap to just 0.6%.

Implications and Future Directions

The implications of this paper are multifaceted. Practically, N2UQ offers a quantization method that combines low-level hardware costs akin to uniform quantization with the robust performance of nonuniform quantization strategies. This presents beneficial advantages for deploying neural networks on resource-constrained devices such as mobile platforms and edge devices.

Theoretically, the introduction of G-STE provides a new perspective on how gradient estimation can be approached in quantized networks, suggesting broader applications beyond just weight and activation quantization. The derivation of G-STE hints at potential improvements across other neural network operations where non-differentiability and thresholding operations are involved.

In conclusion, N2UQ addresses significant limitations in the current state of quantized networks by harmonizing representational flexibility with hardware efficiency. The paper's methods foster potential for enhanced large-scale deployment of machine learning models in diverse environments, and future work could explore alternative architectures and higher bit quantization as directions to further this paper's impact.