Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks (1903.08066v3)

Abstract: We propose a method of training quantization thresholds (TQT) for uniform symmetric quantizers using standard backpropagation and gradient descent. Contrary to prior work, we show that a careful analysis of the straight-through estimator for threshold gradients allows for a natural range-precision trade-off leading to better optima. Our quantizers are constrained to use power-of-2 scale-factors and per-tensor scaling of weights and activations to make it amenable for hardware implementations. We present analytical support for the general robustness of our methods and empirically validate them on various CNNs for ImageNet classification. We are able to achieve near-floating-point accuracy on traditionally difficult networks such as MobileNets with less than 5 epochs of quantized (8-bit) retraining. Finally, we present Graffitist, a framework that enables automatic quantization of TensorFlow graphs for TQT (available at https://github.com/Xilinx/graffitist ).

Analysis of "Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks"

The paper "Trained Quantization Thresholds (TQT) for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks" introduces a novel approach to enhancing the accuracy and efficiency of deep neural network (DNN) inference through strategic quantization. The method challenges conventional static thresholding techniques by implementing a dynamic system where quantization thresholds are optimized as trainable parameters using backpropagation and gradient descent.

Methodology and Contributions

Key to the paper's methodology is the introduction of learnable quantization thresholds, which are optimized concurrently with the DNN's weights. This dual-training process enables the model to better adapt the quantization range to the data distribution encountered during training. The paper elaborates on the use of uniform symmetric quantizers, confining the scale factors to power-of-2 values. This constraint aids in aligning the quantization process with hardware operations, as power-of-2 scaling uses simple bit-shifts instead of computationally expensive floating-point multiplication.

The authors empirically validate their approach on several convolutional neural networks (CNNs) such as MobileNets over the ImageNet dataset. They demonstrate that their quantization method closely approximates floating-point accuracy while significantly reducing the computational load, evidenced by their ability to retrain these networks in under five epochs with 8-bit quantization, achieving accuracy close to precise floating-point computations.

One aspect of the paper that stands out is the analytical and empirical support provided for the robustness of TQT across various networks, especially for models that are traditionally challenging to quantize, such as MobileNets. Additionally, the paper highlights their framework, Graffitist, which automates the quantization process of TensorFlow graphs. This tool reflects their broader contribution to providing accessible quantization solutions for machine learning applications requiring fixed-point arithmetic.

Empirical Results

The results presented indicate a strong advantage of TQT over existing methods, with the quantized MobileNets achieving ImageNet top-1 accuracy of 71.1%, equivalent to the floating-point baseline. Tables in the paper consistently show superior or equivalent performance of the TQT method compared to Google's Quantization-Aware Training (QAT) despite TQT's stricter constraints.

Theoretical Implications

TQT challenges the traditional assumption that fixed quantization thresholds are optimal, advancing the discussion towards dynamic threshold adaptation based on network-wide loss gradients. This method demonstrates how joint optimization of threshold and weights can enable better quantization precision and dynamic range, leading to improved inference performance.

Practical Implications

By reducing the dependency on floating-point operations in deep learning inference, TQT opens the door to deploying DNNs on more resource-constrained environments such as edge devices. The use of the Graffitist framework for automating TQT processes could accelerate the adoption and integration of DNN models on fixed-point hardware platforms, offering a pathway towards more efficient model deployment.

Future Developments

While the paper is comprehensive, it invites further exploration, especially in scenarios where constraints such as power-of-2 scaling are relaxed or where more aggressive quantization is desired. Extensions of TQT to accommodate asymmetrical or non-uniform quantization schemes may further enhance its applicability, especially in scenarios requiring extremely low-bitwidth quantization.

In summary, the proposed trained quantization thresholds method fundamentally enhances the approach to neural network quantization, offering significant improvements in maintaining accuracy while enabling efficient deployment on resource-scarce hardware. This work not only advances the technical understanding of quantization processes but also provides a practical framework for its application in the industry.