A White Paper on Neural Network Quantization (2106.08295v1)

Abstract: While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.

An Analysis of "A White Paper on Neural Network Quantization"

The paper "A White Paper on Neural Network Quantization" presents an exhaustive exploration of techniques for quantizing neural networks, targeting the reduction of computational cost during inference. Given the expanding application of neural networks in power-constrained environments, such as edge devices, the research discusses two primary methods of quantization: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). Both methodologies aim to decrease the bit-width of weights and activations, thereby reducing memory usage and speeding up computations on fixed-point hardware.

Quantization Techniques and Their Benefits

The research commences with a detailed justification for quantization from a hardware perspective, explaining how matrix multiplications benefit from the reduced precision of computations. The significant reduction in data transfer and arithmetic complexity, as discussed in the paper, underscores the potential energy savings and performance gains. The authors describe various quantization schemes, including uniform affine, symmetric, and power-of-two quantizations, while emphasizing practical hardware constraints related to these methods.

Post-Training Quantization (PTQ)

PTQ methods do not require retraining or high compute resources, as they operate on pre-trained FP32 networks. Critical to PTQ is effective range setting, where the paper evaluates multiple methods like min-max, mean squared error (MSE), and cross-entropy-based range selection to balance clipping and rounding errors. The presented PTQ pipeline culminates in a sequence of steps involving CLE and AdaRound optimization for low-bit weight quantization, leading to performance close to full-precision models. The paper reports that 8-bit weight and activation quantization can be achieved with minimal accuracy loss across various models and tasks, including ImageNet and GLUE benchmarks.

Quantization-Aware Training (QAT)

For scenarios where PTQ does not suffice, QAT offers an alternative by simulating quantization during training, allowing models to adapt to quantization noise. The incorporation of the Straight-Through Estimator (STE) addresses non-differentiability within the back-propagation process. Special attention is given to batch normalization folding, ensuring efficient inference simulations. While more computationally intensive than PTQ, QAT facilitates lower precision, achieving 4-bit quantization with competitive accuracy. The distinct advantage of QAT lies in optimizing both the weights and quantization parameters during the retraining process.

Implications and Future Directions

Quantization emerges as a critical step in enabling neural networks to operate on embedded devices and scenarios demanding real-time processing with limited resources. The proposed methods, particularly when integrated into efficient hardware, promise substantial reductions in latency and energy consumption without a significant accuracy trade-off. Looking forward, the development of adaptive quantization techniques and better hardware support for mixed-precision computations could further enhance the applicability and performance of quantized neural models.

In summary, this paper makes a significant contribution to the field by providing a pragmatic approach to deploying quantized networks. With its comprehensive investigation into PTQ and QAT methodologies, the research successfully navigates the complexities of neural network quantization, presenting robust solutions that extend the utility of deep learning models in edge computing environments.