Deep Learning with Low Precision by Half-wave Gaussian Quantization (1702.00953v1)

Abstract: The problem of quantizing the activations of a deep neural network is considered. An examination of the popular binary quantization approach shows that this consists of approximating a classical non-linearity, the hyperbolic tangent, by two functions: a piecewise constant sign function, which is used in feedforward network computations, and a piecewise linear hard tanh function, used in the backpropagation step during network learning. The problem of approximating the ReLU non-linearity, widely used in the recent deep learning literature, is then considered. An half-wave Gaussian quantizer (HWGQ) is proposed for forward approximation and shown to have efficient implementation, by exploiting the statistics of of network activations and batch normalization operations commonly used in the literature. To overcome the problem of gradient mismatch, due to the use of different forward and backward approximations, several piece-wise backward approximators are then investigated. The implementation of the resulting quantized network, denoted as HWGQ-Net, is shown to achieve much closer performance to full precision networks, such as AlexNet, ResNet, GoogLeNet and VGG-Net, than previously available low-precision networks, with 1-bit binary weights and 2-bit quantized activations.

• The paper presents HWGQ, a novel approach that leverages Gaussian statistics to quantize ReLU activations while maintaining stable gradient propagation.
• It employs batch normalization and specialized backward approximations (vanilla, clipped, and log-tailed ReLU) to optimize low-precision training.
• Experiments on ImageNet and CIFAR-10 demonstrate that HWGQ-Net closely matches full-precision performance while significantly cutting computational and memory demands.

Overview of "Deep Learning with Low Precision by Half-wave Gaussian Quantization"

The paper under discussion introduces a novel approach to quantizing activations in deep neural networks, titled "Deep Learning with Low Precision by Half-wave Gaussian Quantization" (HWGQ). The primary motivation is to address the inefficiencies associated with the memory and computational demands of deploying large neural networks, such as AlexNet and ResNet, in resource-constrained environments. The authors propose a method that provides an efficient trade-off between accuracy and quantization, enabling substantial reductions in model size and computational intensity.

Quantization Methods Explored

Quantization in neural networks typically involves two dimensions: weights and activations. While weight quantization has achieved some success with binary and low-bit schemes, activation quantization remains challenging due to the non-differentiable nature of quantization operators, which hampers gradient-based optimization.

The paper critiques binary quantization strategies that rely on approximating the hyperbolic tangent non-linearity with piecewise constant and linear functions. These methods, such as the binary sign and the hard tanh functions, lead to performance degradation due to weak gradient signals during backpropagation. Instead, the authors propose an alternative approach that leverages the Rectified Linear Unit (ReLU) non-linearity, common in deep learning due to its strong gradient properties.

Half-wave Gaussian Quantization (HWGQ)

A key contribution of the paper is the introduction of HWGQ for the approximation of ReLU activations. This method optimizes quantization by utilizing the statistical properties of activations, specifically by assuming a Gaussian distribution. The HWGQ is implemented by normalizing the dot products using batch normalization, ensuring consistent quantization across layers without learning-specific quantization parameters.

The authors propose different backward approximation functions to mitigate gradient mismatch, including the vanilla, clipped, and log-tailed ReLU. These prevent optimization instability by addressing the issue of large gradient discrepancies, particularly for outliers.

Experimental Results

The HWGQ-Net is evaluated against several popular architectures, such as AlexNet, ResNet, GoogLeNet, and VGG, showing that it achieves performance close to full-precision models while using binary weights and 2-3 bits for activations. Specifically, it surpasses state-of-the-art binary networks like XNOR-Net and DOREFA-Net, reducing accuracy gaps significantly.

In experiments on both ImageNet and CIFAR-10 datasets, the HWGQ-Net demonstrates competitive results, confirming its effectiveness across network types and tasks. The results highlight that HWGQ-Net offers a robust solution for low-precision neural network implementation, granting it potential usability in real-world applications with limited computational resources.

Implications and Future Directions

The findings underscore the relevance of activation quantization in advancing low-precision deep learning. By addressing the challenges of gradient propagation in quantized networks, HWGQ-Net opens avenues for deploying complex models on edge devices and other constrained platforms.

Future research could explore the application of HWGQ in other neural architectures, assess the impact of different statistical assumptions on the quantization process, and continue refining backward approximation strategies to further close the performance gap with full-precision networks. Innovations in this field will likely contribute to the broader adoption of deep learning technologies across varied domains given their operational and deployment efficiencies.