A Survey on Methods and Theories of Quantized Neural Networks (1808.04752v2)

Abstract: Deep neural networks are the state-of-the-art methods for many real-world tasks, such as computer vision, natural language processing and speech recognition. For all its popularity, deep neural networks are also criticized for consuming a lot of memory and draining battery life of devices during training and inference. This makes it hard to deploy these models on mobile or embedded devices which have tight resource constraints. Quantization is recognized as one of the most effective approaches to satisfy the extreme memory requirements that deep neural network models demand. Instead of adopting 32-bit floating point format to represent weights, quantized representations store weights using more compact formats such as integers or even binary numbers. Despite a possible degradation in predictive performance, quantization provides a potential solution to greatly reduce the model size and the energy consumption. In this survey, we give a thorough review of different aspects of quantized neural networks. Current challenges and trends of quantized neural networks are also discussed.

• The paper offers an extensive review of quantization methods, including deterministic and stochastic approaches for efficient model compression.
• The paper details techniques for quantizing weights, activations, and gradients, demonstrating significant reductions in memory usage and computational cost.
• The paper discusses future research directions to enhance the robustness and generalizability of quantized neural networks in resource-constrained environments.

An Overview of Quantized Neural Networks: Techniques, Challenges, and Future Directions

The paper, "A Survey on Methods and Theories of Quantized Neural Networks," authored by Yunhui Guo, provides an extensive review of the methods and theories behind quantized neural networks (QNNs). As deep neural networks (DNNs) are acclaimed for their state-of-the-art performance across domains like computer vision, natural language processing, and speech recognition, they also face criticism for significant memory consumption and the corresponding energy overhead. This paper gives an in-depth exposition on how quantization offers a tangible solution by reducing model sizes and energy requirements through the representation of weights, activations, and gradients in lower precision formats.

Overview of Quantization Methods

The paper delineates two broad categories of quantization: deterministic and stochastic approaches.

1. Deterministic Quantization: This includes methods like rounding and vector quantization, which utilize exact mappings or clustering to compress model parameters. Techniques such as BinaryConnect and XNOR-Net exemplify deterministic quantization, facilitating binary or ternary weight representations to reduce memory and computational demands.
2. Stochastic Quantization: Stochastic methods leverage random rounding and probabilistic approaches where parameters are treated as random variables sampled from discrete distributions. This adds noise resembling regularization, potentially aiding generalizability.

Adaptive methods go beyond fixed codebook quantization by tailoring codebooks according to the data distribution, thus enhancing flexibility and performance.

Quantization of Network Components

The scope extends to three primary network components—weights, activations, and gradients:

• Weights: Quantization of weights significantly reduces model size and increases computational efficiency. Techniques like Incremental Network Quantization illustrate how weight partitioning and selective retraining ensure minimal performance loss.
• Activations: Techniques to quantize activations involve replacing floating-point operations with bitwise operations, lowering computational costs. Challenges include managing gradient propagation through non-differentiable quantization functions.
• Gradients: In distributed environments, gradient quantization reduces communication overhead. Techniques like TernGrad evidence substantial bandwidth savings, though maintaining convergence proves challenging.

Implications and Theoretical Perspectives

Quantized neural networks facilitate the deployment of DNNs in resource-constrained environments like mobile and embedded devices. Although quantized models generally approach or match the performance of their full-precision counterparts, the theoretical underpinnings remain under-explored. This calls for a deeper understanding of convergence properties and implications on model expressiveness.

Practical Implications and Future Directions

The emergence of quantized networks illustrates a forward trajectory towards leaner models capable of efficient computation without substantial loss in accuracy. Future research is expected to focus on:

• Developing enhanced quantization techniques for richer tasks like NLP and complex architectures such as RNNs.
• Establishing theoretical analyses to guide quantization strategies and ensure robust learning processes in reduced precision frameworks.
• Addressing specific applications beyond classification tasks, extending quantization benefits across diverse use-cases.

In conclusion, this paper systematically unveils the landscape of quantized neural networks, revealing both the potential and the inherent complexities in creating efficient, compressed, and accurate models suitable for the wide-scale application of deep learning. The path forward involves not only refining existing methods but also pioneering new solutions that address the nuanced requirements of emerging computational paradigms.