Quantizing deep convolutional networks for efficient inference: A whitepaper (1806.08342v1)

Abstract: We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.

Quantizing Deep Convolutional Networks for Efficient Inference: A Detailed Perspective

In this comprehensive investigation into the quantization of convolutional neural networks (CNNs), Krishnamoorthi offers a thorough exploration of techniques aimed at optimizing inference with integer weights and activations. This methodology gains relevance against the backdrop of deploying deep networks in edge devices, which are typically restricted by computational capabilities and memory resources.

Summary of Findings

Krishnamoorthi's research encapsulates several significant findings which can be broadly categorized into quantization techniques, performance implications, and best practices for quantization-aware training:

1. Post-Training Quantization:
   • Utilizing per-channel quantization for weights and per-layer quantization for activations to 8-bit post-training retains classification accuracies within 2% of their floating-point counterparts for a variety of CNN architectures.
   • Model sizes are significantly reduced by a factor of four through 8-bit weight quantization, independent of the support for 8-bit arithmetic on the deployment hardware.
2. Performance Benchmarks:
   • Quantized networks exhibit a 2x-3x speedup in latency on CPUs. Specialized processors like Qualcomm’s QDSPs with HVX capabilities realize up to 10x speedups for quantized implementations over floating point operations.
3. Quantization-Aware Training (QAT):
   • QAT narrows the accuracy gap to floating-point counterparts to within 1% at 8-bit precision and allows for weight precision reduction to four bits, witnessing accuracy drops between 2% to 10%, varying inversely with network sizes.
   • Extensive validation of quantization-aware training reveals that post-training quantization of weights results in minor accuracy losses, which can be further ameliorated through simulated quantization during the training process.
4. Tools and Techniques:
   • TensorFlow and TensorFlowLite offer practical tools for the quantization of convolutional networks, enabling efficient implementations.
   • Best practices reviewed include handling batch normalization during quantization and employing per-channel quantization as the preferred scheme for enhanced hardware acceleration performance.

Experimental Insights

The research offers detailed empirical analysis through various network architectures such as Mobilenet-V1, Inception-V3, NasNet, and several versions of ResNet. Critical observations reveal:

• Post-training weight-only quantization benefits most from per-channel configurations, with asymmetric quantization providing maximum accuracy near floating-point levels.
• Quantizing both weights and activations moves closer to floating-point accuracy under asymmetric, per-channel quantization schemes.
• Quantization-aware training greatly improves performance, demonstrating that even simpler quantization schemes like per-layer quantization can achieve near-floating-point accuracy.

Additionally, the paper explores scenarios utilizing very low bitwidths (e.g., 4-bit quantization) and demonstrates that substantial accuracy restoration is achievable through fine-tuning.

Practical and Theoretical Implications

Practically, the findings assert that edge devices equipped with quantized models can achieve significant computational and memory efficiencies without substantial compromise in model performance. This positions quantization as a vital tool for deploying deep learning models in real-time environments and resource-constrained settings.

Theoretically, the research calls for embracing more aggressive model compression techniques—regularizing models' dynamic ranges, adopting per-layer and per-channel quantization adaptively, and exploring lower precision formats (e.g., 4-bit quantization). It accentuates the essential role of QAT in reducing the precision-related performance gap, asserting its potential to push the boundaries of what can be achieved with low-precision arithmetic.

Future Directions and Recommendations

Future developments in AI hardware and model architecture optimizations could further benefit from this research. Recommendations include:

• Hardware accelerators should support diverse precisions (4, 8, and 16 bits) and optimized operator fusions to maximize throughput and minimize power consumption.
• Investigations into regularization techniques, distilled training methods, and reinforcement learning for per-layer precision allocations can provide deeper insights and further enhancements in model quantization.

By providing robust performance benchmarks and validated best practices, Krishnamoorthi's work offers a valuable framework for enhancing the efficiency of CNN inference through quantization. This aligns perfectly with the trajectory toward leaner, faster, and more power-efficient AI deployments across various domains.