Towards Effective Low-bitwidth Convolutional Neural Networks (1711.00205v2)

Abstract: This paper tackles the problem of training a deep convolutional neural network with both low-precision weights and low-bitwidth activations. Optimizing a low-precision network is very challenging since the training process can easily get trapped in a poor local minima, which results in substantial accuracy loss. To mitigate this problem, we propose three simple-yet-effective approaches to improve the network training. First, we propose to use a two-stage optimization strategy to progressively find good local minima. Specifically, we propose to first optimize a net with quantized weights and then quantized activations. This is in contrast to the traditional methods which optimize them simultaneously. Second, following a similar spirit of the first method, we propose another progressive optimization approach which progressively decreases the bit-width from high-precision to low-precision during the course of training. Third, we adopt a novel learning scheme to jointly train a full-precision model alongside the low-precision one. By doing so, the full-precision model provides hints to guide the low-precision model training. Extensive experiments on various datasets ( i.e., CIFAR-100 and ImageNet) show the effectiveness of the proposed methods. To highlight, using our methods to train a 4-bit precision network leads to no performance decrease in comparison with its full-precision counterpart with standard network architectures ( i.e., AlexNet and ResNet-50).

Effective Strategies for Training Low-bitwidth Convolutional Neural Networks

The paper "Towards Effective Low-bitwidth Convolutional Neural Networks" explores the methodologies for optimizing deep convolutional neural networks (CNNs) with low-precision weights and activations. This is of significant importance since CNNs with millions of parameters and immense computational requirements pose challenges for deployment on resource-constrained devices like mobile hardware and specialized chips. The research addresses the prevalent issue of substantial accuracy loss due to poor local minima encountered during low-precision network training.

The authors propose three primary strategies that can be applied independently or in combination to enhance the training of low-bitwidth networks:

1. Two-Stage Optimization: The proposed two-stage optimization strategy involves a sequential process where initially only the weights are quantized, leaving activations in full precision. Once an adequate model is achieved, the activations are subsequently quantized, and the model is retrained. This progressive approach aims to find a well-conditioned initialization for the final low-precision model by first addressing a simpler subproblem.
2. Progressive Quantization: Following the notion of annealing, this strategy involves gradually reducing the bitwidth from high precision to the target low precision over the course of training. By doing so, the model incrementally adjusts to lower precision, which alleviates the optimization difficulty resulting from abrupt precision changes.
3. Guided Training with a Full-Precision Model: Expanding on techniques such as knowledge distillation, this approach involves training a low-precision network in tandem with a full-precision counterpart. The full-precision model, which is adapted alongside the target network, aids the low-precision model by acting as a guiding benchmark, thereby facilitating a more stable and informed gradient descent process.

The effectiveness of these strategies is substantiated through extensive experiments on CIFAR-100 and ImageNet using architectures like AlexNet and ResNet-50. Notably, the 4-bit precision networks demonstrated no performance decline compared to full-precision models, with some even surpassing the latter in classification accuracy.

The implications of this research are profound for the deployment of deep learning models on low-resource hardware, making these models practical for real-world applications where computational and memory efficiency are critical. The authors’ methods show that it is feasible to maintain model accuracy even when significantly reducing precision, highlighting a pathway towards more energy-efficient and faster inference mechanisms.

Looking forward, this research opens avenues for further exploration into the quantization of other types of neural networks and the development of more sophisticated quantization schemes. It also paves the way for examining the interplay between network architecture design and precision scaling, potentially leading to new hybrid models that dynamically adjust precision levels based on computational constraints or inference accuracy requirements. The integration of these strategies with emerging neural network architectures could yield robust models that are inherently suited for efficient deployment across a variety of platforms.