Towards Accurate Binary Convolutional Neural Network (1711.11294v1)

Abstract: We introduce a novel scheme to train binary convolutional neural networks (CNNs) -- CNNs with weights and activations constrained to {-1,+1} at run-time. It has been known that using binary weights and activations drastically reduce memory size and accesses, and can replace arithmetic operations with more efficient bitwise operations, leading to much faster test-time inference and lower power consumption. However, previous works on binarizing CNNs usually result in severe prediction accuracy degradation. In this paper, we address this issue with two major innovations: (1) approximating full-precision weights with the linear combination of multiple binary weight bases; (2) employing multiple binary activations to alleviate information loss. The implementation of the resulting binary CNN, denoted as ABC-Net, is shown to achieve much closer performance to its full-precision counterpart, and even reach the comparable prediction accuracy on ImageNet and forest trail datasets, given adequate binary weight bases and activations.

Overview of "Towards Accurate Binary Convolutional Neural Network"

The paper presents a novel method for training binary Convolutional Neural Networks (CNNs), designated as ABC-Net, which aims to bridge the accuracy gap between full-precision and binary networks. This innovation is particularly crucial given the increasing demand for deploying CNNs on mobile devices, where memory efficiency and computational speed are significant constraints.

Key Innovations

The authors introduce two primary innovations in binarizing CNNs:

1. Binary Weight Bases: Full-precision weights are approximated by linear combinations of multiple binary weight bases. This approach significantly mitigates the accuracy degradation typically observed in binary networks. The paper demonstrates that using between 3 to 5 binary weight bases suffices to closely approximate full-precision weights.
2. Multiple Binary Activations: To address the challenge of binarized activations, which often incurs higher accuracy losses than binarizing weights, the authors propose employing multiple binary activations. This strategy has been shown to narrow the accuracy gap in complex tasks like ImageNet classification.

Numerical Results

The paper reports strong numerical results, notably on the ImageNet dataset using the ResNet-18 architecture. With five binary weight bases and five binary activations, ABC-Net achieves a Top-1 accuracy of 65.0% and a Top-5 accuracy of 85.9%, placing it comparably close to its full-precision counterpart, which exhibits a Top-1 accuracy of 69.3%.

Implications

The implications of this work are significant for the development of hardware-efficient deep learning models. By reducing the need for high-precision arithmetic operations through binary approximations, ABC-Net offers a path toward more energy-efficient and faster test-time inference, particularly suitable for mobile and edge computing environments. This binarization method holds potential benefits for specialized deep learning hardware, which could leverage the efficient bitwise operations enabled by binary networks.

Future Directions

Future research could explore the extension of this binary scheme to other complex tasks beyond image classification, such as object detection or recurrent neural network models. Additionally, the implementation of ABC-Net on FPGA or ASIC platforms could be further investigated to fully realize the benefits of reduced computation and resource usage during inference.

In summary, the paper presents a robust approach to binary CNNs, narrowing the accuracy gap with full-precision models while enhancing computational efficiency. The method’s potential for widespread application in mobile and low-power devices makes it an important contribution to the field of deep learning.