Training Binary Neural Networks with Real-to-Binary Convolutions (2003.11535v1)

Abstract: This paper shows how to train binary networks to within a few percent points ($\sim 3-5 \%$) of the full precision counterpart. We first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at https://github.com/brais-martinez/real2binary.

• The paper introduces a novel real-to-binary attention matching strategy that aligns binary network outputs with those of real-valued networks using teacher-student pairs.
• It proposes a data-driven channel re-scaling method that leverages real-valued activations to enhance binary network performance in resource-constrained settings.
• Empirical results show that the method reduces the accuracy gap to less than 5% on ImageNet and 3% on CIFAR-100, achieving state-of-the-art performance.

Training Binary Neural Networks with Real-to-Binary Convolutions

The paper "Training Binary Neural Networks with Real-to-Binary Convolutions" presents an innovative approach to narrow the accuracy gap between binary neural networks (BNNs) and their full-precision counterparts. It demonstrates how binary networks can be optimized to achieve a performance close to that of real-valued networks, potentially making them viable alternatives for deployment in resource-constrained environments.

Major Contributions

The authors make several key contributions to the field of binary neural networks:

1. Strong Baseline for BNNs: The paper establishes a robust baseline for binary networks by combining recent methodological insights and applying rigorous optimization techniques. This baseline already achieves state-of-the-art accuracy on ImageNet, marking a significant advancement over previous benchmarks.
2. Real-to-Binary Attention Matching: The authors propose a novel attention matching strategy, where spatial attention maps from a binary network are aligned with those of a real-valued network. This technique is applied progressively through a series of teacher-student networks to minimize architectural discrepancies, which significantly improves training outcomes.
3. Data-Driven Channel Re-Scaling: A new approach is introduced to enhance the representational power of binary networks by using data-driven scaling factors. This involves employing real-valued activations to compute scaling factors, thereby optimizing the re-scaling mechanism beyond fixed pre-trained parameters.
4. Empirical Performance: The proposed methods achieve impressive results, reporting a reduction of the accuracy gap to less than 5% on ImageNet and 3% on CIFAR-100, using the ResNet-18 architecture. This is a notable reduction from the typical gap observed with previous state-of-the-art techniques.

Technical Insights and Results

The paper outlines how carefully modulating and guiding the training process of BNNs can lead to substantial performance improvements. Key ingredients in this success include the use of progressive architectural modifications through teacher-student learning pairs, and the novel application of real-to-binary attention matching, which ensures effective optimization by aligning the binary network's output more closely with a real-valued reference. Importantly, the integration of data-dependent scaling factors introduces an adaptive mechanism that enhances the network’s ability to handle diverse inputs more effectively than traditional fixed scaling approaches.

Practical and Theoretical Implications

Practically, the advances in this paper enable the deployment of binary networks on devices with limited computational resources without severely compromising accuracy. This could expand the applicability of neural networks in contexts where power efficiency and computational simplicity are paramount. Theoretically, this work challenges the perception of binary networks as merely approximate models by demonstrating their capacity to closely match the accuracy of full-precision models when adequately guided and optimized.

Future Directions

Moving forward, the techniques outlined in this paper could be extended and refined to further close the accuracy gap or apply similar principles to other architectures and modalities. Investigation into more advanced teacher-student configurations, improved scaling mechanisms, or generalized attention matching frameworks could yield additional gains.

This paper substantiates the potential of binary neural networks as efficient yet powerful models, suggesting that with thoughtful design and optimization, BNNs can approximate their real-valued counterparts in performance, thus significantly broadening their applicability.