Training Competitive Binary Neural Networks from Scratch (1812.01965v1)

Abstract: Convolutional neural networks have achieved astonishing results in different application areas. Various methods that allow us to use these models on mobile and embedded devices have been proposed. Especially binary neural networks are a promising approach for devices with low computational power. However, training accurate binary models from scratch remains a challenge. Previous work often uses prior knowledge from full-precision models and complex training strategies. In our work, we focus on increasing the performance of binary neural networks without such prior knowledge and a much simpler training strategy. In our experiments we show that we are able to achieve state-of-the-art results on standard benchmark datasets. Further, to the best of our knowledge, we are the first to successfully adopt a network architecture with dense connections for binary networks, which lets us improve the state-of-the-art even further.

• The paper introduces a direct training approach for binary neural networks from scratch without using full-precision pre-training.
• It enhances network performance by incorporating dense shortcut connections and eliminating bottlenecks, achieving up to a 3.2% Top-1 accuracy boost on ImageNet.
• The method simplifies deployment on low-power devices by significantly reducing storage and computational requirements while maintaining competitive accuracy.

Training Competitive Binary Neural Networks from Scratch

The paper "Training Competitive Binary Neural Networks from Scratch" addresses the challenge of training binary neural networks (BNNs) without the reliance on full-precision models and intricate training methodologies. BNNs are notable for their potential to execute efficient inference on low-power devices by using binary weights, offering a storage compression factor of 32× compared to their full-precision counterparts.

Key Contributions and Methodology

The primary contribution of this research is the establishment of a straightforward yet effective training regimen for BNNs, free from prior full-precision model knowledge. The authors showcase the potential of BNNs by achieving state-of-the-art results on widely recognized datasets such as MNIST, CIFAR-10, and ImageNet. Moreover, they introduce a pioneering approach to incorporate dense connections within binary architectures, which significantly enhances performance metrics.

Three main strategies to improve binary model accuracy were identified: avoiding bottleneck designs, augmenting shortcut connections within the network, and selectively replacing certain layers with full-precision layers. These techniques are particularly crucial for preserving the information flow, which is vital given the inherently lower bit density in BNNs.

Empirical Evidence and Results

The authors demonstrate empirically that their approach can match or exceed the accuracy of existing full-precision models. Specific findings include the efficacy of removing bottlenecks and increasing shortcut connections, resulting in notable accuracy improvements. Additionally, they note that scaling factors commonly utilized in fine-tuned networks from full-precision models do not yield the same benefits when training BNNs from scratch.

Notably, the redesigned DenseNet architecture, referred to as DenseNetE, was split to provide more connections, which further enhanced performance. The paper reports superior results using this architecture on the ImageNet dataset, with improvements up to 3.2% in Top-1 accuracy compared to traditional architectures.

Implications and Future Work

The implications of this research are significant for the deployment of machine learning solutions on devices with limited computational capabilities and restricted power consumption. The ability to train competitive binary networks from scratch simplifies deployment pipelines by removing the need for initially training full-precision models, thereby reducing time and resource expenditures.

Future directions highlighted by the authors include the exploration of theoretical approaches to better understand layer importance within networks for information preservation. Such understanding could foster the development of new architectures optimized for binary quantization, further closing the accuracy gap between binary and full-precision networks.

Conclusion

This paper offers a robust methodology for training binary neural networks from scratch, challenging the landscape of model efficiency and deployment by demonstrating exemplary accuracy without traditional dependencies. It lays a foundational approach that may inspire further research to refine and expand the capabilities of BNNs, especially within resource-constrained environments.