Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network (2403.03739v1)

Published 6 Mar 2024 in cs.LG and cs.AI

Abstract: Binary neural networks utilize 1-bit quantized weights and activations to reduce both the model's storage demands and computational burden. However, advanced binary architectures still incorporate millions of inefficient and nonhardware-friendly full-precision multiplication operations. A&B BNN is proposed to directly remove part of the multiplication operations in a traditional BNN and replace the rest with an equal number of bit operations, introducing the mask layer and the quantized RPReLU structure based on the normalizer-free network architecture. The mask layer can be removed during inference by leveraging the intrinsic characteristics of BNN with straightforward mathematical transformations to avoid the associated multiplication operations. The quantized RPReLU structure enables more efficient bit operations by constraining its slope to be integer powers of 2. Experimental results achieved 92.30%, 69.35%, and 66.89% on the CIFAR-10, CIFAR-100, and ImageNet datasets, respectively, which are competitive with the state-of-the-art. Ablation studies have verified the efficacy of the quantized RPReLU structure, leading to a 1.14% enhancement on the ImageNet compared to using a fixed slope RLeakyReLU. The proposed add&bit-operation-only BNN offers an innovative approach for hardware-friendly network architecture.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
  2. K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
  3. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  4. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016.
  5. Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured scenarios. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2778–2788, 2021.
  6. Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829–7833. IEEE, 2020.
  7. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4960–4964. IEEE, 2016.
  8. Improving language understanding by generative pre-training. 2018.
  9. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  10. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  11. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4):485–532, 2020.
  12. A four-megabit compute-in-memory macro with eight-bit precision based on cmos and resistive random-access memory for ai edge devices. Nature Electronics, 4(12):921–930, 2021.
  13. Deep learning in spiking neural networks. Neural networks, 111:47–63, 2019.
  14. A review of learning in biologically plausible spiking neural networks. Neural Networks, 122:253–272, 2020.
  15. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607–617, 2019.
  16. Neuro-inspired computing chips. Nature electronics, 3(7):371–382, 2020.
  17. Brain-inspired computing needs a master plan. Nature, 604(7905):255–260, 2022.
  18. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.
  19. Xnor-net: Imagenet classification using binary convolutional neural networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV, pages 525–542. Springer, 2016.
  20. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
  21. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  22. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pages 722–737, 2018.
  23. Learning architectures for binary networks. In European conference on computer vision, pages 575–591. Springer, 2020.
  24. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 143–159. Springer, 2020.
  25. ” bnn-bn=?”: Training binary neural networks without batch normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4619–4629, 2021.
  26. Characterizing signal propagation to close the performance gap in unnormalized resnets. In International Conference on Learning Representations, 2020.
  27. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pages 1059–1071. PMLR, 2021.
  28. Accurate and compact convolutional neural networks with trained binarization. In 30th British Machine Vision Conference (BMVC 2019), 2019.
  29. Xnor-net++: Improved binary neural networks. arXiv preprint arXiv:1909.13863, 2019.
  30. True gradient-based training of deep binary activated neural networks via continuous binarization. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 2346–2350. IEEE, 2018.
  31. Circulant binary convolutional networks: Enhancing the performance of 1-bit dcnns with circulant back propagation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2691–2699, 2019.
  32. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2250–2259, 2020.
  33. Sparsity-inducing binarized neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 12192–12199, 2020.
  34. Regularizing activation distribution for training binarized deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11408–11417, 2019.
  35. Bimodal-distributed binarized neural networks. Mathematics, 10(21):4107, 2022.
  36. Bnn+: Improved binary network training. 2018.
  37. Improving accuracy of binary neural networks using unbalanced activation distribution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7862–7871, 2021.
  38. Equal bits: Enforcing equally distributed binary network weights. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 1491–1499, 2022.
  39. How do adam and training strategies help bnns optimization. In International conference on machine learning, pages 6936–6946. PMLR, 2021.
  40. Training binary neural networks with real-to-binary convolutions. In International Conference on Learning Representations, 2019.
  41. Lipschitz continuity retained binary neural network. In European conference on computer vision, pages 603–619. Springer, 2022.
  42. Bayesian optimized 1-bit cnns. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4909–4917, 2019.
  43. Incremental network quantization: Towards lossless cnns with low-precision weights. In International Conference on Learning Representations, 2016.
  44. Addernet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1468–1477, 2020.
  45. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310–1318. Pmlr, 2013.
  46. Regularizing and optimizing lstm language models. In International Conference on Learning Representations, 2018.
  47. Learning multiple layers of features from tiny images. Master’s thesis, University of Tront, 2009.
  48. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211–252, 2015.
  49. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  50. A review of binarized neural networks. Electronics, 8(6):661, 2019.
  51. Identity mappings in deep residual networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 630–645. Springer, 2016.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets