Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Comprehensive Benchmarking of Binary Neural Networks on NVM Crossbar Architectures (2308.06227v1)

Published 11 Aug 2023 in cs.ET and cs.AR

Abstract: Non-volatile memory (NVM) crossbars have been identified as a promising technology, for accelerating important machine learning operations, with matrix-vector multiplication being a key example. Binary neural networks (BNNs) are especially well-suited for use with NVM crossbars due to their use of a low-bitwidth representation for both activations and weights. However, the aggressive quantization of BNNs can result in suboptimal accuracy, and the analog effects of NVM crossbars can further degrade the accuracy during inference. This paper presents a comprehensive study that benchmarks BNNs trained and validated on ImageNet and deployed on NeuroSim, a simulator for NVM-crossbar-based PIM architecture. Our study analyzes the impact of various parameters, such as input precision and ADC resolution, on both the accuracy of the inference and the hardware performance metrics. We have found that an ADC resolution of 8-bit with an input precision of 4-bit achieves near-optimal accuracy compared to the original BNNs. In addition, we have identified bottleneck components in the PIM architecture that affect area, latency, and energy consumption, and we demonstrate the impact that different BNN layers have on hardware performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” Int’l Symp. on Computer Architecture (ISCA), 2016.
  2. A. Ankit, I. E. Hajj, S. R. Chalamalasetti, G. Ndu, M. Foltin, R. S. Williams, P. Faraboschi, W.-m. W. Hwu, J. P. Strachan, K. Roy, and D. S. Milojicic, “PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference,” Int’l Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2019.
  3. A. Ankit, I. El Hajj, S. R. Chalamalasetti, S. Agarwal, M. Marinella, M. Foltin, J. P. Strachan, D. Milojicic, W.-M. Hwu, and K. Roy, “Panther: A Programmable Architecture for Neural Network Training Harnessing Energy-Efficient ReRAM,” IEEE Trans. on Computers (TC), vol. 69, no. 8, pp. 1128–1142, 2020.
  4. I. Boybat, M. Le Gallo, S. R. Nandakumar, T. Moraitis, T. Parnell, T. Tuma, B. Rajendran, Y. Leblebici, A. Sebastian, and E. Eleftheriou, “Neuromorphic computing with multi-memristive synapses,” Nature Communications, vol. 9, no. 1, p. 2514, 2018.
  5. J.-M. Hung, C.-J. Jhang, P.-C. Wu, Y.-C. Chiu, and M.-F. Chang, “Challenges and Trends of Nonvolatile In-Memory-Computation Circuits for AI Edge Devices,” IEEE Open Journal of the Solid-State Circuits Society, vol. 1, pp. 171–183, 2021.
  6. S. Yu, “Neuro-Inspired Computing with Emerging Nonvolatile Memorys,” Proceedings of the IEEE, vol. 106, no. 2, pp. 260–285, 2018.
  7. I. Chakraborty, M. Ali, A. Ankit, S. Jain, S. Roy, S. Sridharan, A. Agrawal, A. Raghunathan, and K. Roy, “Resistive Crossbars as Approximate Hardware Building Blocks for Machine Learning: Opportunities and Challenges,” Proceedings of the IEEE, vol. 108, no. 12, pp. 2276–2310, 2020.
  8. Y. Zhang, J. Pan, X. Liu, H. Chen, D. Chen, and Z. Zhang, “FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations,” Int’l Symp. on Field-Programmable Gate Arrays (FPGA), 2021.
  9. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized Neural Networks,” Conf. on Neural Information Processing Systems (NeurIPS), 2016.
  10. Y. Li, T. Geng, A. Li, and H. Yu, “BCNN: Binary Complex Neural Network,” Microprocessors and Microsystems, vol. 87, p. 104359, 2021.
  11. H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N. Sebe, “Binary Neural Networks: A Survey,” Pattern Recognition, vol. 105, p. 107281, 2020.
  12. S. Liang, S. Yin, L. Liu, W. Luk, and S. Wei, “FP-BNN: Binarized Neural Network on FPGA,” Neurocomputing, vol. 275, pp. 1072–1086, 2018.
  13. X. Lin, C. Zhao, and W. Pan, “Towards Accurate Binary Convolutional Neural Network,” Conf. on Neural Information Processing Systems (NeurIPS), 2017.
  14. Z. He, J. Lin, R. Ewetz, J.-S. Yuan, and D. Fan, “Noise Injection Adaption: End-to-End ReRAM Crossbar Non-ideal Effect Adaption for Neural Network Mapping,” Design Automation Conf. (DAC), 2019.
  15. S. Jain, A. Sengupta, K. Roy, and A. Raghunathan, “RxNN: A Framework for Evaluating Deep Neural Networks on Resistive Crossbars,” IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems (TCAD), vol. 40, no. 2, pp. 326–338, 2020.
  16. S. Roy, S. Sridharan, S. Jain, and A. Raghunathan, “Txsim: Modeling Training of Deep Neural Networks on Resistive Crossbar Systems,” IEEE Trans. on Very Large-Scale Integration Systems (TVLSI), vol. 29, no. 4, pp. 730–738, 2021.
  17. Z. Zhu, H. Sun, K. Qiu, L. Xia, G. Krishnan, G. Dai, D. Niu, X. Chen, X. S. Hu, Y. Cao et al., “MNSIM 2.0: A Behavior-Level Modeling Tool for Memristor-based Neuromorphic Computing Systems,” Great Lakes Symposium on VLSI, pp. 83–88, 2020.
  18. X. Peng, S. Huang, Y. Luo, X. Sun, and S. Yu, “DNN+ NeuroSim: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators with Versatile Device Technologies,” IEEE Int’l electron devices meeting (IEDM), 2019.
  19. X. Sun, R. Liu, X. Peng, and S. Yu, “Computing-in-Memory with SRAM and RRAM for Binary Neural Networks,” IEEE Int’l Conf. on Solid-State and Integrated Circuit Technology (ICSICT), 2018.
  20. M. Zahedi, T. Shahroodi, S. Wong, and S. Hamdioui, “BCIM: Efficient Implementation of Binary Neural Network Based on Computation in Memory,” arXiv preprint arXiv:2211.06261, 2022.
  21. X. Sun, X. Peng, P.-Y. Chen, R. Liu, J.-s. Seo, and S. Yu, “Fully Parallel RRAM Synaptic Array for Implementing Binary Neural Network with (+ 1,- 1) Weights and (+ 1, 0) Neurons,” Asia and South Pacific Design Automation Conf. (ASP-DAC), 2018.
  22. X. Sun, S. Yin, X. Peng, R. Liu, J.-s. Seo, and S. Yu, “XNOR-RRAM: A Scalable and Parallel Resistive Synaptic Architecture for Binary Neural Networks,” Design, Automation, and Test in Europe (DATE), 2018.
  23. S. Yin, Z. Jiang, J.-S. Seo, and M. Seok, “XNOR-SRAM: In-Memory Computing SRAM Macro for Binary/Ternary Deep Neural Networks,” IEEE Journal of Solid-State Circuits, vol. 55, no. 6, pp. 1733–1743, 2020.
  24. J. Kim, J. Koo, T. Kim, Y. Kim, H. Kim, S. Yoo, and J.-J. Kim, “Area-Efficient and Variation-Tolerant In-Memory BNN Computing using 6T SRAM Array,” Symposium on VLSI Circuits, pp. C118–C119, 2019.
  25. C. D. Schuman, T. E. Potok, R. M. Patton, J. D. Birdwell, M. E. Dean, G. S. Rose, and J. S. Plank, “A survey of Neuromorphic Computing and Neural Networks in Hardware,” arXiv preprint arXiv:1705.06963, 2017.
  26. H. Caminal, K. Yang, S. Srinivasa, A. K. Ramanathan, K. Al-Hawaj, T. Wu, V. Narayanan, C. Batten, and J. F. Martínez, “CAPE: A Content-Addressable Processing Engine,” Int’l Symp. on High-Performance Computer Architecture (HPCA), 2021.
  27. J. Boukhobza, S. Rubini, R. Chen, and Z. Shao, “Emerging NVM: A survey on Architectural Integration and Research Challenges,” ACM Trans. on Design Automation of Electronic Systems (TODAES), vol. 23, no. 2, pp. 1–32, 2017.
  28. D. Ielmini and H.-S. P. Wong, “In-memory computing with resistive switching devices,” Nature Electronics, vol. 1, no. 6, pp. 333–343, 2018.
  29. A. Sebastian, M. Le Gallo, R. Khaddam-Aljameh, and E. Eleftheriou, “Memory devices and applications for in-memory computing,” Nature Nanotechnology, vol. 15, no. 7, pp. 529–544, 2020.
  30. G. W. Burr, R. M. Shelby, A. Sebastian, S. Kim, S. Kim, S. Sidler, K. Virwani, M. Ishii, P. Narayanan, A. Fumarola, L. L. Sanches, I. Boybat, M. Le Gallo, K. Moon, J. Woo, H. Hwang, and Y. Leblebici, “Neuromorphic computing using non-volatile memory,” Advances in Physics: X, vol. 2, no. 1, pp. 89–124, 2017.
  31. K. Roy, A. Jaiswal, and P. Panda, “Towards spike-based machine intelligence with neuromorphic computing,” Nature, vol. 575, no. 7784, pp. 607–617, 2019.
  32. P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, “PRIME: A Novel Processing-in-Memory Architecture for Neural Network Computation in ReRAM-Based Main Memory,” Int’l Symp. on Computer Architecture (ISCA), 2016.
  33. L. Geiger and P. Team, “Larq: An Open-Source Library for Training Binarized Neural Networks,” Journal of Open Source Software, vol. 5, no. 45, p. 1746, 2020.
  34. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.
  35. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016.
  36. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely Connected Convolutional Networks,” IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
  37. N. Guo, J. Bethge, C. Meinel, and H. Yang, “Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket,” arXiv preprint arXiv:2211.12933, 2022.
  38. Y. Zhang, A. Garg, Y. Cao, Ł. Lew, B. Ghorbani, Z. Zhang, and O. Firat, “Binarized Neural Machine Translation,” arXiv preprint arXiv:2302.04907, 2023.
  39. Z. Liu, Z. Shen, M. Savvides, and K.-T. Cheng, “ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions,” European Conf. on Computer Vision (ECCV), 2020.
  40. Y. Zhang, Z. Zhang, and L. Lew, “PokeBNN: A Binary Pursuit of Lightweight Accuracy,” IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now