Spatial-SpinDrop: Spatial Dropout-based Binary Bayesian Neural Network with Spintronics Implementation (2306.10185v1)
Abstract: Recently, machine learning systems have gained prominence in real-time, critical decision-making domains, such as autonomous driving and industrial automation. Their implementations should avoid overconfident predictions through uncertainty estimation. Bayesian Neural Networks (BayNNs) are principled methods for estimating predictive uncertainty. However, their computational costs and power consumption hinder their widespread deployment in edge AI. Utilizing Dropout as an approximation of the posterior distribution, binarizing the parameters of BayNNs, and further to that implementing them in spintronics-based computation-in-memory (CiM) hardware arrays provide can be a viable solution. However, designing hardware Dropout modules for convolutional neural network (CNN) topologies is challenging and expensive, as they may require numerous Dropout modules and need to use spatial information to drop certain elements. In this paper, we introduce MC-SpatialDropout, a spatial dropout-based approximate BayNNs with spintronics emerging devices. Our method utilizes the inherent stochasticity of spintronic devices for efficient implementation of the spatial dropout module compared to existing implementations. Furthermore, the number of dropout modules per network layer is reduced by a factor of $9\times$ and energy consumption by a factor of $94.11\times$, while still achieving comparable predictive performance and uncertainty estimates compared to related works.
- D. Kiela et al., “Dynabench: Rethinking benchmarking in nlp,” arXiv preprint arXiv:2104.14337, 2021.
- Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning. PMLR, 2016, pp. 1050–1059.
- Z. Wang et al., “Resistive switching materials for information processing,” Nature Reviews Materials, vol. 5, no. 3, pp. 173–195, 2020.
- S. T. Ahmed et al., “Binary bayesian neural networks for efficient uncertainty estimation leveraging inherent stochasticity of spintronic devices,” in IEEE/ACM NANOARCH, 2022.
- ——, “Spindrop: Dropout-based bayesian binary neural networks with spintronic implementation,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 13, no. 1, pp. 150–164, 2023.
- T. Y. Lee et al., “World-most energy-efficient MRAM technology for non-volatile RAM applications,” in 2022 International Electron Devices Meeting (IEDM), Dec. 2022, pp. 10.7.1–10.7.4, iSSN: 2156-017X.
- N. Srivastava et al., “Dropout: a simple way to prevent neural networks from overfitting,” The journal of machine learning research, vol. 15, no. 1, pp. 1929–1958, 2014.
- C. Blundell et al., “Weight uncertainty in neural network,” in International conference on machine learning. PMLR, 2015, pp. 1613–1622.
- B. Lakshminarayanan et al., “Simple and scalable predictive uncertainty estimation using deep ensembles,” Advances in neural information processing systems, vol. 30, 2017.
- T. Gokmen et al., “Training deep convolutional neural networks with resistive cross-point devices,” Frontiers in neuroscience, vol. 11, p. 538, 2017.
- X. Peng et al., “Optimizing weight mapping and data flow for convolutional neural networks on rram based processing-in-memory architecture,” in IEEE ISCAS. IEEE, 2019, pp. 1–5.
- H. Qin et al., “Forward and backward information retention for accurate binary neural networks,” in IEEE CVPR, 2020.
- B. Razavi, “The strongarm latch [a circuit for all seasons],” IEEE Solid-State Circuits Magazine, vol. 7, no. 2, pp. 12–17, 2015.
- R. Ding et al., “Regularizing activation distribution for training binarized deep networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 11 408–11 417.
- S. Zhou et al., “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,” arXiv preprint arXiv:1606.06160, 2016.
- R. Gong et al., “Differentiable soft quantization: Bridging full-precision and low-bit neural networks,” in Proceedings of the ICCV, 2019, pp. 4852–4861.
- L. Hou et al., “Loss-aware binarization of deep networks,” in International Conference on Learning Representations (ICLR), 2017.
- M. Rastegari et al., “Xnor-net: Imagenet classification using binary convolutional neural networks,” in ECCV 2016. Springer, 2016, pp. 525–542.
- I. Hubara et al., “Binarized neural networks,” Advances in neural information processing systems, vol. 29, 2016.
- R. Cai et al., “Vibnn: Hardware acceleration of bayesian neural networks,” ACM SIGPLAN Notices, vol. 53, no. 2, pp. 476–488, 2018.
- X. Jia et al., “Efficient Computation Reduction in Bayesian Neural Networks Through Feature Decomposition and Memorization,” IEEE Trans. on Neural Networks and Learning Systems, vol. 32, Apr. 2021.
- H. Awano et al., “Bynqnet: Bayesian neural network with quadratic activations for sampling-free uncertainty estimation on fpga,” in Proceeding of DATE. IEEE, 2020, pp. 1402–1407.
- A. Malhotra et al., “Exploiting oxide based resistive ram variability for bayesian neural network hardware design,” IEEE Trans. on Nano., vol. 19, pp. 328–331, 2020.
- Soyed Tuhin Ahmed (10 papers)
- Kamal Danouchi (4 papers)
- Michael Hefenbrock (10 papers)
- Guillaume Prenat (8 papers)
- Lorena Anghel (9 papers)
- Mehdi B. Tahoori (25 papers)