Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations (2401.12416v1)
Abstract: Bayesian Neural Networks (BayNNs) naturally provide uncertainty in their predictions, making them a suitable choice in safety-critical applications. Additionally, their realization using memristor-based in-memory computing (IMC) architectures enables them for resource-constrained edge applications. In addition to predictive uncertainty, however, the ability to be inherently robust to noise in computation is also essential to ensure functional safety. In particular, memristor-based IMCs are susceptible to various sources of non-idealities such as manufacturing and runtime variations, drift, and failure, which can significantly reduce inference accuracy. In this paper, we propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in IMC architectures. To achieve this, we introduce a novel normalization layer combined with stochastic affine transformations. Empirical results in various benchmark datasets show a graceful degradation in inference accuracy, with an improvement of up to $58.11\%$.
- S. T. Ahmed et al., “Binary bayesian neural networks for efficient uncertainty estimation leveraging inherent stochasticity of spintronic devices,” in IEEE/ACM NANOARCH. ACM, 2022.
- T. Y. Lee et al., “World-most energy-efficient MRAM technology for non-volatile RAM applications,” in IEEE IEDM, Dec. 2022.
- L.-H. Tsai et al., “Robust processing-in-memory neural networks via noise-aware normalization,” arXiv preprint arXiv:2007.03230, 2020.
- N. Ye et al., “Improving the robustness of analog deep neural networks through a bayes-optimized noise injection approach,” Communications Engineering, vol. 2, p. 25, 2023.
- R. Faria et al., “Implementing Bayesian networks with embedded stochastic MRAM,” AIP Advances, vol. 8, p. 045101, Apr. 2018.
- S. T. Ahmed et al., “Spinbayes: Algorithm-hardware co-design for uncertainty estimation using bayesian in-memory approximation on spintronic-based architectures,” ACM Trans. on Embedded Comp. Systems, vol. 22, pp. 1–25, 2023.
- ——, “Spatial-spindrop: Spatial dropout-based binary bayesian neural network with spintronics implementation,” arXiv preprint arXiv:2306.10185, 2023.
- ——, “Spindrop: Dropout-based bayesian binary neural networks with spintronic implementation,” IEEE JETCAS, vol. 13, pp. 150–164, 2023.
- ——, “Scalable spintronics-based bayesian neural network for uncertainty estimation,” in DATE, 2023, pp. 1–6.
- D. Gao et al., “Bayesian inference based robust computing on memristor crossbar,” in 2021 58th ACM/IEEE DAC, 2021, pp. 121–126.
- N. Ye et al., “Bayesft: Bayesian optimization for fault tolerant neural network architecture,” in 58th ACM/IEEE DAC, 2021, pp. 487–492.
- Z. Wang et al., “Resistive switching materials for information processing,” Nature Reviews Materials, vol. 5, pp. 173–195, 2020.
- W. Li et al., “Rramedy: Protecting reram-based neural network from permanent and soft faults during its lifetime,” in IEEE ICCD, 2019, pp. 91–99.
- S. Yoshikiyo et al., “Nn algorithm aware alternate layer retraining on computation-in-memory for write variation compensation of non-volatile memories at edge ai,” in 2023 7th IEEE EDTM, 2023, pp. 1–3.
- V. Joshi et al., “Accurate deep neural network inference using computational phase-change memory,” Nature communications, vol. 11, p. 2473, 2020.
- H. Kim et al., “Efficient precise weight tuning protocol considering variation of the synaptic devices and target accuracy,” Neurocomputing, vol. 378, pp. 189–196, 2020.
- Y. Gal et al., “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in international conference on machine learning. PMLR, 2016, pp. 1050–1059.
- H. Qin et al., “Forward and backward information retention for accurate binary neural networks,” in IEEE/CVF CVPR, 2020, pp. 2250–2259.
- J. Choi et al., “Pact: Parameterized clipping activation for quantized neural networks,” arXiv preprint arXiv:1805.06085, 2018.