Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hardware-aware training of models with synaptic delays for digital event-driven neuromorphic processors (2404.10597v1)

Published 16 Apr 2024 in cs.NE, cs.AI, and cs.ET

Abstract: Configurable synaptic delays are a basic feature in many neuromorphic neural network hardware accelerators. However, they have been rarely used in model implementations, despite their promising impact on performance and efficiency in tasks that exhibit complex (temporal) dynamics, as it has been unclear how to optimize them. In this work, we propose a framework to train and deploy, in digital neuromorphic hardware, highly performing spiking neural network models (SNNs) where apart from the synaptic weights, the per-synapse delays are also co-optimized. Leveraging spike-based back-propagation-through-time, the training accounts for both platform constraints, such as synaptic weight precision and the total number of parameters per core, as a function of the network size. In addition, a delay pruning technique is used to reduce memory footprint with a low cost in performance. We evaluate trained models in two neuromorphic digital hardware platforms: Intel Loihi and Imec Seneca. Loihi offers synaptic delay support using the so-called Ring-Buffer hardware structure. Seneca does not provide native hardware support for synaptic delays. A second contribution of this paper is therefore a novel area- and memory-efficient hardware structure for acceleration of synaptic delays, which we have integrated in Seneca. The evaluated benchmark involves several models for solving the SHD (Spiking Heidelberg Digits) classification task, where minimal accuracy degradation during the transition from software to hardware is demonstrated. To our knowledge, this is the first work showcasing how to train and deploy hardware-aware models parameterized with synaptic delays, on multicore neuromorphic hardware accelerators.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. C. Stoelzel, Y. Bereshpolova, J.-M. Alonso, and H. Swadlow, “Axonal conduction delays, brain state, and corticogeniculate communication,” The Journal of Neuroscience, vol. 37, pp. 0444–17, 05 2017.
  2. F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla, N. Imam, Y. Nakamura, P. Datta, G. Nam, B. Taba, M. Beakes, B. Brezzo, J. B. Kuang, R. Manohar, W. P. Risk, B. Jackson, and D. S. Modha, “Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 34, no. 10, pp. 1537–1557, 2015.
  3. G. Tang, K. Vadivel, Y. Xu, R. Bilgic, K. Shidqi, P. Detterer, S. Traferro, M. Konijnenburg, M. Sifalakis, G.-J. van Schaik, and A. Yousefzadeh, “Seneca: building a fully digital neuromorphic processor, design trade-offs and challenges,” Frontiers in Neuroscience, vol. 17, 2023. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fnins.2023.1187252
  4. S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, “The spinnaker project,” Proceedings of the IEEE, vol. 102, no. 5, pp. 652–665, 2014.
  5. M. Davies, N. Srinivasa, T. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain, Y. Liao, C. Lin, A. Lines, R. Liu, D. Mathaikutty, S. McCoy, A. Paul, J. Tse, G. Venkataramanan, Y. Weng, A. Wild, Y. Yang, and H. Wang, “Loihi: A neuromorphic manycore processor with on-chip learning,” IEEE Micro, vol. 38, no. 1, 2018.
  6. J. A. Starzyk, Ł. Maciura, and A. Horzyk, “Associative memories with synaptic delays,” IEEE Transactions on Neural Networks and Learning Systems, vol. 31, no. 1, pp. 331–344, 2019.
  7. B. Cohen, D. Saad, and E. Marom, “Efficient training of recurrent neural network with time delays,” Neural Networks, vol. 10, no. 1, 1997.
  8. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, “Phoneme recognition using time-delay neural networks,” IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, 1989.
  9. J. Santos and P. Fernandez, “Evolved synaptic delay based neural controllers for walking patterns in hexapod robotic structures,” Natural Computing, vol. 16, 06 2017.
  10. S. Day and M. Davenport, “Continuous-time temporal back-propagation with adaptable time delays,” IEEE Transactions on Neural Networks, vol. 4, no. 2, pp. 348–354, 1993.
  11. R. Duro and J. Reyes, “Discrete-time backpropagation for training synaptic delay-based artificial neural networks,” IEEE Transactions on Neural Networks, vol. 10, no. 4, pp. 779–789, 1999.
  12. R. Boné and H. Cardot, “Time delay learning by gradient descent in recurrent neural networks,” in Proceedings of the 15th International Conference on Artificial Neural Networks: Formal Models and Their Applications - Volume Part II, ser. ICANN’05.   Berlin, Heidelberg: Springer-Verlag, 2005, p. 175–180.
  13. S. B. Shrestha and Q. Song, “Adaptive delay learning in spikeprop based on delay convergence analysis,” in 2016 International Joint Conference on Neural Networks (IJCNN), 2016, pp. 277–284.
  14. C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).   Los Alamitos, CA, USA: IEEE Computer Society, July 2017.
  15. A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A Generative Model for Raw Audio,” in Proc. 9th ISCA Workshop on Speech Synthesis Workshop (SSW 9), 2016, p. 125.
  16. A. Patiño-Saucedo, A. Yousefzadeh, G. Tang, F. Corradi, B. Linares-Barranco, and M. Sifalakis, “Empirical study on the efficiency of spiking neural networks with axonal delays, and algorithm-hardware benchmarking,” in 2023 IEEE International Symposium on Circuits and Systems (ISCAS).   IEEE, 2023, pp. 1–5.
  17. M. Zhang, J. Wu, A. Belatreche, Z. Pan, X. Xie, Y. Chua, G. Li, H. Qu, and H. Li, “Supervised learning in spiking neural networks with synaptic delay-weight plasticity,” Neurocomputing, vol. 409, pp. 103–118, 2020.
  18. I. Hammouamri, I. Khalfaoui-Hassani, and T. Masquelier, “Learning delays in spiking neural networks using dilated convolutions with learnable spacings,” arXiv preprint arXiv:2306.17670, 2023.
  19. X. Wang, X. Lin, and X. Dang, “A delay learning algorithm based on spike train kernels for spiking neurons,” Frontiers in Neuroscience, vol. 13, 2019.
  20. P. Sun, E. Eqlimi, Y. Chua, P. Devos, and D. Botteldooren, “Adaptive axonal delays in feedforward spiking neural networks for accurate spoken word recognition,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2023, pp. 1–5.
  21. G. Tang, A. Safa, K. Shidqi, P. Detterer, S. Traferro, M. Konijnenburg, M. Sifalakis, G.-J. van Schaik, and A. Yousefzadeh, “Open the box of digital neuromorphic processor: Towards effective algorithm-hardware co-design,” in 2023 IEEE International Symposium on Circuits and Systems (ISCAS).   IEEE, 2023, pp. 1–5.
  22. S. B. Shrestha and G. Orchard, “Slayer: Spike layer error reassignment in time,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, ser. NIPS’18.   Red Hook, NY, USA: Curran Associates Inc., 2018, p. 1419–1428.
  23. K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra, “Deep autoregressive networks,” in Proceedings of the 31st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, E. P. Xing and T. Jebara, Eds., vol. 32, no. 2.   Bejing, China: PMLR, 22–24 Jun 2014, pp. 1242–1250.
  24. V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in Proc. Interspeech 2015, 2015, pp. 3214–3218.
  25. A. Morrison, C. Mehring, T. Geisel, A. Aertsen, and M. Diesmann, “Advancing the boundaries of high-connectivity network simulation with distributed computing,” Neural computation, vol. 17, no. 8, 2005.
  26. P. Sun, L. Zhu, and D. Botteldooren, “Axonal delay as a short-term memory for feed forward deep spiking neural networks,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2022, pp. 8932–8936.
  27. B. Linares-Barranco, T. Serrano-Gotarredona, L. Camuñas-Mesa, J. Perez-Carrasco, C. Zamarreño-Ramos, and T. Masquelier, “On spike-timing-dependent-plasticity, memristive devices, and building a self-learning visual cortex,” Frontiers in Neuroscience, vol. 5, 2011.
  28. P. Diehl and M. Cook, “Unsupervised learning of digit recognition using spike-timing-dependent plasticity,” Frontiers in Computational Neuroscience, vol. 9, 2015.
  29. Y. Wu, L. Deng, G. Li, J. Zhu, and L. Shi, “Spatio-temporal backpropagation for training high-performance spiking neural networks,” Frontiers in Neuroscience, vol. 12, 2018.
  30. E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51–63, 2019.
  31. F. Zenke and T. P. Vogels, “The remarkable robustness of surrogate gradient learning for instilling complex function in spiking neural networks,” Neural computation, vol. 33, no. 4, pp. 899–925, 2021.
  32. B. Cramer, Y. Stradmann, J. Schemmel, and F. Zenke, “The heidelberg spiking data sets for the systematic evaluation of spiking neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 7, pp. 2744–2757, 2022.
  33. M. M. Khan, D. R. Lester, L. A. Plana, A. Rast, X. Jin, E. Painkras, and S. B. Furber, “Spinnaker: mapping neural networks onto a massively-parallel chip multiprocessor,” in 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).   Ieee, 2008, pp. 2849–2856.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now