Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Accelerating SNN Training with Stochastic Parallelizable Spiking Neurons (2306.12666v1)

Published 22 Jun 2023 in cs.NE

Abstract: Spiking neural networks (SNN) are able to learn spatiotemporal features while using less energy, especially on neuromorphic hardware. The most widely used spiking neuron in deep learning is the Leaky Integrate and Fire (LIF) neuron. LIF neurons operate sequentially, however, since the computation of state at time t relies on the state at time t-1 being computed. This limitation is shared with Recurrent Neural Networks (RNN) and results in slow training on Graphics Processing Units (GPU). In this paper, we propose the Stochastic Parallelizable Spiking Neuron (SPSN) to overcome the sequential training limitation of LIF neurons. By separating the linear integration component from the non-linear spiking function, SPSN can be run in parallel over time. The proposed approach results in performance comparable with the state-of-the-art for feedforward neural networks on the Spiking Heidelberg Digits (SHD) dataset, outperforming LIF networks while training 10 times faster and outperforming non-spiking networks with the same network architecture. For longer input sequences of 10000 time-steps, we show that the proposed approach results in 4000 times faster training, thus demonstrating the potential of the proposed approach to accelerate SNN training for very large datasets.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. J. D. Owens, M. Houston, D. Luebke, S. Green, J. E. Stone, and J. C. Phillips, “GPU computing,” Proceedings of the IEEE, vol. 96, no. 5, pp. 879–899, 2008.
  2. J. Nickolls and W. J. Dally, “The GPU computing era,” IEEE micro, vol. 30, no. 2, pp. 56–69, 2010.
  3. N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers et al., “In-datacenter performance analysis of a tensor processing unit,” in Proceedings of the 44th annual international symposium on computer architecture, 2017, pp. 1–12.
  4. Z. Fan, F. Qiu, A. Kaufman, and S. Yoakum-Stover, “GPU cluster for high performance computing,” in SC’04: Proceedings of the 2004 ACM/IEEE conference on Supercomputing.   IEEE, 2004, pp. 47–47.
  5. Y. E. Wang, G.-Y. Wei, and D. Brooks, “Benchmarking TPU, GPU, and CPU platforms for deep learning,” arXiv preprint arXiv:1907.10701, 2019.
  6. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition.   Ieee, 2009, pp. 248–255.
  7. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 5485–5551, 2020.
  8. S. Ghosh-Dastidar and H. Adeli, “Spiking neural networks,” International journal of neural systems, vol. 19, no. 04, pp. 295–308, 2009.
  9. A. Tavanaei, M. Ghodrati, S. R. Kheradpisheh, T. Masquelier, and A. Maida, “Deep learning in spiking neural networks,” Neural networks, vol. 111, pp. 47–63, 2019.
  10. B. Li, E. Zhou, B. Huang, J. Duan, Y. Wang, N. Xu, J. Zhang, and H. Yang, “Large scale recurrent neural network on GPU,” in 2014 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2014, pp. 4062–4069.
  11. V. Khomenko, O. Shyshkov, O. Radyvonenko, and K. Bokhan, “Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization,” in 2016 IEEE First International Conference on Data Stream Mining & Processing (DSMP).   IEEE, 2016, pp. 100–103.
  12. A. Voelker, I. Kajić, and C. Eliasmith, “Legendre memory units: Continuous-time representation in recurrent neural networks,” Advances in neural information processing systems, vol. 32, 2019.
  13. N. R. Chilkuri and C. Eliasmith, “Parallelizing legendre memory unit training,” in International Conference on Machine Learning.   PMLR, 2021, pp. 1898–1907.
  14. W. Maass, “To spike or not to spike: that is the question,” Proceedings of the IEEE, vol. 103, no. 12, pp. 2219–2224, 2015.
  15. D. Dold, I. Bytschok, A. F. Kungl, A. Baumbach, O. Breitwieser, W. Senn, J. Schemmel, K. Meier, and M. A. Petrovici, “Stochasticity from function—why the bayesian brain may need no noise,” Neural networks, vol. 119, pp. 200–213, 2019.
  16. G. A. Fonseca Guerra and S. B. Furber, “Using stochastic spiking neural networks on spinnaker to solve constraint satisfaction problems,” Frontiers in neuroscience, vol. 11, p. 714, 2017.
  17. S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana, “The spinnaker project,” Proceedings of the IEEE, vol. 102, no. 5, pp. 652–665, 2014.
  18. L. Buesing, J. Bill, B. Nessler, and W. Maass, “Neural dynamics as sampling: a model for stochastic computation in recurrent networks of spiking neurons,” PLoS computational biology, vol. 7, no. 11, p. e1002211, 2011.
  19. M. A. Petrovici, J. Bill, I. Bytschok, J. Schemmel, and K. Meier, “Stochastic inference with deterministic spiking neurons,” arXiv preprint arXiv:1311.3211, 2013.
  20. R. Brette and D. F. Goodman, “Simulating spiking neural networks on gpu,” Network: Computation in Neural Systems, vol. 23, no. 4, pp. 167–182, 2012.
  21. A. K. Fidjeland, E. B. Roesch, M. P. Shanahan, and W. Luk, “Nemo: a platform for neural modelling of spiking neurons using gpus,” in 2009 20th IEEE international conference on application-specific systems, architectures and processors.   IEEE, 2009, pp. 137–144.
  22. E. Yavuz, J. Turner, and T. Nowotny, “Genn: a code generation framework for accelerated brain simulations,” Scientific reports, vol. 6, no. 1, pp. 1–14, 2016.
  23. D. F. Goodman and R. Brette, “The brian simulator,” Frontiers in neuroscience, p. 26, 2009.
  24. B. Kasap and A. J. van Opstal, “Dynamic parallelism for synaptic updating in gpu-accelerated spiking neural network simulations,” Neurocomputing, vol. 302, pp. 55–65, 2018.
  25. L. Ślażyński and S. Bohte, “Streaming parallel gpu acceleration of large-scale filter-based spiking neural networks,” Network: Computation in Neural Systems, vol. 23, no. 4, pp. 183–211, 2012.
  26. D. L. Manna, A. Vicente-Sola, P. Kirkland, T. J. Bihl, and G. Di Caterina, “Frameworks for snns: a review of data science-oriented software and an expansion of spyketorch,” arXiv preprint arXiv:2302.07624, 2023.
  27. E. P. Simoncelli, L. Paninski, J. Pillow, O. Schwartz et al., “Characterization of neural responses with stochastic stimuli,” The cognitive neurosciences, vol. 3, no. 327-338, p. 1, 2004.
  28. E. M. Izhikevich, “Which model to use for cortical spiking neurons?” IEEE transactions on neural networks, vol. 15, no. 5, pp. 1063–1070, 2004.
  29. M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou, P. Joshi, N. Imam, S. Jain et al., “Loihi: A neuromorphic manycore processor with on-chip learning,” Ieee Micro, vol. 38, no. 1, pp. 82–99, 2018.
  30. E. Hunsberger and C. Eliasmith, “Spiking deep networks with LIF neurons,” arXiv preprint arXiv:1510.08829, 2015.
  31. F. Corradi, S. Pande, J. Stuijt, N. Qiao, S. Schaafsma, G. Indiveri, and F. Catthoor, “ECG-based heartbeat classification in neuromorphic hardware,” in 2019 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2019, pp. 1–8.
  32. A. N. Burkitt, “A review of the integrate-and-fire neuron model: I. homogeneous synaptic input,” Biological cybernetics, vol. 95, pp. 1–19, 2006.
  33. E. O. Neftci, H. Mostafa, and F. Zenke, “Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks,” IEEE Signal Processing Magazine, vol. 36, no. 6, pp. 51–63, 2019.
  34. Y. Bengio, N. Léonard, and A. Courville, “Estimating or propagating gradients through stochastic neurons for conditional computation,” arXiv preprint arXiv:1308.3432, 2013.
  35. E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with Gumbel-Softmax,” arXiv preprint arXiv:1611.01144, 2016.
  36. B. Yin, F. Corradi, and S. M. Bohté, “Effective and efficient computation with multiple-timescale spiking recurrent neural networks,” in International Conference on Neuromorphic Systems 2020, 2020, pp. 1–8.
  37. B. Cramer, Y. Stradmann, J. Schemmel, and F. Zenke, “The Heidelberg spiking data sets for the systematic evaluation of spiking neural networks,” IEEE Transactions on Neural Networks and Learning Systems, 2020.
  38. T. Nowotny, J. P. Turner, and J. C. Knight, “Loss shaping enhances exact gradient learning with eventprop in spiking neural networks,” arXiv preprint arXiv:2212.01232, 2022.
  39. C. Yu, Z. Gu, D. Li, G. Wang, A. Wang, and E. Li, “Stsc-snn: Spatio-temporal synaptic connection with temporal convolution and attention for spiking neural networks,” arXiv preprint arXiv:2210.05241, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Sidi Yaya Arnaud Yarga (3 papers)
  2. Sean U. N. Wood (7 papers)
Citations (7)
View on Semantic Scholar