Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Balanced Resonate-and-Fire Neurons (2402.14603v2)

Published 2 Feb 2024 in cs.NE and cs.LG

Abstract: The resonate-and-fire (RF) neuron, introduced over two decades ago, is a simple, efficient, yet biologically plausible spiking neuron model, which can extract frequency patterns within the time domain due to its resonating membrane dynamics. However, previous RF formulations suffer from intrinsic shortcomings that limit effective learning and prevent exploiting the principled advantage of RF neurons. Here, we introduce the balanced RF (BRF) neuron, which alleviates some of the intrinsic limitations of vanilla RF neurons and demonstrates its effectiveness within recurrent spiking neural networks (RSNNs) on various sequence learning tasks. We show that networks of BRF neurons achieve overall higher task performance, produce only a fraction of the spikes, and require significantly fewer parameters as compared to modern RSNNs. Moreover, BRF-RSNN consistently provide much faster and more stable training convergence, even when bridging many hundreds of time steps during backpropagation through time (BPTT). These results underscore that our BRF-RSNN is a strong candidate for future large-scale RSNN architectures, further lines of research in SNN methodology, and more efficient hardware implementations.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
  2. Comprehensive review of artificial neural network applications to pattern recognition. IEEE access, 7:158820–158846, 2019.
  3. Deep spiking neural networks with resonate-and-fire neurons. arXiv preprint arXiv:2109.08234, 2021.
  4. Subthreshold na+-dependent theta-like rhythmicity in stellate cells of entorhinal cortex layer ii. Nature, 342(6246):175–177, 1989.
  5. Atkinson, K. E. An introduction to numerical analysis, john wiley and sons. New York, 19781, 1989.
  6. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
  7. A surrogate gradient spiking baseline for speech command recognition. Frontiers in Neuroscience, 16:865897, 2022.
  8. The heidelberg spiking data sets for the systematic evaluation of spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(7):2744–2757, 2020.
  9. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1):82–99, 2018.
  10. Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
  11. Evidence for grid cells in a human memory network. Nature, 463(7281):657–661, 2010.
  12. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  2641–2651, 2021. doi: 10.1109/ICCV48922.2021.00266.
  13. Efficient neuromorphic signal processing with resonator neurons. Journal of Signal Processing Systems, 94(10):917–927, 2022.
  14. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  15. Learning delays in spiking neural networks using dilated convolutions with learnable spacings. arXiv preprint arXiv:2306.17670, 2023.
  16. Resonate-and-fire neurons for radar interference detection. In Proceedings of the International Conference on Neuromorphic Systems 2022, pp.  1–4, 2022.
  17. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
  18. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of physiology, 117(4):500, 1952.
  19. Izhikevich, E. M. Resonate-and-fire neurons. Neural networks, 14(6-7):883–894, 2001.
  20. Izhikevich, E. M. Simple model of spiking neurons. IEEE Transactions on neural networks, 14(6):1569–1572, 2003.
  21. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  22. Learning multiple layers of features from tiny images, 2009.
  23. A database for evaluation of algorithms for measurement of qt and other waveform intervals in the ecg. Computers in Cardiology, 24(1):673–676, 1997.
  24. Direct signal encoding with analog resonate-and-fire neurons. IEEE Access, 2023.
  25. On the variance of the adaptive learning rate and beyond. In Proceedings of the Eighth International Conference on Learning Representations (ICLR 2020), April 2020.
  26. In vitro neurons in mammalian cortical layer 4 exhibit intrinsic oscillatory activity in the 10-to 50-hz frequency range. Proceedings of the National Academy of Sciences, 88(3):897–901, 1991.
  27. Dying relu and initialization: Theory and numerical examples. arXiv preprint arXiv:1903.06733, 2019.
  28. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51–63, 2019.
  29. Automatic differentiation in pytorch. In NIPS-W, 2017.
  30. Computing with spiking neuron networks. Handbook of natural computing, 1:1–47, 2012.
  31. Dendritic calcium conductances generate high-frequency oscillation in thalamocortical neurons. Proceedings of the National Academy of Sciences, 94(2):724–728, 1997.
  32. Deep learning with spiking neurons: Opportunities and challenges. Frontiers in neuroscience, 12:774, 2018.
  33. Efficient video and audio processing with loihi 2. arXiv preprint arXiv:2310.03251, 2023.
  34. Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
  35. Development and application of artificial neural network. Wireless Personal Communications, 102:1645–1656, 2018.
  36. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905–913, 2021.
  37. Long short-term memory with two-compartment spiking neuron. arXiv preprint arXiv:2307.07231, 2023.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now