Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency (2401.10843v1)

Published 5 Jan 2024 in cs.NE and cs.LG

Abstract: Spiking Neural Networks (SNNs) that operate in an event-driven manner and employ binary spike representation have recently emerged as promising candidates for energy-efficient computing. However, a cost bottleneck arises in obtaining high-performance SNNs: training a SNN model requires a large number of time steps in addition to the usual learning iterations, hence this limits their energy efficiency. This paper proposes a general training framework that enhances feature learning and activation efficiency within a limited time step, providing a new solution for more energy-efficient SNNs. Our framework allows SNN neurons to learn robust spike feature from different receptive fields and update neuron states by utilizing both current stimuli and recurrence information transmitted from other neurons. This setting continuously complements information within a single time step. Additionally, we propose a projection function to merge these two stimuli to smoothly optimize neuron weights (spike firing threshold and activation). We evaluate the proposal for both convolution and recurrent models. Our experimental results indicate state-of-the-art visual classification tasks, including CIFAR10, CIFAR100, and TinyImageNet.Our framework achieves 72.41% and 72.31% top-1 accuracy with only 1 time step on CIFAR100 for CNNs and RNNs, respectively. Our method reduces 10x and 3x joule energy than a standard ANN and SNN, respectively, on CIFAR10, without additional time steps.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural information processing systems, 31, 2018.
  2. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
  3. Optimized potential initialization for low-latency spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11–20, 2022.
  4. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1):54–66, 2015.
  5. Multi-tier platform for cognizing massive electroencephalogram. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, pages 2464–2470, 2022.
  6. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning. In ECCV, pages 709–726, 2022a.
  7. Towards ultra low latency spiking neural networks for vision and sequential tasks using temporal pruning. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XI, pages 709–726. Springer, 2022b.
  8. Towards energy-efficient, low-latency and accurate spiking lstms, 2022a.
  9. Towards energy-efficient, low-latency and accurate spiking lstms. arXiv preprint arXiv:2210.12613, 2022b.
  10. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476, 2021.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  12. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056–21069, 2021a.
  13. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2661–2671, 2021b.
  14. Multi-level firing with spiking ds-resnet: Enabling better and deeper directly-trained spiking neural networks. arXiv preprint arXiv:2210.06386, 2022.
  15. Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13558–13567, 2020.
  16. Mark Horowitz. 1.1 computing’s energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pages 10–14. IEEE, 2014.
  17. Spiking deep networks with lif neurons. arXiv preprint arXiv:1510.08829, 2015.
  18. Deep neural networks with weighted spikes. Neurocomputing, 311:373–386, 2018.
  19. Mikhail Kiselev. Rate coding vs. temporal coding-is optimum between? In 2016 international joint conference on neural networks (IJCNN), pages 1355–1359. IEEE, 2016.
  20. Learning multiple layers of features from tiny images. 2009.
  21. Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3953–3962, 2021.
  22. Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  23. A time-to-first-spike coding and conversion aware training for energy-efficient deep spiking neural network processor design. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 265–270, 2022.
  24. Brain-inspired multilayer perceptron with spiking neurons. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 783–793, 2022.
  25. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning, pages 6316–6325. PMLR, 2021a.
  26. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34:23426–23439, 2021b.
  27. Spikeconverter: An efficient conversion framework zipping the gap between artificial neural networks and spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1692–1701, 2022.
  28. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  29. Long short-term memory spiking networks and their applications. In International Conference on Neuromorphic Systems 2020, pages 1–9, 2020.
  30. Unsupervised learning of visual features through spike timing dependent plasticity. PLoS computational biology, 3(2):e31, 2007.
  31. Autosnn: towards energy-efficient spiking neural networks. In International Conference on Machine Learning, pages 16253–16269. PMLR, 2022.
  32. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51–63, 2019.
  33. Spiking neural networks with improved inherent recurrence dynamics for sequential learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8001–8008, 2022.
  34. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658, 2020.
  35. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, pages 1–9, 2021a.
  36. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021b.
  37. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations, 2020.
  38. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in neuroscience, 11:682, 2017.
  39. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  40. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
  41. Snn2ann: A fast and memory-efficient training framework for spiking neural networks. arXiv preprint arXiv:2206.09449, 2022.
  42. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
  43. The art of data augmentation. Journal of Computational and Graphical Statistics, 10(1):1–50, 2001.
  44. Learning local representation by gradient-isolated memorizing of spiking neural network. In 2022 IEEE 24th Int Conf on High Performance Computing & Communications; (HPCC), pages 733–740, 2022.
  45. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12:331, 2018.
  46. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1311–1318, 2019.
  47. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  48. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032, 2019.
  49. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11062–11070, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Yunpeng Yao (2 papers)
  2. Man Wu (7 papers)
  3. Zheng Chen (221 papers)
  4. Renyuan Zhang (19 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now