Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Memory-Efficient Reversible Spiking Neural Networks (2312.07922v1)

Published 13 Dec 2023 in cs.CV and cs.NE

Abstract: Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs) due to their high energy-efficiency on neuromorphic hardware. However, SNNs are unfolded over simulation time steps during the training process. Thus, SNNs require much more memory than ANNs, which impedes the training of deeper SNN models. In this paper, we propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training. Firstly, we extend the reversible architecture along temporal dimension and propose the reversible spiking block, which can reconstruct the computational graph and recompute all intermediate variables in forward pass with a reverse process. On this basis, we adopt the state-of-the-art SNN models to the reversible variants, namely reversible spiking ResNet (RevSResNet) and reversible spiking transformer (RevSFormer). Through experiments on static and neuromorphic datasets, we demonstrate that the memory cost per image of our reversible SNNs does not increase with the network depth. On CIFAR10 and CIFAR100 datasets, our RevSResNet37 and RevSFormer-4-384 achieve comparable accuracies and consume 3.79x and 3.00x lower GPU memory per image than their counterparts with roughly identical model complexity and parameters. We believe that this work can unleash the memory constraints in SNN training and pave the way for training extremely large and deep SNNs. The code is available at https://github.com/mi804/RevSNN.git.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. A low power, fully event-based gesture recognition system. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7243–7252.
  2. A partially reversible U-Net for memory-efficient volumetric image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part III 22, 429–437. Springer.
  3. Optimized Potential Initialization for Low-latency Spiking Neural Networks. arXiv preprint arXiv:2202.01440.
  4. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1): 82–99.
  5. Optimal conversion of conventional artificial neural networks to spiking neural networks. arXiv preprint arXiv:2103.00476.
  6. Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.11946.
  7. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International joint conference on neural networks (IJCNN), 1–8. ieee.
  8. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516.
  9. SpikingJelly. https://github.com/fangwei123456/spikingjelly. Accessed: 2022-08-15.
  10. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34: 21056–21069.
  11. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, 2661–2671.
  12. Multi-level firing with spiking ds-resnet: Enabling better and deeper directly-trained spiking neural networks. arXiv preprint arXiv:2210.06386.
  13. Spiking neuron models: Single neurons, populations, plasticity. Cambridge university press.
  14. The reversible residual network: Backpropagation without storing activations. Advances in neural information processing systems, 30.
  15. Layer-wise invertibility for extreme memory cost reduction of cnn training. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  17. Advancing Spiking Neural Networks towards Deep Residual Learning. arXiv preprint arXiv:2112.08954.
  18. Neural architecture search for spiking neural networks. In European Conference on Computer Vision, 36–56. Springer.
  19. Learning multiple layers of features from tiny images.
  20. Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 14: 439.
  21. Enabling spike-based backpropagation for training deep neural network architectures. Frontiers in neuroscience, 119.
  22. m-revnet: Deep reversible neural networks with momentum. arXiv preprint arXiv:2108.05862.
  23. Training graph neural networks with 1000 layers. In International conference on machine learning, 6437–6449. PMLR.
  24. Cifar10-dvs: an event-stream dataset for object classification. Frontiers in neuroscience, 11: 309.
  25. Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems, 34: 23426–23439.
  26. Maass, W. 1997. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9): 1659–1671.
  27. Reversible recurrent neural networks. Advances in Neural Information Processing Systems, 31.
  28. Reversible vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10830–10840.
  29. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12444–12453.
  30. Autosnn: Towards energy-efficient spiking neural networks. In International Conference on Machine Learning, 16253–16269. PMLR.
  31. Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6): 51–63.
  32. CuPy: A NumPy-Compatible Library for NVIDIA GPU Calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS).
  33. Diet-snn: Direct input encoding with leakage and threshold optimization in deep spiking neural networks. arXiv preprint arXiv:2008.03658.
  34. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01807.
  35. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784): 607–617.
  36. Momentum residual neural networks. In International Conference on Machine Learning, 9276–9287. PMLR.
  37. Slayer: Spike layer error reassignment in time. Advances in neural information processing systems, 31.
  38. Mintnet: Building invertible neural networks with masked convolutions. Advances in Neural Information Processing Systems, 32.
  39. Signed neuron with memory: Towards simple, accurate and high-efficient ann-snn conversion. In International Joint Conference on Artificial Intelligence.
  40. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12: 331.
  41. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 1311–1318.
  42. Rethinking” batch” in batchnorm. arXiv preprint arXiv:2105.07576.
  43. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. IEEE Transactions on Neural Networks and Learning Systems, 33(11): 6249–6262.
  44. Temporal-wise attention spiking neural networks for event streams classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10221–10230.
  45. Glif: A unified gated leaky integrate-and-fire neuron for spiking neural networks. Advances in Neural Information Processing Systems, 35: 32160–32171.
  46. Energy-Efficient Spiking Segmenter for Frame and Event-Based Images. Biomimetics, (4).
  47. Temporal spike sequence learning via backpropagation for deep spiking neural networks. Advances in Neural Information Processing Systems, 33: 12022–12033.
  48. Direct Training High-Performance Spiking Neural Networks for Object Recognition and Detection. Frontiers in Neuroscience, 17: 1229951.
  49. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 11062–11070.
  50. Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network. arXiv preprint arXiv:2304.11954.
  51. Spikformer: When spiking neural network meets transformer. arXiv preprint arXiv:2209.15425.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Hong Zhang (272 papers)
  2. Yu Zhang (1400 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - mi804/RevSNN (17 stars)