Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
121 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Brain-Inspired Efficient Pruning: Exploiting Criticality in Spiking Neural Networks (2311.16141v3)

Published 5 Nov 2023 in cs.NE, cs.AI, cs.CV, and cs.LG

Abstract: Spiking Neural Networks (SNNs) have gained significant attention due to the energy-efficient and multiplication-free characteristics. Despite these advantages, deploying large-scale SNNs on edge hardware is challenging due to limited resource availability. Network pruning offers a viable approach to compress the network scale and reduce hardware resource requirements for model deployment. However, existing SNN pruning methods cause high pruning costs and performance loss because they lack efficiency in processing the sparse spike representation of SNNs. In this paper, inspired by the critical brain hypothesis in neuroscience and the high biological plausibility of SNNs, we explore and leverage criticality to facilitate efficient pruning in deep SNNs. We firstly explain criticality in SNNs from the perspective of maximizing feature information entropy. Second, We propose a low-cost metric for assess neuron criticality in feature transmission and design a pruning-regeneration method that incorporates this criticality into the pruning process. Experimental results demonstrate that our method achieves higher performance than the current state-of-the-art (SOTA) method with up to 95.26\% reduction of pruning cost. The criticality-based regeneration process efficiently selects potential structures and facilitates consistent feature representation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE transactions on computer-aided design of integrated circuits and systems, 34(10): 1537–1557.
  2. Beggs, J. M. 2008. The criticality hypothesis: how local cortical networks might optimize information processing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366(1864): 329–343.
  3. Being critical of criticality in the brain. Frontiers in physiology, 3: 163.
  4. Long short-term memory and learning-to-learn in networks of spiking neurons. Advances in neural information processing systems, 31.
  5. Pruning of Deep Spiking Neural Networks through Gradient Rewiring. In Zhou, Z.-H., ed., Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, 1713–1721. International Joint Conferences on Artificial Intelligence Organization. Main Track.
  6. Loihi: A neuromorphic manycore processor with on-chip learning. Ieee Micro, 38(1): 82–99.
  7. Comprehensive snn compression using admm optimization and activity regularization. IEEE transactions on neural networks and learning systems.
  8. Landau–Ginzburg theory of cortex dynamics: Scale-free avalanches emerge at the edge of synchronization. Proceedings of the National Academy of Sciences, 115(7): E1356–E1365.
  9. Resrep: Lossless cnn pruning via decoupling remembering and forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4510–4520.
  10. Global sparse momentum sgd for pruning very deep neural networks. Advances in Neural Information Processing Systems, 32.
  11. SpikingJelly. https://github.com/fangwei123456/spikingjelly. Accessed: 2022-5-1.
  12. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635.
  13. Self-organized criticality in single-neuron excitability. Physical Review E, 88(6): 062717.
  14. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574.
  15. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28.
  16. Hansen, L. 2015. Tiny ImageNet challenge submission. CS 231N, 5.
  17. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  18. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 4340–4349.
  19. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, 1389–1397.
  20. Criticality, connectivity, and neural disorder: a multifaceted approach to neural computation. Frontiers in computational neuroscience, 15: 611183.
  21. Earthquake cycles and neural reverberations: collective oscillations in systems with pulse-coupled threshold elements. Physical review letters, 75(6): 1222.
  22. Self-organized criticality as a fundamental property of neural systems. Frontiers in systems neuroscience, 8: 166.
  23. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250.
  24. Izhikevich, E. M. 2003. Simple model of spiking neurons. IEEE Transactions on neural networks, 14(6): 1569–1572.
  25. Network plasticity as Bayesian inference. PLoS computational biology, 11(11): e1004485.
  26. Kiang, M. Y. 2001. Extending the Kohonen self-organizing map networks for clustering analysis. Computational Statistics & Data Analysis, 38(2): 161–180.
  27. Exploring temporal information dynamics in spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 8308–8316.
  28. Exploring lottery ticket hypothesis in spiking neural networks. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XII, 102–120. Springer.
  29. Optimal dynamical range of excitable networks at criticality. Nature physics, 2(5): 348–351.
  30. Learning multiple layers of features from tiny images.
  31. Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3953–3962.
  32. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1529–1538.
  33. Dynsnn: A dynamic approach to reduce redundancy in spiking neural networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2130–2134. IEEE.
  34. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34: 9908–9922.
  35. Application of deep compression technique in spiking neural network chip. IEEE transactions on biomedical circuits and systems, 14(2): 274–282.
  36. Learning Efficient Convolutional Networks Through Network Slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
  37. Maass, W. 1997. Networks of spiking neurons: the third generation of neural network models. Neural networks, 10(9): 1659–1671.
  38. Maass, W. 2014. Noise as a resource for computation and learning in networks of spiking neurons. Proceedings of the IEEE, 102(5): 860–880.
  39. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11264–11272.
  40. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
  41. Stochastic synapses enable efficient brain-inspired learning machines. Frontiers in neuroscience, 10: 241.
  42. New K-means clustering methods that minimize the total intra-cluster variance. Afr. J. Math. Stat. Stud, 3: 42–54.
  43. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 572(7767): 106–111.
  44. Noise in integrate-and-fire neurons: from stochastic input to escape rates. Neural computation, 12(2): 367–384.
  45. Jointly learning network connections and link weights in spiking neural networks. In IJCAI, 1597–1603.
  46. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784): 607–617.
  47. Neuronal avalanches imply maximum dynamic range in cortical networks at criticality. Journal of neuroscience, 29(49): 15595–15600.
  48. In-hardware learning of multilayer spiking neural networks on a neuromorphic processor. In 2021 58th ACM/IEEE Design Automation Conference (DAC), 367–372. IEEE.
  49. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  50. Turing, A. M. 2009. Computing machinery and intelligence. Springer.
  51. Neural Pruning via Growing Regularization. In International Conference on Learning Representations (ICLR).
  52. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in neuroscience, 12: 331.
  53. Direct training for spiking neural networks: Faster, larger, better. In Proceedings of the AAAI conference on artificial intelligence, volume 33, 1311–1318.
  54. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957.
  55. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 11062–11070.
  56. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878.

Summary

  • The paper introduces a criticality-guided pruning method that leverages a novel surrogate derivative metric to identify and preserve high-excitation neurons in SNNs.
  • It demonstrates significant improvements, achieving up to 95.26% reduction in pruning costs while improving test accuracy on benchmarks like CIFAR100.
  • The approach employs a regeneration mechanism to reactivate critical neurons, aligning artificial pruning processes with self-organized criticality observed in the brain.

Criticality-Guided Efficient Pruning in Spiking Neural Networks

Overview

This paper introduces an innovative approach to pruning Spiking Neural Networks (SNNs), leveraging concepts inspired by the critical brain hypothesis. It presents a criticality-guided pruning methodology aimed at enhancing feature extraction while significantly reducing the required computational resources. By prioritizing neurons with high criticality for preservation and reactivation during pruning iterations, the proposed method addresses key issues associated with existing SNN pruning techniques.

Methodology

The core component of this research involves the development of a metric for neuron criticality. Drawing inspiration from the critical brain hypothesis, the authors propose a metric based on the derivative of a surrogate function, which reflects a neuron's proximity to excitation threshold conditions. This metric aims to identify neurons exhibiting high excitation potential, aligning with the self-organized criticality observed in biological neural systems.

Complementing this criticality assessment, the authors implement a regeneration mechanism which acts post-pruning to reinstate neurons that demonstrate higher criticality scores. This regeneration supports the aim of maintaining a model's critical state, thereby enhancing its capacity for feature extraction and efficient information processing, alongside faster convergence.

Experimental Evaluation

The methodology was evaluated through comprehensive experiments on various SNN architectures, such as VGG16 and ResNet19, using benchmark datasets like CIFAR10, CIFAR100, and Tiny-ImageNet. The proposed method was compared against multiple state-of-the-art SNN pruning techniques.

  • On CIFAR100, the criticality-guided method achieved superior test accuracy across varying levels of network sparsity, with pruning cost reductions of up to 95.26% compared to benchmarks.
  • Structured pruning performance was enhanced as well, with significant improvements in model accuracy without compromising on computational efficiency.

Results and Implications

The paper demonstrates that criticality-based pruning achieves competitive performance while drastically reducing the computational burden traditionally associated with SNN training. By drawing parallels to biological phenomena, it offers an avenue for aligning artificial network pruning methods more closely with naturally efficient neural optimization processes.

The implications of this work are significant for the development of energy-efficient, high-performance SNNs, particularly in resource-constrained environments. It introduces a biologically inspired framework that may foster further exploration into criticality-based learning and adaptation mechanisms in artificial neural networks.

Future Directions

This research opens pathways for further exploration into the applicability of critical brain-inspired optimization across various neural network modalities. Future work may address scaling the criticality-guided methods to larger and more complex networks, as well as exploring its potential synergies with other biologically inspired neural network methodologies. Furthermore, its extension to neuromorphic computing architectures presents an exciting avenue for exploration, potentially leading to advancements in both theoretical neuroscience and applied machine learning.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets