Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Tensor Decomposition Based Attention Module for Spiking Neural Networks (2310.14576v2)

Published 23 Oct 2023 in cs.LG and cs.CV

Abstract: The attention mechanism has been proven to be an effective way to improve spiking neural network (SNN). However, based on the fact that the current SNN input data flow is split into tensors to process on GPUs, none of the previous works consider the properties of tensors to implement an attention module. This inspires us to rethink current SNN from the perspective of tensor-relevant theories. Using tensor decomposition, we design the \textit{projected full attention} (PFA) module, which demonstrates excellent results with linearly growing parameters. Specifically, PFA is composed by the \textit{linear projection of spike tensor} (LPST) module and \textit{attention map composing} (AMC) module. In LPST, we start by compressing the original spike tensor into three projected tensors using a single property-preserving strategy with learnable parameters for each dimension. Then, in AMC, we exploit the inverse procedure of the tensor decomposition process to combine the three tensors into the attention map using a so-called connecting factor. To validate the effectiveness of the proposed PFA module, we integrate it into the widely used VGG and ResNet architectures for classification tasks. Our method achieves state-of-the-art performance on both static and dynamic benchmark datasets, surpassing the existing SNN models with Transformer-based and CNN-based backbones.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Scalable tensor factorizations for incomplete data. Chemometrics and Intelligent Laboratory Systems, 106(1):41–56, March 2011. 10.1016/j.chemolab.2010.08.004. URL https://doi.org/10.1016/j.chemolab.2010.08.004.
  2. Factorized dynamic fully-connected layers for neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1374–1383, 2023.
  3. Error-backpropagation in temporally encoded networks of spiking neurons. Neurocomputing, 48(1-4):17–37, October 2002. 10.1016/s0925-2312(01)00658-0. URL https://doi.org/10.1016/s0925-2312(01)00658-0.
  4. A spatial-channel-temporal-fused attention for spiking neural networks, 2022. URL https://arxiv.org/abs/2209.10837.
  5. Tensor low-rank reconstruction for semantic segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 52–69, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58520-4.
  6. Lisnn: improving spiking neural networks with lateral interactions for robust object recognition. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20, 2021. ISBN 9780999241165.
  7. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 29(6):141–142, 2012. 10.1109/MSP.2012.2211477.
  8. Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting. In International Conference on Learning Representations (ICLR), 2021.
  9. Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2015. 10.1109/IJCNN.2015.7280696.
  10. Spikingjelly. https://github.com/fangwei123456/spikingjelly, 2020. Accessed: 2023-01-16.
  11. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056–21069, 2021a.
  12. Incorporating Learnable Membrane Time Constant To Enhance Learning of Spiking Neural Networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2661–2671, 2021b.
  13. Johan Håstad. Tensor rank is NP-complete. Journal of Algorithms, 11(4):644–654, December 1990. 10.1016/0196-6774(90)90014-6. URL https://doi.org/10.1016/0196-6774(90)90014-6.
  14. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2016. 10.1109/cvpr.2016.90. URL https://doi.org/10.1109/cvpr.2016.90.
  15. A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology, 117(4):500–544, August 1952. 10.1113/jphysiol.1952.sp004764. URL https://doi.org/10.1113/jphysiol.1952.sp004764.
  16. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018a.
  17. Spiking deep residual network, 2018b. URL https://arxiv.org/abs/1805.01352.
  18. Advancing deep residual learning by solving the crux of degradation in spiking neural networks, 2022. URL https://arxiv.org/abs/2201.07209.
  19. Eugene M Izhikevich. Simple model of spiking neurons. IEEE Transactions on neural networks, 14(6):1569–1572, 2003.
  20. 2019.
  21. Fully spiking variational autoencoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7059–7067, 2022.
  22. Optimizing Deeper Spiking Neural Networks for Dynamic Vision Sensing. Neural Networks, 144:686–698, 2021.
  23. Neural architecture search for spiking neural networks. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV, pages 36–56. Springer, 2022.
  24. Tensor decompositions and applications. SIAM Review, 51(3):455–500, August 2009a. 10.1137/07070111x. URL https://doi.org/10.1137/07070111x.
  25. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009b. 10.1137/07070111X. URL https://doi.org/10.1137/07070111X.
  26. Tensor regression networks, 2017. URL https://arxiv.org/abs/1707.08308.
  27. Factorized higher-order cnns with an application to spatio-temporal emotion estimation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6059–6068, 2019. URL https://api.semanticscholar.org/CorpusID:214743123.
  28. Factorized higher-order cnns with an application to spatio-temporal emotion estimation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020. 10.1109/cvpr42600.2020.00610. URL http://dx.doi.org/10.1109/CVPR42600.2020.00610.
  29. Cifar-10 (canadian institute for advanced research). URL http://www.cs.toronto.edu/~kriz/cifar.html.
  30. Learning multiple layers of features from tiny images. 2009.
  31. Louis Lapique. Recherches quantitatives sur l’excitation electrique des nerfs traitee comme une polarization. Journal of Physiology and Pathology, 9:620–635, 1907.
  32. Large separable kernel attention: Rethinking the large kernel attention design in cnn. Expert Systems with Applications, 236:121352, February 2024. ISSN 0957-4174. 10.1016/j.eswa.2023.121352. URL http://dx.doi.org/10.1016/j.eswa.2023.121352.
  33. CIFAR10-DVS: An event-stream dataset for object classification. Frontiers in Neuroscience, 11, May 2017. 10.3389/fnins.2017.00309. URL https://doi.org/10.3389/fnins.2017.00309.
  34. Differentiable Spike: Rethinking Gradient-Descent for Training Spiking Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 23426–23439, 2021.
  35. Tensor decomposition for model reduction in neural networks: A review [feature]. IEEE Circuits and Systems Magazine, 23(2):8–28, 2023.
  36. Wolfgang Maass. Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9):1659–1671, 1997. ISSN 0893-6080. https://doi.org/10.1016/S0893-6080(97)00011-7. URL https://www.sciencedirect.com/science/article/pii/S0893608097000117.
  37. Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation. ArXiv preprint arXiv:2205.00459, 2022.
  38. Recurrent models of visual attention, 2014. URL https://arxiv.org/abs/1406.6247.
  39. Tensorizing neural networks. In Neural Information Processing Systems, 2015. URL https://api.semanticscholar.org/CorpusID:290242.
  40. Converting static image datasets to spiking neuromorphic datasets using saccades. Frontiers in Neuroscience, 9, November 2015. 10.3389/fnins.2015.00437. URL https://doi.org/10.3389/fnins.2015.00437.
  41. Vtsnn: A virtual temporal spiking neural network. Frontiers in neuroscience, 17:1091097, 2023.
  42. Gated attention coding for training high-performance and efficient spiking neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 601–610, 2024.
  43. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2021.
  44. Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. In International Conference on Learning Representations (ICLR).
  45. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience, 11, December 2017. 10.3389/fnins.2017.00682. URL https://doi.org/10.3389/fnins.2017.00682.
  46. Tensor decomposition for signal processing and machine learning. IEEE Transactions on Signal Processing, 65(13):3551–3582, 2017. 10.1109/TSP.2017.2690524.
  47. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556, 2015.
  48. Inertial accelerated sgd algorithms for solving large-scale lower-rank tensor cp decomposition problems. Journal of Computational and Applied Mathematics, 423:114948, 2023.
  49. Signed neuron with memory: Towards simple, accurate and high-efficient ann-snn conversion. In International Joint Conference on Artificial Intelligence, 2022.
  50. Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10552–10562, 2023.
  51. Event-driven learning for spiking neural networks. arXiv preprint arXiv:2403.00270, 2024.
  52. A tandem learning rule for effective training and rapid inference of deep spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2021a.
  53. Progressive tandem learning for pattern recognition with deep spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7824–7840, 2021b.
  54. Spatio-temporal backpropagation for training high-performance spiking neural networks. Frontiers in Neuroscience, 12, May 2018. 10.3389/fnins.2018.00331. URL https://doi.org/10.3389/fnins.2018.00331.
  55. Direct training for spiking neural networks: Faster, larger, better. Proceedings of the AAAI Conference on Artificial Intelligence, 33:1311–1318, July 2019a. 10.1609/aaai.v33i01.33011311. URL https://doi.org/10.1609/aaai.v33i01.33011311.
  56. Direct Training for Spiking Neural Networks: Faster, Larger, Better. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 1311–1318, 2019b. 10.1609/aaai.v33i01.33011311.
  57. LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing. IEEE Transactions on Neural Networks and Learning Systems, pages 1–14, 2021c. 10.1109/TNNLS.2021.3073016.
  58. Tensor wheel decomposition and its tensor completion application. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27008–27020. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/acbfe708197ff78ad04cc1beb1710979-Paper-Conference.pdf.
  59. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. ArXiv, abs/1708.07747, 2017. URL https://api.semanticscholar.org/CorpusID:702279.
  60. Fast autoregressive tensor decomposition for online real-time traffic flow prediction. Knowledge-Based Systems, 282:111125, December 2023. ISSN 0950-7051. 10.1016/j.knosys.2023.111125. URL http://dx.doi.org/10.1016/j.knosys.2023.111125.
  61. Deep multi-task representation learning: A tensor factorisation approach. ArXiv, abs/1605.06391, 2016. URL https://api.semanticscholar.org/CorpusID:3047732.
  62. Temporal-wise attention spiking neural networks for event streams classification. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, October 2021. 10.1109/iccv48922.2021.01006. URL https://doi.org/10.1109/iccv48922.2021.01006.
  63. Attention spiking neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  64. Bio-inspired active learning method in spiking neural network. Knowledge-Based Systems, 261:110193, February 2023. ISSN 0950-7051. 10.1016/j.knosys.2022.110193. URL http://dx.doi.org/10.1016/j.knosys.2022.110193.
  65. Rectified linear postsynaptic potential function for backpropagation in deep spiking neural networks. IEEE transactions on neural networks and learning systems, 33(5):1947–1958, 2021.
  66. Spike-train level backpropagation for training deep recurrent spiking neural networks. Curran Associates Inc., Red Hook, NY, USA, 2019.
  67. Going deeper with directly-trained larger spiking neural networks, 2020. URL https://arxiv.org/abs/2011.05280.
  68. Going Deeper With Directly-Trained Larger Spiking Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 11062–11070, 2021.
  69. Spikformer: When spiking neural network meets transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=frE4fUwz_h.
  70. Tcja-snn: Temporal-channel joint attention for spiking neural networks, 2022. URL https://arxiv.org/abs/2206.10177.
Citations (6)

Summary

We haven't generated a summary for this paper yet.