Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AutoST: Training-free Neural Architecture Search for Spiking Transformers (2307.00293v2)

Published 1 Jul 2023 in cs.NE, cs.CV, and cs.LG

Abstract: Spiking Transformers have gained considerable attention because they achieve both the energy efficiency of Spiking Neural Networks (SNNs) and the high capacity of Transformers. However, the existing Spiking Transformer architectures, derived from Artificial Neural Networks (ANNs), exhibit a notable architectural gap, resulting in suboptimal performance compared to their ANN counterparts. Manually discovering optimal architectures is time-consuming. To address these limitations, we introduce AutoST, a training-free NAS method for Spiking Transformers, to rapidly identify high-performance Spiking Transformer architectures. Unlike existing training-free NAS methods, which struggle with the non-differentiability and high sparsity inherent in SNNs, we propose to utilize Floating-Point Operations (FLOPs) as a performance metric, which is independent of model computations and training dynamics, leading to a stronger correlation with performance. Our extensive experiments show that AutoST models outperform state-of-the-art manually or automatically designed SNN architectures on static and neuromorphic datasets. Full code, model, and data are released for reproduction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (25)
  1. Wolfgang Maass, “Networks of spiking neurons: The third generation of neural network models,” Neural networks, vol. 10, no. 9, pp. 1659–1671, 1997.
  2. “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020.
  3. “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
  4. “Spikformer: When Spiking Neural Network Meets Transformer,” arXiv preprint arXiv:2209.15425, 2022.
  5. “Efficient Spiking Transformer Enabled By Partial Information,” Oct. 2022, arXiv:2210.01208 [cs].
  6. “Neural architecture search for spiking neural networks,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV. 2022, pp. 36–56, Springer.
  7. “AutoSNN: Towards energy-efficient spiking neural networks,” in International Conference on Machine Learning. 2022, pp. 16253–16269, PMLR.
  8. “Training-free transformer architecture search,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10894–10903.
  9. “Proxylessnas: Direct neural architecture search on target task and hardware,” arXiv preprint arXiv:1812.00332, 2018.
  10. “Pruning neural networks without any data by iteratively conserving synaptic flow,” Advances in neural information processing systems, vol. 33, pp. 6377–6389, 2020.
  11. “Snip: Single-shot network pruning based on connection sensitivity,” arXiv preprint arXiv:1810.02340, 2018.
  12. “Neural tangent kernel: Convergence and generalization in neural networks,” Advances in neural information processing systems, vol. 31, 2018.
  13. “Neural architecture search without training,” in International Conference on Machine Learning. 2021, pp. 7588–7598, PMLR.
  14. “Neuromorphic vision datasets for pedestrian detection, action recognition, and fall detection,” Frontiers in neurorobotics, vol. 13, pp. 38, 2019.
  15. “Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks,” in International Conference on Learning Representations, 2021.
  16. “Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting,” arXiv preprint arXiv:2202.11946, 2022.
  17. “Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12444–12453.
  18. “DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks,” Dec. 2020.
  19. “Rmp-snn: Residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13558–13567.
  20. “Going deeper with directly-trained larger spiking neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 11062–11070.
  21. “Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 11, pp. 6249–6262, 2021.
  22. “Incorporating learnable membrane time constant to enhance learning of spiking neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2661–2671.
  23. “Differentiable spike: Rethinking gradient-descent for training spiking neural networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 23426–23439, 2021.
  24. “Going Deeper With Directly-Trained Larger Spiking Neural Networks,” Dec. 2020.
  25. “Scaling vision transformers,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12104–12113.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now