Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks (2303.04347v1)

Abstract: Spiking Neural Networks (SNNs) have gained great attraction due to their distinctive properties of low power consumption and fast inference on neuromorphic hardware. As the most effective method to get deep SNNs, ANN-SNN conversion has achieved comparable performance as ANNs on large-scale datasets. Despite this, it requires long time-steps to match the firing rates of SNNs to the activation of ANNs. As a result, the converted SNN suffers severe performance degradation problems with short time-steps, which hamper the practical application of SNNs. In this paper, we theoretically analyze ANN-SNN conversion error and derive the estimated activation function of SNNs. Then we propose the quantization clip-floor-shift activation function to replace the ReLU activation function in source ANNs, which can better approximate the activation function of SNNs. We prove that the expected conversion error between SNNs and ANNs is zero, enabling us to achieve high-accuracy and ultra-low-latency SNNs. We evaluate our method on CIFAR-10/100 and ImageNet datasets, and show that it outperforms the state-of-the-art ANN-SNN and directly trained SNNs in both accuracy and time-steps. To the best of our knowledge, this is the first time to explore high-performance ANN-SNN conversion with ultra-low latency (4 time-steps). Code is available at https://github.com/putshua/SNN\_conversion\_QCFS

Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks

This paper explores an advanced methodology for converting artificial neural networks (ANNs) to spiking neural networks (SNNs), achieving remarkable accuracy and ultra-low latency in SNN inference. The authors introduce a novel activation function to better approximate the activation dynamics of SNNs, addressing inherent conversion errors that have traditionally hindered practical applications of SNNs, particularly when operating under short inference time-steps.

Context and Motivation

SNNs are gaining traction due to their potential for energy-efficient computation and suitability for neuromorphic hardware, which mimics the event-driven nature of biological neurons. While direct training of deep SNNs using techniques such as backpropagation-through-time has advanced, it remains computationally expensive. ANN-SNN conversion offers a promising alternative, leveraging the robust training methodologies of ANNs, yet must overcome specific conversion-related challenges.

Conversion Challenges

Traditional ANN-SNN conversion approaches often suffer performance degradation at low latencies due to disparate firing rates between ANNs and SNNs. These conversion errors include clipping errors, quantization errors, and unevenness errors, each contributing to deviations in expected firing rates in SNNs compared to activation values in ANNs.

Proposed Solution

The authors propose replacing the ReLU activation function in source ANNs with the quantization clip-floor-shift (QCFS) activation function. This approach theoretically reduces the expected conversion error to zero, regardless of the mismatch between time-steps $T$ and quantization steps $L$. The QCFS introduces a shift term enabling accurate activation mapping irrespective of individual neuron variations. Empirically, this results in high-performance SNNs operating at ultra-low latency, as demonstrated on benchmark datasets such as CIFAR-10, CIFAR-100, and ImageNet.

Empirical Validation

The paper reports comprehensive evaluations showing the proposed methodology surpassing state-of-the-art techniques in both accuracy and inference speed on diverse dataset scales. Notably, it achieves top-1 accuracy on CIFAR-10 with significantly reduced time-steps, thus ensuring broad applicability of SNNs on scalable neuromorphic systems.

Implications and Future Work

This work promises significant implications for the deployment of SNNs in energy-sensitive applications and real-time processing tasks. The findings suggest a pathway for enhancing neuromorphic hardware efficiency, potentially impacting fields requiring robust, low-power computing. Future research may explore extending these conversion mechanisms to other network architectures and applications beyond image classification, providing further insights into the dynamic interactions within spiking systems.

Conclusion

The developed ANN-SNN conversion strategy marks a significant step towards practical and efficient use of spiking neural networks. By finely tuning the activation dynamics and addressing conversion errors via the QCFS function, this research contributes to optimizing the balance between computational efficiency and accuracy, a critical advance for neuromorphic computing and beyond.