Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks (1706.02609v3)

Abstract: Compared with artificial neural networks (ANNs), spiking neural networks (SNNs) are promising to explore the brain-like behaviors since the spikes could encode more spatio-temporal information. Although pre-training from ANN or direct training based on backpropagation (BP) makes the supervised training of SNNs possible, these methods only exploit the networks' spatial domain information which leads to the performance bottleneck and requires many complicated training skills. Another fundamental issue is that the spike activity is naturally non-differentiable which causes great difficulties in training SNNs. To this end, we build an iterative LIF model that is more friendly for gradient descent training. By simultaneously considering the layer-by-layer spatial domain (SD) and the timing-dependent temporal domain (TD) in the training phase, as well as an approximated derivative for the spike activity, we propose a spatio-temporal backpropagation (STBP) training framework without using any complicated technology. We achieve the best performance of multi-layered perceptron (MLP) compared with existing state-of-the-art algorithms over the static MNIST and the dynamic N-MNIST dataset as well as a custom object detection dataset. This work provides a new perspective to explore the high-performance SNNs for future brain-like computing paradigm with rich spatio-temporal dynamics.

• The paper introduces a novel spatio-temporal backpropagation method that fuses spatial and temporal dynamics for improved SNN training.
• It employs an iterative LIF model and gradient approximation techniques to address the challenges posed by non-differentiable spike activities.
• The approach achieves state-of-the-art accuracies of 98.89% on MNIST and 98.78% on N-MNIST, indicating strong potential for real-time neuromorphic applications.

Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks

The paper by Yujie Wu et al., titled "Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks," introduces a novel training framework for Spiking Neural Networks (SNNs) that leverages both spatial and temporal domain information. This distinct approach addresses some of the prevalent challenges in training SNNs and shows an enhancement in performance over traditional methods.

Overview

Spiking Neural Networks (SNNs) provide a promising avenue for exploring brain-like behaviors as they can encode more spatio-temporal information compared to traditional Artificial Neural Networks (ANNs). The primary challenge in training SNNs arises from the complex dynamics and non-differentiable nature of spike activities. Existing methods, including unsupervised learning, indirect supervised learning through ANN pre-training, and direct supervised learning, are typically constrained by their reliance on spatial domain information, which often leads to performance bottlenecks.

Contributions

The authors present the Spatio-Temporal Backpropagation (STBP) training framework that targets these limitations by integrating both spatial domain (SD) and temporal domain (TD) information during the training phase. The key contributions of this work include:

1. Iterative Leaky Integrate-and-Fire (LIF) Model: An iterative version of the LIF model is proposed to make gradient descent training more feasible for SNNs. This model encapsulates both spatial accumulations of synaptic inputs and the leaky temporal memory of the neuronal potential, thus enabling a richer representation of spatio-temporal dynamics.
2. Spatio-Temporal Backpropagation (STBP): The STBP algorithm explicitly considers error propagation across both spatial and temporal dimensions. This dual consideration is realized by unfolding the state space in both directions, allowing for chain-rule propagation and iterative error updates.
3. Derivative Approximation for Spike Activity: Given the non-differentiable nature of spike activities, the authors introduce several approximation curves (rectangular, polynomial, sigmoid, and Gaussian functions) to smooth the gradients and facilitate effective training.

Numerical Results

The effectiveness of the STBP framework is demonstrated through experiments on both static (MNIST, custom object detection dataset) and dynamic (N-MNIST) datasets. Highlights include:

• Static Dataset (MNIST): The STBP approach achieved a testing accuracy of 98.89% on the MNIST dataset, surpassing existing state-of-the-art SNN models.
• Dynamic Dataset (N-MNIST): On the dynamic N-MNIST dataset, the proposed method attained an accuracy of 98.78%, outperforming both ANN and other SNN methods.

Implications and Future Work

Theoretical Implications: The integration of temporal domain dynamics in backpropagation frameworks could substantially improve the accuracy and stability of SNNs. This methodology highlights the potential of SNNs to exploit temporal features more effectively than traditional DNNs, particularly for tasks that involve dynamic data.

Practical Implications: The STBP framework's avoidance of complex training techniques like error normalization and weight regularization signifies a more streamlined and potentially hardware-friendly approach. This poses significant advantages for real-time, online learning scenarios on neuromorphic hardware.

Future Directions:

1. Dynamic Data Processing: Extending the applicability of STBP to other domains such as video stream analysis and speech recognition can help in fully leveraging its capacity to handle dynamic data.
2. Hardware Acceleration: Investigating the implementation efficiency of STBP on GPUs/CPUs and neuromorphic chips can facilitate the widespread use of large-scale SNNs.

Conclusion

The paper presents a comprehensive approach to enhance the performance of SNNs through spatio-temporal backpropagation. By addressing the non-differentiability of spikes and leveraging both spatial and temporal domain information, the proposed method achieves superior performance on benchmark datasets. This work lays the foundation for future explorations into high-performance SNN training methodologies and their practical applications.