Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation (2005.01807v1)

Abstract: Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) which can potentially lead to higher energy-efficiency in neuromorphic hardware implementations. Many works have shown that an SNN for inference can be formed by copying the weights from a trained Artificial Neural Network (ANN) and setting the firing threshold for each layer as the maximum input received in that layer. These type of converted SNNs require a large number of time steps to achieve competitive accuracy which diminishes the energy savings. The number of time steps can be reduced by training SNNs with spike-based backpropagation from scratch, but that is computationally expensive and slow. To address these challenges, we present a computationally-efficient training technique for deep SNNs. We propose a hybrid training methodology: 1) take a converted SNN and use its weights and thresholds as an initialization step for spike-based backpropagation, and 2) perform incremental spike-timing dependent backpropagation (STDB) on this carefully initialized network to obtain an SNN that converges within few epochs and requires fewer time steps for input processing. STDB is performed with a novel surrogate gradient function defined using neuron's spike time. The proposed training methodology converges in less than 20 epochs of spike-based backpropagation for most standard image classification datasets, thereby greatly reducing the training complexity compared to training SNNs from scratch. We perform experiments on CIFAR-10, CIFAR-100, and ImageNet datasets for both VGG and ResNet architectures. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.

• The paper presents a hybrid training framework that combines ANN-SNN conversion with spike timing dependent backpropagation to rapidly converge SNNs with fewer time steps.
• It introduces a novel surrogate gradient based on spike timing, enabling effective temporal credit assignment without excessive memory overhead.
• Experiments show near state-of-the-art accuracy on datasets like CIFAR-10 while reducing inference steps by 10 to 25 times, promoting energy-efficient neuromorphic applications.

Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation

The paper "Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation" presents a novel training methodology for Spiking Neural Networks (SNNs) that aims to address the limitations of both traditional ANN-SNN conversion techniques and direct spike-based backpropagation methods. The authors focus on improving the computational efficiency and reducing the latency of deep SNNs, which are known for their potential energy-efficiency in neuromorphic hardware implementations.

Problem Statement

SNNs, which mimic biological neural processes through asynchronous discrete spike events, offer a promising alternative to conventional Artificial Neural Networks (ANNs) for low-power applications. However, SNNs face significant challenges in training due to the non-differentiability of their spike-based nature, making standard gradient descent techniques ineffective. Traditional ANN-to-SNN conversion approaches allow for high accuracy by transferring weights from trained ANNs, but they often require thousands of time steps to match the performance of ANNs, significantly undermining the energy benefits. On the other hand, spike-based backpropagation can result in competitive accuracy with fewer time steps but is computationally intensive and slow to converge.

Proposed Hybrid Training Method

To solve these issues, the authors propose a hybrid training technique that leverages both ANN-SNN conversion and spike timing-dependent backpropagation (STDB). The proposed methodology involves:

1. Initialization: Convert an ANN to an SNN by deriving initial weights and thresholds, ensuring an accurate starting point.
2. Incremental Training with STDB: Use STDB, a novel learning rule that computes surrogate gradients based on neurons' spike timings, to perform fine-tuning. This approach allows the SNN to converge more rapidly, reducing the total number of required training epochs and enabling a significant decrease in the number of time steps needed for inference.

Key Contributions

The paper makes several important contributions:

• Hybrid Training Framework: By combining the strengths of ANN-SNN conversion and spike-based backpropagation, the approach achieves a balance between computational efficiency and accuracy.
• Novel Surrogate Gradient: The introduction of a spike-time dependent surrogate gradient for backpropagation differentiates this work from other spike-based training methodologies. This surrogate gradient allows for effective temporal credit assignment without accumulating excessive memory overhead from storing a neuron's complete spike history.
• Scalability and Performance: Experiments with this approach demonstrate near state-of-the-art accuracies on standard datasets, such as CIFAR-10, CIFAR-100, and ImageNet, at significantly reduced computational cost. Specifically, the hybrid methodology allowed networks to achieve similar accuracies using 10 to 25 times fewer time steps than purely converted SNNs, with a convergence of SNNs in less than 20 epochs.

Implications and Future Directions

This research has significant implications for energy-efficient AI, particularly in edge computing and applications constrained by power and computational resources. The hybrid training technique reduces the computational burden of SNN training, making it more feasible for large-scale applications.

Looking forward, advancing this methodology could involve exploring further optimizations in the surrogate gradient computation to handle even more complex tasks. Additionally, adopting similar principles in other local learning algorithms or expanding the hybrid learning framework across different neural network architectures and neuromorphic hardware platforms can drive further improvements in efficiency and scalability.

Overall, this paper contributes to the ongoing advancement of low-power, efficient AI systems by proposing a significant step forward in the training of SNNs, aligning computational methods closer to the promises of neuromorphic engineering.