Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures (1903.06379v4)

Abstract: Spiking Neural Networks (SNNs) have recently emerged as a prominent neural computing paradigm. However, the typical shallow SNN architectures have limited capacity for expressing complex representations while training deep SNNs using input spikes has not been successful so far. Diverse methods have been proposed to get around this issue such as converting off-the-shelf trained deep Artificial Neural Networks (ANNs) to SNNs. However, the ANN-SNN conversion scheme fails to capture the temporal dynamics of a spiking system. On the other hand, it is still a difficult problem to directly train deep SNNs using input spike events due to the discontinuous, non-differentiable nature of the spike generation function. To overcome this problem, we propose an approximate derivative method that accounts for the leaky behavior of LIF neurons. This method enables training deep convolutional SNNs directly (with input spike events) using spike-based backpropagation. Our experiments show the effectiveness of the proposed spike-based learning on deep networks (VGG and Residual architectures) by achieving the best classification accuracies in MNIST, SVHN and CIFAR-10 datasets compared to other SNNs trained with a spike-based learning. Moreover, we analyze sparse event-based computations to demonstrate the efficacy of the proposed SNN training method for inference operation in the spiking domain.

Enabling Spike-based Backpropagation for Training Deep Neural Network Architectures

This paper presents a method for training deep Spiking Neural Networks (SNNs) using spike-based backpropagation, addressing a significant challenge in neural computation. Traditional approaches to SNN training have struggled with the non-differentiable nature of spike generation, limiting the depth and expressiveness of SNNs. The authors propose an approximate derivative that incorporates the leaky dynamics of Leaky Integrate and Fire (LIF) neurons, enabling direct training of deep convolutional SNNs with spike inputs.

Technical Contribution

The authors present a novel spike-based supervised gradient descent backpropagation algorithm. This method adjusts for the discontinuity in the spike activation function by defining a pseudo-derivative for the LIF neuronal model. The approximation involves comparing the membrane potential dynamics of LIF neurons to Integrate and Fire (IF) neurons, accounting for the leaky behavior that necessitates more input current to reach firing thresholds.

The use of deep convolutional architectures like VGG and ResNet with small convolutional kernels and residual connections marks a significant advancement. These architectures facilitate the construction of deeper SNNs, mimicking successful models from ANN architectures, thereby enhancing pattern recognition capabilities.

Experimental Validation

The authors validate their methodology through experiments on standard datasets—MNIST, SVHN, CIFAR-10—and a neuromorphic dataset, N-MNIST. The proposed SNNs achieve superior or comparable classification accuracies to previous SNN models, particularly excelling beyond traditional spike-based learning methods. Notably, the method exhibits a classification accuracy on CIFAR-10 that rivals ANN-SNN conversion techniques.

In terms of computational efficiency, the paper presents evidence that deep SNNs trained with this method achieve significant reductions in inference latency and total spikes required per image compared to ANN-SNN converted networks. For instance, VGG9 and ResNet11 architectures notably require fewer computational resources for inference, suggesting potential advantages for deployment in energy-efficient neuromorphic hardware.

Implications and Future Prospects

By enabling effective training of very deep SNNs, this research contributes to bridging the performance gap between SNNs and ANNs. The framework described has implications for the development of neuromorphic hardware applications that leverage the sparse, event-driven nature of SNNs, offering potential enhancements in power efficiency and processing speed.

Future research could explore further integration of this methodology with emerging neuromorphic platforms to realize ultra-low-power computing solutions across diverse real-world applications. Additionally, extending this training approach to even more complex datasets and architectures could provide insights into the scalability of SNNs in practical scenarios.