Deep Learning in Spiking Neural Networks (1804.08150v4)

Abstract: In recent years, deep learning has been a revolution in the field of machine learning, for computer vision in particular. In this approach, a deep (multilayer) artificial neural network (ANN) is trained in a supervised manner using backpropagation. Huge amounts of labeled examples are required, but the resulting classification accuracy is truly impressive, sometimes outperforming humans. Neurons in an ANN are characterized by a single, static, continuous-valued activation. Yet biological neurons use discrete spikes to compute and transmit information, and the spike times, in addition to the spike rates, matter. Spiking neural networks (SNNs) are thus more biologically realistic than ANNs, and arguably the only viable option if one wants to understand how the brain computes. SNNs are also more hardware friendly and energy-efficient than ANNs, and are thus appealing for technology, especially for portable devices. However, training deep SNNs remains a challenge. Spiking neurons' transfer function is usually non-differentiable, which prevents using backpropagation. Here we review recent supervised and unsupervised methods to train deep SNNs, and compare them in terms of accuracy, but also computational cost and hardware friendliness. The emerging picture is that SNNs still lag behind ANNs in terms of accuracy, but the gap is decreasing, and can even vanish on some tasks, while the SNNs typically require much fewer operations.

Deep Learning in Spiking Neural Networks

The paper, "Deep Learning in Spiking Neural Networks," by Amirhossein Tavanaei et al., provides a comprehensive review of the advancements in deep learning models based on spiking neural networks (SNNs). The work predominantly explores the training methodologies of deep SNNs and evaluates their performance relative to conventional deep neural networks (DNNs).

Introduction

The authors begin by highlighting the foundational distinctions between artificial neural networks (ANNs) and biological neural networks. ANNs typically use continuous-valued activations, while biological neurons communicate via discrete spikes. This biologically accurate spiking communication method makes SNNs more realistic and potentially more energy-efficient than traditional ANNs, especially relevant for portable devices. However, the non-differentiable nature of spiking neurons poses significant challenges for training deep SNNs.

SNN Architectures and Learning Methods

The paper segments recent advances into multiple facets of SNN architectures and learning methods:

1. Spiking Neural Networks: A Biologically Inspired Approach to Information Processing:
   • SNNs operate through spiking neurons and synapses modeled by adjustable weights, translating analog inputs into spike trains. Various neuron models such as the Hodgkin-Huxley model, Izhikevich neurons, and the leaky integrate-and-fire (LIF) model are discussed.
2. Learning Rules in SNNs:
   • Unsupervised Learning via STDP: The paper explores spike-timing-dependent plasticity (STDP) as a primary unsupervised learning mechanism, emphasizing the local adaptation of synapses based on spike timing.
   • Probabilistic STDP: It examines probabilistic interpretations of STDP facilitating Bayesian inference mechanisms in SNNs.
   • Supervised Learning: Methods such as SpikeProp, ReSuMe, and the Chronotron employing variations of backpropagation and gradient descent tailored for SNNs are explored. These include emerging strategies like BP-STDP and temporal backpropagation to overcome non-differentiability challenges.

Deep Learning in SNNs

The synthesis elucidates the structure and performance of various deep learning architectures in the context of SNNs:

1. Deep, Fully Connected SNNs:
   • The development of multi-layer SNNs using gradient-based methods, like those by O'Connor et al. and Lee et al., which achieve high-performance metrics on benchmarks like MNIST, is reviewed. The transition from offline training to hardware-optimized implementations is also examined.
2. Spiking Convolutional Neural Networks (Spiking CNNs):
   • Spiking CNNs leverage convolutional and pooling layers to draw features from visual data. Layer-wise unsupervised learning rules like STDP and representation learning methods, and their efficacy verified through empirical results, are discussed. Conversion techniques to translate conventional CNNs to SNNs, as explored by Diehl et al. and Rueckauer et al., are also analyzed.
3. Spiking Deep Belief Networks (Spiking DBNs):
   • The structure of spiking DBNs, stacking spiking restricted Boltzmann machines (RBMs) through methodological training steps, is outlined. Studies illustrating the implementation of spiking RBMs and DBNs in energy-efficient neuromorphic systems are cited.
4. Recurrent SNNs:
   • Recurrent SNNs are critical for processing temporal sequences, with approaches such as spiking LSTMs and Liquid State Machines (LSMs) being emphasized. The adaptation of traditional RNN architectures to spiking domains and their applications in handling sequential data is covered thoroughly.

Performance Comparison and Implications

The authors present a meticulous performance evaluation of various SNN architectures on standard benchmarks like MNIST and CIFAR, highlighting that while SNNs currently lag behind DNNs in some aspects, the accuracy gap is diminishing. Notably, SNNs tend to require fewer computational resources, showcasing their potential for power-efficient implementations on hardware, making them advantageous for portable and real-time applications.

Conclusions and Future Directions

The review concludes by acknowledging the tangible progress in SNN research, thanks to innovative training methods and architectural advancements. The confluence of deep learning techniques with spiking models promises a future where SNNs could surpass traditional neural networks in both performance and energy efficiency. The paper posits that ongoing developments will continue to bridge the accuracy gap, fostering advances in both theoretical neuroscience and practical AI applications. Future research will likely focus on improving training algorithms and exploring novel SNN architectures to fully harness the power of bio-inspired computing.

In essence, this comprehensive review by Tavanaei et al. elucidates the multifaceted progress in the field of SNNs and sets a foundation for further innovation aimed at aligning engineering applications with biologically realistic models.