Surrogate Gradient Learning in Spiking Neural Networks (1901.09948v2)

Abstract: Spiking neural networks are nature's versatile solution to fault-tolerant and energy efficient signal processing. To translate these benefits into hardware, a growing number of neuromorphic spiking neural network processors attempt to emulate biological neural networks. These developments have created an imminent need for methods and tools to enable such systems to solve real-world signal processing problems. Like conventional neural networks, spiking neural networks can be trained on real, domain specific data. However, their training requires overcoming a number of challenges linked to their binary and dynamical nature. This article elucidates step-by-step the problems typically encountered when training spiking neural networks, and guides the reader through the key concepts of synaptic plasticity and data-driven learning in the spiking setting. To that end, it gives an overview of existing approaches and provides an introduction to surrogate gradient methods, specifically, as a particularly flexible and efficient method to overcome the aforementioned challenges.

• The paper introduces surrogate gradient methods that replace non-differentiable spike functions with smooth approximations for effective gradient-based training.
• It maps spiking neural network dynamics to recurrent neural networks, leveraging established optimization techniques from traditional ANNs.
• The approach enhances the potential for energy-efficient and fault-tolerant neuromorphic computing systems in practical applications.

Surrogate Gradient Learning in Spiking Neural Networks: An Overview

The paper "Surrogate Gradient Learning in Spiking Neural Networks," authored by Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke, addresses key challenges faced in training Spiking Neural Networks (SNNs) and introduces the concept of Surrogate Gradient (SG) methods as a promising solution.

The primary motivation behind training SNNs is to leverage their energy efficiency and fault-tolerant capabilities, which are inspired by biological neural systems. Unlike traditional Artificial Neural Networks (ANNs), SNNs process information through spikes, analogous to the way biological neurons operate. This spiking mechanism introduces binary and dynamical complexities that are challenging to optimize using conventional gradient-based methods.

Key Contributions and Methods

1. Mapping SNNs to RNNs: The authors demonstrate a formal equivalence between SNNs and Recurrent Neural Networks (RNNs) by mapping the leaky integrate-and-fire (LIF) neuron model to an RNN with binary activation functions. This mapping is crucial as it allows the application of existing RNN training methods to SNNs, providing a conceptual framework for the rest of the paper.
2. Challenges in Training SNNs: The paper identifies two major challenges in training SNNs:
   • The non-differentiable nature of the spiking function, which hinders the computation of gradients required for optimization.
   • The computational and memory constraints associated with backpropagation through time (BPTT) and other gradient-based methods.
3. Surrogate Gradient Methods: The core contribution of the paper is the introduction and detailed discussion of SG methods. These methods involve replacing the non-differentiable spiking function with a smooth, differentiable surrogate function during backpropagation. This approach facilitates the use of gradient-based optimization algorithms while retaining the spiking dynamics of the network.
4. Smoothed Approaches vs. SG Methods: The paper contrasts SG methods with smoothed approaches that alter the neuron model itself to ensure differentiability. SG methods are preferred as they do not change the underlying model but provide a continuous relaxation of the gradient for optimization purposes.
5. Applications and Algorithms: The paper reviews various applications of SG methods, including:
   • Feedback Alignment (FA) and Direct Feedback Alignment (DFA): These methods approximate gradient backpropagation by using random matrices for error propagation, making the algorithm more local and efficient.
   • SuperSpike: A biologically plausible three-factor learning rule that combines SG methods with synaptic eligibility traces to solve the temporal credit assignment problem. It has been shown to perform well in supervised learning tasks involving spike timing.
   • Local Errors: The introduction of local loss functions to generate local errors, facilitating scalable and efficient training of deep SNNs.

Implications and Future Directions

The implications of this research are considerable for both theoretical and practical aspects of machine learning and neuromorphic computing. The SG methods provide a bridge between the high efficiency of SNNs and the powerful optimization techniques developed for ANNs. This research opens several avenues for future exploration:

• Theoretical Understanding: Further studies are needed to better understand the theoretical foundations of SG methods and their relationship to biological learning rules.
• Hardware Implementations: The principles discussed could be implemented in neuromorphic hardware to develop low-power, high-efficiency computing systems.
• Advanced Architectures: Extending SG methods to more complex network architectures, such as convolutional SNNs and recurrent SNNs, could enhance their applicability to a broader range of real-world problems.

In conclusion, the paper provides a comprehensive overview of the challenges and solutions in training SNNs using SG methods. By addressing the non-differentiability of spiking functions and proposing solutions that balance computational efficiency with biological plausibility, this research represents a significant step forward in the development of energy-efficient, fault-tolerant neural network models.