Going Deeper in Spiking Neural Networks: VGG and Residual Architectures (1802.02627v4)

Abstract: Over the past few years, Spiking Neural Networks (SNNs) have become popular as a possible pathway to enable low-power event-driven neuromorphic hardware. However, their application in machine learning have largely been limited to very shallow neural network architectures for simple problems. In this paper, we propose a novel algorithmic technique for generating an SNN with a deep architecture, and demonstrate its effectiveness on complex visual recognition problems such as CIFAR-10 and ImageNet. Our technique applies to both VGG and Residual network architectures, with significantly better accuracy than the state-of-the-art. Finally, we present analysis of the sparse event-driven computations to demonstrate reduced hardware overhead when operating in the spiking domain.

An Advanced Evaluation of Deep Spiking Neural Networks through VGG and Residual Architectures

Introduction

Spiking Neural Networks (SNNs) have garnered increasing interest due to their potential to enable low-power, event-driven neuromorphic hardware. Unlike traditional Artificial Neural Networks (ANNs) that rely on analog units, SNNs leverage binary spike-based computations, making them inherently more biologically plausible. Although SNNs offer significant advantages in power efficiency, their application has largely been confined to shallow architectures and simple datasets such as MNIST. This paper presents a comprehensive paper demonstrating the scalability and effectiveness of deep SNNs on complex visual recognition benchmarks like CIFAR-10 and ImageNet, using VGG-16 and Residual (ResNet) network architectures.

Methodology

The research explores the ANN-SNN conversion process, focusing on a proposed technique called "Spike-Norm." The algorithm aims to balance the threshold of each layer by accounting for the actual operation of the SNN during conversion. This technique contrasts with prior methods, which relied solely on ANN model activations for weight normalizations.

VGG Architectures

For the VGG-16 model, the proposed Spike-Norm algorithm sequentially normalizes the weights and thresholds of the neural layers by examining the maximum summation of weighted spike inputs, ensuring near-lossless conversion from ANN to SNN. The paper evaluated this on CIFAR-10 and ImageNet datasets, achieving a minimal error increment of 0.15% and 0.56%, respectively.

Residual Architectures

The research extended this exploration to ResNet architectures, incorporating several design constraints:

1. Inclusion of ReLUs at Junction Points: To balance the temporal delays between the identity and non-identity paths.
2. Uniform Thresholds for Fan-In Layers: Ensuring that all aggregated input layers have consistent spiking thresholds.
3. Pre-Processing Non-Residual Layers: Integrating initial plain convolutional layers to stabilize the threshold-balancing process.

These constraints enabled the researchers to implement deep SNNs with ResNet-20 on CIFAR-10 and ResNet-34 on ImageNet, achieving reasonable classification accuracies with minimal loss.

Results

The paper's results are noteworthy:

• CIFAR-10 with VGG-16: The baseline ANN error was 8.3%, and the converted SNN exhibited an error of 8.45% using Spike-Norm.
• ImageNet with VGG-16: A baseline top-1 error of 29.48% was slightly increased to 30.04% in SNN.
• CIFAR-10 with ResNet-20: The ANN baseline error of 10.9% increased to 12.54% post-conversion.
• ImageNet with ResNet-34: The baseline top-1 error of 29.31% rose to 34.53% for SNN.

The paper also highlighted the computational efficiency afforded by SNNs. It demonstrated that neuron spiking activity becomes sparser in deeper network layers, suggesting that event-driven hardware's power efficiency may increase with network depth.

Implications

The conversion of deep ANNs to SNNs without significant performance degradation has substantial implications for power-efficient neuromorphic computing. SNNs can now be considered viable for complex visual recognition tasks, paving the path for their use in large-scale machine learning applications. This development opens avenues for more adaptive, real-time AI systems that can operate on power-constrained devices.

Future Directions

The paper provides several future research directions:

1. Bias Incorporation in Spiking Neurons: Exploring methods to include bias terms in SNN operations to leverage Batch-Normalization.
2. Diverse Neuronal Functions: Investigating alternative neuronal models that may offer better ANN-SNN conversion.
3. Enhanced Threshold-Balancing: Developing more refined weight normalization techniques, especially for Residual Architectures, to bridge the performance gap further.

Conclusion

This work significantly advances the capabilities of SNNs, enabling their application to more complex datasets and deeper network structures like VGG-16 and ResNet. The proposed Spike-Norm algorithm has shown to effectively reduce accuracy loss during ANN-SNN conversion, establishing a critical step toward practical, low-power neuromorphic computation in sophisticated AI applications.