Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks (2103.00476v1)

Abstract: Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) that comprise of spiking neurons to process asynchronous discrete signals. While more efficient in power consumption and inference speed on the neuromorphic hardware, SNNs are usually difficult to train directly from scratch with spikes due to the discreteness. As an alternative, many efforts have been devoted to converting conventional ANNs into SNNs by copying the weights from ANNs and adjusting the spiking threshold potential of neurons in SNNs. Researchers have designed new SNN architectures and conversion algorithms to diminish the conversion error. However, an effective conversion should address the difference between the SNN and ANN architectures with an efficient approximation \DSK{of} the loss function, which is missing in the field. In this work, we analyze the conversion error by recursive reduction to layer-wise summation and propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. This pipeline enables almost no accuracy loss between the converted SNNs and conventional ANNs with only $\sim1/10$ of the typical SNN simulation time. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.

Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks

The paper "Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks" presents a novel approach for converting traditional Artificial Neural Networks (ANNs) to Spiking Neural Networks (SNNs). SNNs are known for their biological inspiration, efficiency in power consumption, and rapid inference speeds on neuromorphic hardware. However, the training of SNNs is hindered by their discrete nature, which makes direct training challenging. This research focuses on optimizing the conversion process from ANNs to SNNs to mitigate accuracy loss and reduce simulation time.

Overview

The traditional approach to converting ANNs to SNNs involves transferring the weights from ANNs to SNNs and adjusting the spiking neuron threshold potentials to address errors arising from architectural differences. However, prior methods have not effectively accounted for the discrepancies between ANN and SNN architectures through an efficient approximation of loss functions. This paper introduces an innovative conversion strategy that leverages threshold balancing and soft-reset mechanisms to achieve minimal accuracy loss between the converted SNNs and original ANNs. Remarkably, the proposed pipeline operates at approximately one-tenth the usual simulation time required for SNNs.

Methodology

The authors critically analyze the conversion error by reducing it recursively to a layer-wise summation. They propose a strategic pipeline that modifies the ReLU activation function in the source ANN, aligning it more closely with the spiking frequency observed in the target SNN. This is accomplished through:

1. Threshold ReLU Modification: The activation function in the source ANN is customized by thresholding the maximum activation and shifting the turning point. This ensures uniformity in activation values across neurons, facilitating a reduction in simulation time required for activation.
2. Conversion Algorithm: The paper introduces a conversion algorithm that optimally controls the difference in activation values between the source ANN and target SNN, significantly reducing simulation length requirements compared to existing methods.

Results

The paper demonstrates through both theoretical analysis and empirical evidence that their conversion approach minimizes accuracy loss effectively. The accuracy of the converted models closely approximates that of the original ANNs, demanding much shorter simulation times. For complex models such as VGG-16 and ResNet-20, the proposed method outperforms existing conversion techniques in terms of both accuracy and efficiency.

Implications and Future Directions

From a practical perspective, the ability to convert ANNs to SNNs with minimal loss opens pathways for deploying SNNs on embedded platforms with restricted energy and memory resources. Theoretically, this work contributes to the understanding of conversion error hierarchy and its minimization through layer-wise analysis, potentially influencing the broader fields of neural network quantization and neuro-inspired computing.

Future research could explore further extensions of this framework to larger-scale problems and diverse datasets. The dual advantages of reduced simulation length and preserved accuracy in converted SNNs could catalyze advancements in applying SNNs to real-time applications in embedded systems, drone technology, and IoT devices.

This innovative method marks a significant step toward leveraging the biological advantages of SNNs in practical applications, fostering a deeper integration between bio-inspired computing and existing digital technologies.