Parallel Hyperparameter Optimization Of Spiking Neural Network (2403.00450v1)

Abstract: Spiking Neural Networks (SNN). SNNs are based on a more biologically inspired approach than usual artificial neural networks. Such models are characterized by complex dynamics between neurons and spikes. These are very sensitive to the hyperparameters, making their optimization challenging. To tackle hyperparameter optimization of SNNs, we initially extended the signal loss issue of SNNs to what we call silent networks. These networks fail to emit enough spikes at their outputs due to mistuned hyperparameters or architecture. Generally, search spaces are heavily restrained, sometimes even discretized, to prevent the sampling of such networks. By defining an early stopping criterion detecting silent networks and by designing specific constraints, we were able to instantiate larger and more flexible search spaces. We applied a constrained Bayesian optimization technique, which was asynchronously parallelized, as the evaluation time of a SNN is highly stochastic. Large-scale experiments were carried-out on a multi-GPU Petascale architecture. By leveraging silent networks, results show an acceleration of the search, while maintaining good performances of both the optimization algorithm and the best solution obtained. We were able to apply our methodology to two popular training algorithms, known as spike timing dependent plasticity and surrogate gradient. Early detection allowed us to prevent worthless and costly computation, directing the search toward promising hyperparameter combinations. Our methodology could be applied to multi-objective problems, where the spiking activity is often minimized to reduce the energy consumption. In this scenario, it becomes essential to find the delicate frontier between low-spiking and silent networks. Finally, our approach may have implications for neural architecture search, particularly in defining suitable spiking architectures.

• The paper introduces a parallel Bayesian algorithm that efficiently optimizes hyperparameters of SNNs by eliminating silent networks early.
• It employs a spike-based early stopping criterion to allocate computational resources to viable, dynamic spiking networks, reducing overhead.
• Experiments on STDP and SLAYER training across MNIST and DVS128 show significant improvements in resource utilization and scalability.

Enhancing the Efficiency of Spiking Neural Networks Through Parallel Hyperparameter Optimization

Introduction

The optimization of Spiking Neural Networks (SNNs) poses a unique set of challenges not encountered in the training of traditional Artificial Neural Networks. This paper explores a novel approach to hyperparameter optimization (HPO) in SNNs that leverages parallel computation and constrained Bayesian optimization. The focus is on addressing the challenges posed by "silent networks" - a scenario where networks show insufficient spiking activity, making them infeasible solutions for tasks requiring dynamic spiking responses.

Methodology

The methodology centers around the development of a scalable, constrained Bayesian-based optimization algorithm, specifically designed to mitigate the sampling of silent networks within high-dimensional search spaces. The optimization process is enhanced by a spike-based early stopping criterion, which swiftly identifies and discards silent networks, thereby speeding up the optimization cycle. Furthermore, this approach is parallelized through asynchronous computation on multi-GPU Petascale architectures, ensuring scalability and efficiency.

Experiments

Experiments are carried out using two primary training algorithms: Spike Timing Dependent Plasticity (STDP) and a surrogate gradient method (SLAYER), across different datasets (MNIST encoded in Poisson spikes and DVS128 Gesture). The paper rigorously tests the optimization algorithm across a variety of SNN configurations, assessing its effectiveness in navigating through high-dimensional search spaces while avoiding the computation of silent networks.

Results

The optimization algorithm demonstrates a remarkable ability to identify and focus on viable spiking networks, significantly mitigating the computational overhead associated with silent networks. For instance, despite a considerable portion of the networks being stopped due to insufficient spiking activity (73% in one experiment), these networks only consumed 36% of the total computational resources. This indicates the efficiency of the spike-based early stopping criterion in conjunction with the optimization algorithm in preserving computational resources for more promising network configurations.

Implications

The findings have several practical and theoretical implications:

• Scalability: The approach effectively handles high-dimensional search spaces, a crucial consideration for the complex architectures typical of SNNs.
• Efficiency: By focusing computational resources on feasible solutions, the methodology significantly improves the efficiency of the HPO process.
• Generalization: The methodology is adaptable to both popular families of SNN training algorithms, showcasing its versatility.
• Future Research Directions: The paper opens avenues for further investigation, particularly around multi-objective optimization and its applicability in Neural Architecture Search (NAS) for SNNs.

Conclusion

This paper presents a pioneering approach to the hyperparameter optimization of Spiking Neural Networks by integrating a novel early stopping criterion and constrained Bayesian optimization within a parallel computational framework. The methodology not only addresses the specific challenge of silent networks but also sets a foundation for future explorations into efficient and scalable HPO strategies for SNNs.