Spiking-Diffusion: Vector Quantized Discrete Diffusion Model with Spiking Neural Networks (2308.10187v4)
Abstract: Spiking neural networks (SNNs) have tremendous potential for energy-efficient neuromorphic chips due to their binary and event-driven architecture. SNNs have been primarily used in classification tasks, but limited exploration on image generation tasks. To fill the gap, we propose a Spiking-Diffusion model, which is based on the vector quantized discrete diffusion model. First, we develop a vector quantized variational autoencoder with SNNs (VQ-SVAE) to learn a discrete latent space for images. In VQ-SVAE, image features are encoded using both the spike firing rate and postsynaptic potential, and an adaptive spike generator is designed to restore embedding features in the form of spike trains. Next, we perform absorbing state diffusion in the discrete latent space and construct a spiking diffusion image decoder (SDID) with SNNs to denoise the image. Our work is the first to build the diffusion model entirely from SNN layers. Experimental results on MNIST, FMNIST, KMNIST, Letters, and Cifar10 demonstrate that Spiking-Diffusion outperforms the existing SNN-based generation model. We achieve FIDs of 37.50, 91.98, 59.23, 67.41, and 120.5 on the above datasets respectively, with reductions of 58.60\%, 18.75\%, 64.51\%, 29.75\%, and 44.88\% in FIDs compared with the state-of-art work. Our code will be available at \url{https://github.com/Arktis2022/Spiking-Diffusion}.
- “22.6 anp-i: A 28nm 1.5 pj/sop asynchronous spiking neural network processor enabling sub-o. 1 μ𝜇\muitalic_μj/sample on-chip learning for edge-ai applications,” in 2023 IEEE International Solid-State Circuits Conference (ISSCC). IEEE, 2023, pp. 21–23.
- “Spiking neural networks: A survey,” IEEE Access, vol. 10, pp. 60738–60764, 2022.
- “Spiking-gan: A spiking generative adversarial network using time-to-first-spike coding,” in 2022 International Joint Conference on Neural Networks (IJCNN). IEEE, 2022, pp. 1–7.
- “Fully spiking variational autoencoder,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2022, pp. 7059–7067.
- “The variational fair autoencoder,” arXiv preprint arXiv:1511.00830, 2015.
- “Neural discrete representation learning,” Advances in neural information processing systems, vol. 30, 2017.
- “Structured denoising diffusion models in discrete state-spaces,” Advances in Neural Information Processing Systems, vol. 34, pp. 17981–17993, 2021.
- Li Deng, “The mnist database of handwritten digit images for machine learning research [best of the web],” IEEE signal processing magazine, vol. 29, no. 6, pp. 141–142, 2012.
- “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747, 2017.
- “Deep learning for classical japanese literature,” arXiv preprint arXiv:1812.01718, 2018.
- “Emnist: Extending mnist to handwritten letters,” in 2017 international joint conference on neural networks (IJCNN). IEEE, 2017, pp. 2921–2926.
- “Learning multiple layers of features from tiny images,” 2009.
- “Spikingjelly,” https://github.com/fangwei123456/spikingjelly, 2020, Accessed: 2023-04-18.
- RB Stein and Alan Lloyd Hodgkin, “The frequency of nerve action potentials generated by applied currents,” Proceedings of the Royal Society of London. Series B. Biological Sciences, vol. 167, no. 1006, pp. 64–86, 1967.
- “Incorporating learnable membrane time constant to enhance learning of spiking neural networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2661–2671.
- “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
- “Deep learning in spiking neural networks,” Neural networks, vol. 111, pp. 47–63, 2019.
- “Rate coding or direct coding: Which one is better for accurate, robust, and energy-efficient spiking neural networks?,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 71–75.
- “Encoding of steady-state vowels in the auditory nerve: representation in terms of discharge rate,” The Journal of the Acoustical Society of America, vol. 66, no. 2, pp. 470–479, 1979.
- “Experience-dependent plasticity in adult visual cortex,” Neuron, vol. 52, no. 4, pp. 577–585, 2006.
- “Superspike: Supervised learning in multilayer spiking neural networks,” Neural computation, vol. 30, no. 6, pp. 1514–1541, 2018.
- “Going deeper with directly-trained larger spiking neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2021, pp. 11062–11070.