Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Temporal Spiking Neural Networks with Synaptic Delay for Graph Reasoning (2405.16851v1)

Published 27 May 2024 in cs.NE, cs.AI, and cs.LG

Abstract: Spiking neural networks (SNNs) are investigated as biologically inspired models of neural computation, distinguished by their computational capability and energy efficiency due to precise spiking times and sparse spikes with event-driven computation. A significant question is how SNNs can emulate human-like graph-based reasoning of concepts and relations, especially leveraging the temporal domain optimally. This paper reveals that SNNs, when amalgamated with synaptic delay and temporal coding, are proficient in executing (knowledge) graph reasoning. It is elucidated that spiking time can function as an additional dimension to encode relation properties via a neural-generalized path formulation. Empirical results highlight the efficacy of temporal delay in relation processing and showcase exemplary performance in diverse graph reasoning tasks. The spiking model is theoretically estimated to achieve $20\times$ energy savings compared to non-spiking counterparts, deepening insights into the capabilities and potential of biologically inspired SNNs for efficient reasoning. The code is available at https://github.com/pkuxmq/GRSNN.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Provable advantages for graph algorithms in spiking neural networks. In ACM Symposium on Parallelism in Algorithms and Architectures (SPAA), 2021.
  2. Lowfer: Low-rank bilinear pooling for link prediction. In International Conference on Machine Learning (ICML), 2020.
  3. Long short-term memory and learning-to-learn in networks of spiking neurons. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
  4. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11(1):1–15, 2020.
  5. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems (NeurIPS), 2013.
  6. Heterogeneous neuronal and synaptic dynamics for spike-efficient unsupervised learning: Theory and design principles. In International Conference on Learning Representations (ICLR), 2023.
  7. Temporal coding in spiking neural networks with alpha synaptic function. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
  8. Hyperspherical variational auto-encoders. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
  9. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro, 38(1):82–99, 2018.
  10. Convolutional 2d knowledge graph embeddings. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
  11. Dold, D. Relational representation learning with spike trains. In International Joint Conference on Neural Networks (IJCNN), 2022.
  12. Spike: Spike-based embeddings for multi-relational graph data. In International Joint Conference on Neural Networks (IJCNN), 2021.
  13. Brain-inspired graph spiking neural networks for commonsense knowledge representation and reasoning. arXiv preprint arXiv:2207.05561, 2022.
  14. Incorporating learnable membrane time constant to enhance learning of spiking neural networks. In International Conference on Computer Vision (ICCV), 2021.
  15. node2vec: Scalable feature learning for networks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
  16. Learning delays in spiking neural networks using dilated convolutions with learnable spacings. In International Conference on Learning Representations (ICLR), 2024.
  17. Horowitz, M. 1.1 computing’s energy problem (and what we can do about it). In IEEE International Solid-State Circuits Conference (ISSCC), 2014.
  18. Independent rate and temporal coding in hippocampal pyramidal cells. Nature, 425(6960):828–832, 2003.
  19. Katz, L. A new status index derived from sociometric analysis. Psychometrika, 18(1):39–43, 1953.
  20. The discovery of structural form. Proceedings of the National Academy of Sciences (PNAS), 105(31):10687–10692, 2008.
  21. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016.
  22. Relational retrieval using a combination of path-constrained random walks. Machine Learning, 81:53–67, 2010.
  23. Scaling up dynamic graph representation learning via spiking neural networks. In AAAI Conference on Artificial Intelligence (AAAI), 2023.
  24. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. In International Conference on Machine Learning (ICML), 2021.
  25. The link-prediction problem for social networks. Journal of the American Society for Information Science and Technology, 58(7):1019–1031, 2007.
  26. Spiking convolutional neural networks for text classification. In International Conference on Learning Representations (ICLR), 2023.
  27. Maass, W. Networks of spiking neurons: the third generation of neural network models. Neural Networks, 10(9):1659–1671, 1997.
  28. Fine-grained evaluation of rule-and embedding-based systems for knowledge graph completion. In International Semantic Web Conference (ISWC), 2018.
  29. Training high-performance low-latency spiking neural networks by differentiation on spike representation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
  30. Training much deeper spiking neural networks with a small number of time-steps. Neural Networks, 153:254–268, 2022b.
  31. Mostafa, H. Supervised learning based on temporal coding in spiking neural networks. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 29(7):3227–3235, 2017.
  32. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11–33, 2015.
  33. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford InforLab, 1999.
  34. Empirical study on the efficiency of spiking neural networks with axonal delays, and algorithm-hardware benchmarking. In IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 2023.
  35. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 572(7767):106–111, 2019.
  36. Deepwalk: Online learning of social representations. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2014.
  37. A long short-term memory for AI applications in spike-based neuromorphic hardware. Nature Machine Intelligence, 4(5):467–479, 2022.
  38. Temporal coding of visual information in the thalamus. Journal of Neuroscience, 20(14):5392–5400, 2000.
  39. Towards spike-based machine intelligence with neuromorphic computing. Nature, 575(7784):607–617, 2019.
  40. Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Frontiers in Neuroscience, 11:682, 2017.
  41. Drum: End-to-end differentiable rule mining on knowledge graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
  42. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  43. Modeling relational data with graph convolutional networks. In European Semantic Web Conference (ESWC), 2018.
  44. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2(1):10–19, 2022.
  45. Collective classification in network data. AI Magazine, 29(3):93–93, 2008.
  46. Slayer: spike layer error reassignment in time. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
  47. Optimized spiking neurons can classify images with high accuracy through temporal coding with two spikes. Nature Machine Intelligence, 3(3):230–238, 2021.
  48. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations (ICLR), 2019.
  49. Line: Large-scale information network embedding. In International Conference on World Wide Web (WWW), 2015.
  50. Inductive relation prediction by subgraph reasoning. In International Conference on Machine Learning (ICML), 2020.
  51. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality, 2015.
  52. Complex embeddings for simple link prediction. In International Conference on Machine Learning (ICML), 2016.
  53. Composition-based multi-relational graph convolutional networks. In International Conference on Learning Representations (ICLR), 2020.
  54. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, 2023.
  55. Training feedback spiking neural networks by implicit differentiation on the equilibrium state. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  56. Online training through time for spiking neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  57. Hebbian learning based orthogonal projection for continual learning of spiking neural networks. In International Conference on Learning Representations (ICLR), 2024.
  58. Sampling complex topology structures for spiking neural networks. Neural Networks, 172:106121, 2024.
  59. Link prediction with persistent homology: An interactive view. In International Conference on Machine Learning (ICML), 2021.
  60. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations (ICLR), 2015.
  61. Differentiable learning of logical rules for knowledge base reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
  62. Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks. Nature Machine Intelligence, 3(10):905–913, 2021.
  63. Link prediction based on graph neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
  64. Efficient and generalizable cross-patient epileptic seizure detection through a spiking neural network. Frontiers in Neuroscience, 17:1303564, 2024.
  65. Dance of snn and ann: Solving binding problem by combining spike timing and reconstructive attention. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  66. Temporal-coded deep spiking neural network with easy training and robust performance. In AAAI Conference on Artificial Intelligence (AAAI), 2021.
  67. Neural bellman-ford networks: A general graph neural network framework for link prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  68. Spiking graph convolutional networks. In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets