Papers
Topics
Authors
Recent
Search
2000 character limit reached

Non-convolutional Graph Neural Networks

Published 31 Jul 2024 in cs.LG and cs.AI | (2408.00165v3)

Abstract: Rethink convolution-based graph neural networks (GNN) -- they characteristically suffer from limited expressiveness, over-smoothing, and over-squashing, and require specialized sparse kernels for efficient computation. Here, we design a simple graph learning module entirely free of convolution operators, coined random walk with unifying memory (RUM) neural network, where an RNN merges the topological and semantic graph features along the random walks terminating at each node. Relating the rich literature on RNN behavior and graph topology, we theoretically show and experimentally verify that RUM attenuates the aforementioned symptoms and is more expressive than the Weisfeiler-Lehman (WL) isomorphism test. On a variety of node- and graph-level classification and regression tasks, RUM not only achieves competitive performance, but is also robust, memory-efficient, scalable, and faster than the simplest convolutional GNNs.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Semi-supervised classification with graph convolutional networks. CoRR, abs/1609.02907, 2016. URL http://arxiv.org/abs/1609.02907.
  2. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018a.
  3. Neural message passing for quantum chemistry. arXiv preprint arXiv:1704.01212, 2017.
  4. Inductive representation learning on large graphs. In Advances in neural information processing systems, pages 1024–1034, 2017.
  5. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
  6. Graph attention networks, 2018.
  7. Simplifying graph convolutional networks. CoRR, abs/1902.07153, 2019. URL http://arxiv.org/abs/1902.07153.
  8. GRAND: graph neural diffusion. CoRR, abs/2106.10934, 2021. URL https://arxiv.org/abs/2106.10934.
  9. The reduction of a graph to canonical form and the algebra which appears therein.
  10. Principal neighbourhood aggregation for graph nets, 2020.
  11. Generalization and representational limits of graph neural networks. CoRR, abs/2002.06157, 2020a. URL https://arxiv.org/abs/2002.06157.
  12. A note on over-smoothing for graph neural networks, 2020.
  13. A survey on oversmoothing in graph neural networks, 2023.
  14. On the bottleneck of graph neural networks and its practical implications. CoRR, abs/2006.05205, 2020. URL https://arxiv.org/abs/2006.05205.
  15. Understanding over-squashing and bottlenecks on graphs via curvature, 2022.
  16. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR, abs/1406.1078, 2014. URL http://arxiv.org/abs/1406.1078.
  17. Raw-gnn: Random walk aggregation based graph neural network, 2022.
  18. Graph learning with 1d convolutions on random walks. CoRR, abs/2102.08786, 2021. URL https://arxiv.org/abs/2102.08786.
  19. Walking out of the weisfeiler leman hierarchy: Graph learning beyond message passing, 2023.
  20. Anonymous walk embeddings, 2018.
  21. Inductive representation learning in temporal networks via causal anonymous walks, 2022a.
  22. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning, 2018.
  23. Agent-based graph neural networks, 2023.
  24. Random walk graph neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16211–16222. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ba95d78a7c942571185308775a97a3a0-Paper.pdf.
  25. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):657–668, January 2023. ISSN 1939-3539. doi: 10.1109/tpami.2022.3154319. URL http://dx.doi.org/10.1109/TPAMI.2022.3154319.
  26. The truly deep graph convolutional networks for node classification. CoRR, abs/1907.10903, 2019. URL http://arxiv.org/abs/1907.10903.
  27. Predict then propagate: Graph neural networks meet personalized pagerank, 2022a.
  28. Diffusion improves graph learning, 2022b.
  29. Joint adaptive feature smoothing and topology extraction via generalized pagerank gnns. CoRR, abs/2006.07988, 2020. URL https://arxiv.org/abs/2006.07988.
  30. The pagerank citation ranking : Bringing order to the web. In The Web Conference, 1999. URL https://api.semanticscholar.org/CorpusID:1508503.
  31. Representation learning on graphs with jumping knowledge networks. CoRR, abs/1806.03536, 2018b. URL http://arxiv.org/abs/1806.03536.
  32. Pairnorm: Tackling oversmoothing in gnns. CoRR, abs/1909.12223, 2019. URL http://arxiv.org/abs/1909.12223.
  33. Graph-coupled oscillator networks. CoRR, abs/2202.02296, 2022. URL https://arxiv.org/abs/2202.02296.
  34. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385.
  35. Simple and deep graph convolutional networks, 2020.
  36. Deep graph library: A graph-centric, highly-performant package for graph neural networks, 2020.
  37. Reconstructing markov processes from independent and anonymous experiments. Discrete Applied Mathematics, 200:108–122, 2016. ISSN 0166-218X. doi: https://doi.org/10.1016/j.dam.2015.06.035. URL https://www.sciencedirect.com/science/article/pii/S0166218X15003212.
  38. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http://arxiv.org/abs/1706.03762.
  39. A comprehensive study on large-scale graph training: Benchmarking and rethinking, 2023.
  40. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  41. Recurrent neural networks are universal approximators. In Artificial Neural Networks–ICANN 2006: 16th International Conference, Athens, Greece, September 10-14, 2006. Proceedings, Part I 16, pages 632–640. Springer, 2006.
  42. Paul J Kelly. A congruence theorem for trees. 1957.
  43. Andreas Loukas. What graph neural networks cannot learn: depth vs width. CoRR, abs/1907.03199, 2019. URL http://arxiv.org/abs/1907.03199.
  44. Generalization and representational limits of graph neural networks, 2020b.
  45. Revisiting semi-supervised learning with graph embeddings. CoRR, abs/1603.08861, 2016. URL http://arxiv.org/abs/1603.08861.
  46. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994.
  47. Beyond exploding and vanishing gradients: analysing rnn training using attractors and smoothness. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2370–2380. PMLR, 26–28 Aug 2020. URL https://proceedings.mlr.press/v108/ribeiro20a.html.
  48. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.
  49. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43–52, 2015.
  50. Geom-gcn: Geometric graph convolutional networks, 2020.
  51. Tudataset: A collection of benchmark datasets for learning with graphs, 2020.
  52. Moleculenet: a benchmark for molecular machine learning. Chem. Sci., 9:513–530, 2018. doi: 10.1039/C7SC02664A. URL http://dx.doi.org/10.1039/C7SC02664A.
  53. Open graph benchmark: Datasets for machine learning on graphs. CoRR, abs/2005.00687, 2020. URL https://arxiv.org/abs/2005.00687.
  54. Gauche: A library for gaussian processes in chemistry, 2023.
  55. Geometric deep learning on graphs and manifolds using mixture model cnns. CoRR, abs/1611.08402, 2016. URL http://arxiv.org/abs/1611.08402.
  56. Efficient graphlet kernels for large graph comparison. In Artificial intelligence and statistics, pages 488–495. PMLR, 2009.
  57. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
  58. Relational pooling for graph representations, 2019.
  59. Benchmarking graph neural networks, 2022.
  60. Provably powerful graph networks, 2020.
  61. Stochastic aggregation in graph neural networks. arXiv preprint arXiv:2102.12648, 2021.
  62. Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912.01703, 2019. URL http://arxiv.org/abs/1912.01703.
  63. Graph random neural networks for semi-supervised learning on graphs. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22092–22103. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/fb4c835feb0a65cc39739320d7a51c02-Paper.pdf.
  64. A survey on graph kernels. CoRR, abs/1903.11835, 2019. URL http://arxiv.org/abs/1903.11835.
  65. End-to-end differentiable construction of molecular mechanics force fields. Chemical Science, 13(41):12016–12033, 2022b.
  66. Espaloma-0.3. 0: Machine-learned molecular mechanics force field for the simulation of protein-ligand systems and beyond. arXiv preprint arXiv:2307.07085, 2023.
  67. Druggym: A testbed for the economics of autonomous drug discovery. bioRxiv, 2024. doi: 10.1101/2024.05.28.596296. URL https://www.biorxiv.org/content/early/2024/06/02/2024.05.28.596296.
  68. Espalomacharge: Machine learning-enabled ultrafast partial charge assignment. The Journal of Physical Chemistry A, 128(20):4160–4167, 2024.
  69. Yuanqing Wang. Graph Machine Learning for (Bio)Molecular Modeling and Force Field Construction. PhD thesis, 2023. URL http://proxy.library.nyu.edu/login?qurl=https%3A%2F%2Fwww.proquest.com%2Fdissertations-theses%2Fgraph-machine-learning-bio-molecular-modeling%2Fdocview%2F2789704784%2Fse-2%3Faccountid%3D12768. Copyright - Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated - 2024-04-24.
  70. Spatial attention kinetic networks with e (n)-equivariance. In ICLR 2023, 2023.
  71. Machine learning for social science: An agnostic approach. Annual Review of Political Science, 24:395–419, 2021.
  72. Artificial intelligence in drug design. Molecules, 23(10):2520, 2018.
  73. Adam: A method for stochastic optimization, 2017.
  74. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017.
  75. Ray: A distributed framework for emerging ai applications, 2018.
  76. Ae: A domain-agnostic platform for adaptive experimentation. In Conference on neural information processing systems, pages 1–8, 2018.
  77. On the difficulty of training recurrent neural networks, 2013.
  78. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  79. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. CoRR, abs/1905.07953, 2019. URL http://arxiv.org/abs/1905.07953.
  80. Graphsaint: Graph sampling based inductive learning method. CoRR, abs/1907.04931, 2019. URL http://arxiv.org/abs/1907.04931.
  81. Fastgcn: Fast learning with graph convolutional networks via importance sampling. CoRR, abs/1801.10247, 2018. URL http://arxiv.org/abs/1801.10247.
  82. Layer-dependent importance sampling for training deep and large graph convolutional networks. CoRR, abs/1911.07323, 2019. URL http://arxiv.org/abs/1911.07323.
  83. Scalable and adaptive graph neural networks with self-label-enhanced training, 2021.
  84. Lanczosnet: Multi-scale deep graph convolutional networks. ArXiv, abs/1901.01484, 2019. URL https://api.semanticscholar.org/CorpusID:57573752.
  85. A piece-wise polynomial filtering approach for graph neural networks, 2021.
  86. Do transformers really perform bad for graph representation? CoRR, abs/2106.05234, 2021. URL https://arxiv.org/abs/2106.05234.
  87. Specformer: Spectral graph neural networks meet transformers, 2023.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 11 likes about this paper.

Sign Up for Free