Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Over-Squashing in Riemannian Graph Neural Networks (2311.15945v1)

Published 27 Nov 2023 in cs.LG and stat.ML

Abstract: Most graph neural networks (GNNs) are prone to the phenomenon of over-squashing in which node features become insensitive to information from distant nodes in the graph. Recent works have shown that the topology of the graph has the greatest impact on over-squashing, suggesting graph rewiring approaches as a suitable solution. In this work, we explore whether over-squashing can be mitigated through the embedding space of the GNN. In particular, we consider the generalization of Hyperbolic GNNs (HGNNs) to Riemannian manifolds of variable curvature in which the geometry of the embedding space is faithful to the graph's topology. We derive bounds on the sensitivity of the node features in these Riemannian GNNs as the number of layers increases, which yield promising theoretical and empirical results for alleviating over-squashing in graphs with negative curvature.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (34)
  1. “The Graph Neural Network Model” In IEEE Transactions on Neural Networks 20.1, 2008, pp. 61–80
  2. Will Hamilton, Zhitao Ying and Jure Leskovec “Inductive Representation Learning on Large Graphs” In Advances in Neural Information Processing Systems 30 Curran Associates, Inc., 2017 URL: https://proceedings.neurips.cc/paper_files/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html
  3. Thomas N. Kipf and Max Welling “Semi-Supervised Classification with Graph Convolutional Networks” In International Conference on Learning Representations, 2017 URL: https://openreview.net/forum?id=SJU4ayYgl
  4. “Neural Message Passing for Quantum Chemistry” In Proceedings of the 34th International Conference on Machine Learning 70, Proceedings of Machine Learning Research PMLR, 2017, pp. 1263–1272 URL: https://proceedings.mlr.press/v70/gilmer17a.html
  5. “On the Bottleneck of Graph Neural Networks and its Practical Implications” In International Conference on Learning Representations, 2021 URL: https://openreview.net/forum?id=i80OPhOCVH2
  6. “Understanding over-squashing and bottlenecks on graphs via curvature” In International Conference on Learning Representations, 2022 URL: https://openreview.net/forum?id=7UmjRGzp-A
  7. “Oversquashing in GNNs through the lens of information contraction and graph expansion” In 2022 58th Annual Allerton Conference on Communication, Control, and Computing (Allerton) Monticello, IL, USA: IEEE, 2022, pp. 1–8 URL: https://ieeexplore.ieee.org/document/9929363/
  8. “Understanding Oversquashing in GNNs through the Lens of Effective Resistance” In Proceedings of the 40th International Conference on Machine Learning, ICML’23 Honolulu, Hawaii, USA: JMLR.org, 2023
  9. Ralph Abboud, Radoslav Dimitrov and Ismail Ilkan Ceylan “Shortest Path Networks for Graph Property Prediction” In The First Learning on Graphs Conference, 2022 URL: https://openreview.net/forum?id=mWzWvMxuFg1
  10. “DiffWire: Inductive Graph Rewiring via the Lovász Bound” In The First Learning on Graphs Conference, 2022 URL: https://openreview.net/forum?id=IXvfIex0mX6f
  11. Andreea Deac, Marc Lackenby and Petar Veličković “Expander Graph Propagation” In The First Learning on Graphs Conference, 2022 URL: https://openreview.net/forum?id=IKevTLt3rT
  12. Kedar Karhadkar, Pradeep Kr. Banerjee and Guido Montufar “FoSR: First-order spectral rewiring for addressing oversquashing in GNNs” In The Eleventh International Conference on Learning Representations, 2023 URL: https://openreview.net/forum?id=3YjQfCLdrzz
  13. “On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topology” arXiv:2302.02941 [cs, stat] arXiv, 2023 URL: http://arxiv.org/abs/2302.02941
  14. Qi Liu, Maximilian Nickel and Douwe Kiela “Hyperbolic Graph Neural Networks” In Advances in Neural Information Processing Systems 32 Curran Associates, Inc., 2019 URL: https://proceedings.neurips.cc/paper_files/paper/2019/hash/103303dd56a731e377d01f6a37badae3-Abstract.html
  15. “Pseudo-Riemannian Graph Convolutional Networks” In Advances in Neural Information Processing Systems 35 Curran Associates, Inc., 2022, pp. 3488–3501 URL: https://proceedings.neurips.cc/paper_files/paper/2022/file/16c628ab12dc4caca8e7712affa6c767-Paper-Conference.pdf
  16. “Learning Mixed-Curvature Representations in Product Spaces” In International Conference on Learning Representations, 2019 URL: https://openreview.net/forum?id=HJxeWnCcF7
  17. Francesco Di Giovanni, Giulia Luise and Michael M. Bronstein “Heterogeneous manifolds for curvature-aware graph embedding” In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022 URL: https://openreview.net/forum?id=rtUxsN-kaxc
  18. Calin Cruceru, Gary Bécigneul and Octavian-Eugen Ganea “Computationally Tractable Riemannian Manifolds for Graph Embeddings” In AAAI Conference on Artificial Intelligence, 2020
  19. “Symmetric Spaces for Graph Embeddings: A Finsler-Riemannian Approach” ISSN: 2640-3498 In Proceedings of the 38th International Conference on Machine Learning PMLR, 2021, pp. 7090–7101 URL: https://proceedings.mlr.press/v139/lopez21a.html
  20. Octavian Ganea, Gary Becigneul and Thomas Hofmann “Hyperbolic Neural Networks” In Advances in Neural Information Processing Systems 31 Curran Associates, Inc., 2018 URL: https://proceedings.neurips.cc/paper_files/paper/2018/hash/dbab2adc8f9d078009ee3fa810bea142-Abstract.html
  21. Mitsuhiro Itoh “Some Geometrical Aspects of Riemannian Manifolds With a Pole” In Tsukuba Journal of Mathematics 4.2 Editorial Committee of Tsukuba Journal of Mathematics, 1980, pp. 291–301 URL: http://www.jstor.org/stable/43686608
  22. “Hyperbolic Graph Convolutional Neural Networks” In Advances in Neural Information Processing Systems 32 Curran Associates, Inc., 2019 URL: https://papers.nips.cc/paper_files/paper/2019/hash/0415740eaa4d9decbc8da001d3fd805f-Abstract.html
  23. “Collective Classification in Network Data” In AI Magazine 29.3, 2008, pp. 93 URL: https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/2157
  24. “Query-driven Active Surveying for Collective Classification” In Workshop on Mining and Learning with Graphs (MLG), 2012 URL: https://api.semanticscholar.org/CorpusID:15384610
  25. Aaron Lou, Maximilian Nickel and Brandon Amos “Deep Riemannian Manifold Learning” In Differential Geometry for Machine Learning Workshop at NeurIPS, 2020
  26. Peter Petersen “Riemannian Geometry” 171, Graduate Texts in Mathematics Cham: Springer International Publishing, 2016 DOI: 10.1007/978-3-319-26654-1
  27. Robin Forman “Bochner’s Method for Cell Complexes and Combinatorial Ricci Curvature” In Discrete and Computational Geometry 29.3 Springer ScienceBusiness Media LLC, 2003, pp. 323–374 DOI: 10.1007/s00454-002-0743-x
  28. Yann Ollivier “Ricci curvature of metric spaces” In Comptes Rendus Mathematique 345.11 Elsevier BV, 2007, pp. 643–646 DOI: 10.1016/j.crma.2007.10.041
  29. Yann Ollivier “Ricci curvature of Markov chains on metric spaces” In Journal of Functional Analysis 256, 2007, pp. 810–864
  30. “Spherical and Hyperbolic Embeddings of Data” In IEEE Transactions on Pattern Analysis and Machine Intelligence 36.11, 2014, pp. 2255–2269 DOI: 10.1109/TPAMI.2014.2316836
  31. “Poincaré Embeddings for Learning Hierarchical Representations” In Advances in Neural Information Processing Systems 30 Curran Associates, Inc., 2017 URL: https://proceedings.neurips.cc/paper_files/paper/2017/hash/59dfa2df42d9e3d41f5b02bfc32229dd-Abstract.html
  32. Benjamin Paul Chamberlain, James Clough and Marc Peter Deisenroth “Neural Embeddings of Graphs in Hyperbolic Space” In 13th international workshop on mining and learning from graphs held in conjunction with KDD, 2017
  33. “SphereFace: Deep Hypersphere Embedding for Face Recognition” In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu, HI: IEEE, 2017, pp. 6738–6746 DOI: 10.1109/CVPR.2017.713
  34. “Representation Tradeoffs for Hyperbolic Embeddings” ISSN: 2640-3498 In Proceedings of the 35th International Conference on Machine Learning PMLR, 2018, pp. 4460–4469 URL: https://proceedings.mlr.press/v80/sala18a.html

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now