Papers
Topics
Authors
Recent
Search
2000 character limit reached

Geometric instability of graph neural networks on large graphs

Published 19 Aug 2023 in cs.LG and cs.SI | (2308.10099v2)

Abstract: We analyse the geometric instability of embeddings produced by graph neural networks (GNNs). Existing methods are only applicable for small graphs and lack context in the graph domain. We propose a simple, efficient and graph-native Graph Gram Index (GGI) to measure such instability which is invariant to permutation, orthogonal transformation, translation and order of evaluation. This allows us to study the varying instability behaviour of GNN embeddings on large graphs for both node classification and link prediction.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. William L. Hamilton. Graph representation learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 14(3):1–159.
  2. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, pages 701–710, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450329569. doi: 10.1145/2623330.2623732. URL https://doi.org/10.1145/2623330.2623732.
  3. node2vec: Scalable feature learning for networks. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 855–864. ACM, 2016. doi: 10.1145/2939672.2939754. URL https://doi.org/10.1145/2939672.2939754.
  4. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009. doi: 10.1109/TNN.2008.2005605.
  5. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM), pages 333–341. SIAM, 2021.
  6. The surprising power of graph neural networks with random node initialization. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2112–2118. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ijcai.2021/291. URL https://doi.org/10.24963/ijcai.2021/291. Main Track.
  7. Pitfalls of graph neural network evaluation, 2019.
  8. Are word embedding methods stable and should we care about it? Proceedings of the 32nd ACM Conference on Hypertext and Social Media, 2021. URL https://api.semanticscholar.org/CorpusID:233296559.
  9. Alignment and stability of embeddings: measurement and inference improvement. Neurocomputing, 553:126517, 2023.
  10. Evaluating the stability of embedding-based word similarities. Transactions of the Association for Computational Linguistics, 6:107–119, 2018. doi: 10.1162/tacl_a_00008. URL https://aclanthology.org/Q18-1008.
  11. David Picard. Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. CoRR, abs/2109.08203, 2021. URL https://arxiv.org/abs/2109.08203.
  12. On the prediction instability of graph neural networks, 2022.
  13. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR, 2017.
  14. Graph Attention Networks. In ICLR, 2018.
  15. The effects of randomness on the stability of node embeddings. CoRR, abs/2005.10039, 2020. URL https://arxiv.org/abs/2005.10039.
  16. Inductive Representation Learning on Large Graphs. In NIPS, pages 1024–1034, 2017.
  17. On the surprising behaviour of node2vec, 2022.
  18. Twhin: Embedding the twitter heterogeneous information network for personalized recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, pages 2842–2850, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393850. doi: 10.1145/3534678.3539080. URL https://doi.org/10.1145/3534678.3539080.
  19. Cultural shift or linguistic drift? comparing two computational measures of semantic change. In Proceedings of the conference on empirical methods in natural language processing. Conference on empirical methods in natural language processing, volume 2016, page 2116. NIH Public Access, 2016.
  20. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019.
  21. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836, 2018.
  22. Garmt B Dijksterhuis John C Gower. Procrustes Problems. Oxford Academic, 2007.
  23. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519–3529. PMLR, 2019.
  24. On the universality of invariant networks, 2019.
  25. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021.
  26. Second order properties of error surfaces: Learning time and generalization. In Proceedings of the 3rd International Conference on Neural Information Processing Systems, NIPS’90, pages 918–924, San Francisco, CA, USA, 1990. Morgan Kaufmann Publishers Inc. ISBN 1558601848.
  27. On the geometry of feedforward neural network error surfaces. Neural Comput., 5(6):910–927, nov 1993. ISSN 0899-7667. doi: 10.1162/neco.1993.5.6.910. URL https://doi.org/10.1162/neco.1993.5.6.910.
  28. E(n) equivariant graph neural networks, 2022.
  29. Reliability of cka as a similarity measure in deep learning. arXiv preprint arXiv:2210.16156, 2022.
  30. How powerful are graph neural networks?, 2019.
  31. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020.
  32. Edge directionality improves learning on heterophilic graphs, 2023.
  33. A critical look at the evaluation of gnns under heterophily: are we really making progress?, 2023.
  34. GraphSAINT: Graph sampling based inductive learning method. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=BJe8pkHFwS.
  35. Matching shapes with a reference point. Int. J. Comput. Geometry Appl., 7:349–363, 01 1997.
  36. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6076–6085. Curran Associates, Inc., 2017.
  37. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  38. Revisiting semi-supervised learning with graph embeddings, 2016.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up