Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Generalization of Graph Neural Networks through the Lens of Homomorphism (2403.06079v2)

Published 10 Mar 2024 in cs.LG

Abstract: Despite the celebrated popularity of Graph Neural Networks (GNNs) across numerous applications, the ability of GNNs to generalize remains less explored. In this work, we propose to study the generalization of GNNs through a novel perspective - analyzing the entropy of graph homomorphism. By linking graph homomorphism with information-theoretic measures, we derive generalization bounds for both graph and node classifications. These bounds are capable of capturing subtleties inherent in various graph structures, including but not limited to paths, cycles and cliques. This enables a data-dependent generalization analysis with robust theoretical guarantees. To shed light on the generality of of our proposed bounds, we present a unifying framework that can characterize a broad spectrum of GNN models through the lens of graph homomorphism. We validate the practical applicability of our theoretical findings by showing the alignment between the proposed bounds and the empirically observed generalization gaps over both real-world and synthetic datasets.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Stronger generalization bounds for deep nets via a compression approach. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp.  254–263, 2018.
  2. Graph neural networks with local graph parameters. Advances in Neural Information Processing Systems, 34:25280–25293, 2021.
  3. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463–482, 2002.
  4. Equivariant subgraph aggregation networks. In International Conference on Learning Representations, 2022.
  5. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM), 36(4):929–965, 1989.
  6. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Trans. Pattern Anal. Mach. Intell., 45(1):657–668, 2023.
  7. An optimal lower bound on the number of variables for graph identification. Combinatorica, 12(4):389–410, 1992.
  8. Fractal structure and generalization properties of stochastic optimization algorithms. In Advances in Neural Information Processing Systems, pp.  18774–18788, 2021.
  9. Loss landscapes are all you need: Neural network generalization can be explained without the implicit bias of gradient descent. In International Conference on Learning Representations, 2023.
  10. Measuring generalization with optimal transport. Advances in Neural Information Processing Systems, 2021.
  11. Some intersection theorems for ordered sets and graphs. J. Comb. Theory, Ser. A, 43(1):23–37, 1986.
  12. On provable benefits of depth in training graph convolutional networks. In Advances in Neural Information Processing Systems, pp.  9936–9949, 2021a.
  13. On provable benefits of depth in training graph convolutional networks. Advances in Neural Information Processing Systems, 34:9936–9949, 2021b.
  14. Fast neighborhood subgraph pairwise distance kernel. In International Conference on Machine Learning, pp.  255–262, 2010.
  15. Homomorphisms are a good basis for counting small subgraphs. In Symposium on Theory of Computing, pp.  210–223, 2017.
  16. Lovász meets weisfeiler and leman. In International Colloquium on Automata, Languages, and Programming, 2018.
  17. Generalization bounds using data-dependent fractal dimensions. In International Conference on Machine Learning, volume 202, pp.  8922–8968, 2023.
  18. Dvorák, Z. On recognizing graphs by numbers of homomorphisms. J. Graph Theory, 64(4):330–342, 2010.
  19. Benchmarking graph neural networks. J. Mach. Learn. Res., 24:43:1–43:48, 2023.
  20. Data-dependent pac-bayes priors via differential privacy. In Advances in Neural Information Processing Systems, pp.  8440–8450, 2018.
  21. Learning theory can (sometimes) explain generalisation in graph neural networks. In Annual Conference on Neural Information Processing Systems, pp.  27043–27056, 2021.
  22. Tighter expected generalization error bounds via wasserstein distance. In Advances in Neural Information Processing Systems, pp.  19109–19121, 2021.
  23. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning, volume 119, pp.  3419–3430, 2020.
  24. Cyclic pattern kernels for predictive graph mining. In International Conference on Knowledge Discovery and Data Mining, pp.  158–167, 2004.
  25. Generalization in graph neural networks: Improved pac-bayesian bounds on graph diffusion. In International Conference on Artificial Intelligence and Statistics, volume 206, pp.  6314–6341, 2023.
  26. Robustness implies generalization via data-dependent generalization bounds. In International Conference on Machine Learning, volume 162, pp.  10866–10894, 2022.
  27. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
  28. Generalization guarantee of training graph convolutional networks with graph topology sampling. In International Conference on Machine Learning, volume 162, pp.  13014–13051, 2022.
  29. A pac-bayesian approach to generalization bounds for graph neural networks. In International Conference on Learning Representations, 2021.
  30. Lovász, L. Large Networks and Graph Limits, volume 60 of Colloquium Publications. American Mathematical Society, 2012. ISBN 978-0-8218-9085-1.
  31. Provably powerful graph networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
  32. Generalization analysis of message passing neural networks on large random graphs. In Advances in Neural Information Processing Systems, 2022.
  33. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence, pp.  4602–4609, 2019a.
  34. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence, volume 33, pp.  4602–4609, 2019b.
  35. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020.
  36. WL meet VC. In International Conference on Machine Learning, volume 202, pp.  25275–25302, 2023.
  37. Graph homomorphism convolution. In International Conference on Machine Learning, volume 119, pp.  7306–7316, 2020.
  38. Optimization and generalization analysis of transduction through gradient boosting and application to multi-scale graph neural networks. Advances in Neural Information Processing Systems, 33, 2020.
  39. Pérez-Cruz, F. Kullback-leibler divergence estimation of continuous distributions. In 2008 IEEE International Symposium on Information Theory, ISIT, pp.  1666–1670, 2008.
  40. Lecture notes on information theory. Lecture Notes for ECE563 (UIUC) and, 6(2012-2016):7, 2014.
  41. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks, 108:248–259, 2018.
  42. Rate-distortion theoretic bounds on generalization error for distributed learning. In Advances in Neural Information Processing Systems, 2022.
  43. Collective classification in network data. AI Mag., 29(3):93–106, 2008.
  44. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
  45. $k$-variance: A clustered notion of variance. SIAM J. Math. Data Sci., 4(3):957–978, 2022.
  46. Towards understanding the generalization of graph neural networks. In International Conference on Machine Learning, 2023.
  47. Autobahn: Automorphism-based graph neural nets. In Annual Conference on Neural Information Processing Systems, pp.  29922–29934, 2021.
  48. Stability and generalization of graph convolutional neural networks. In International Conference on Knowledge Discovery & Data Mining, pp.  1539–1548, 2019.
  49. Villani, C. et al. Optimal transport: old and new, volume 338. Springer, 2009.
  50. N-wl: A new hierarchy of expressivity for graph neural networks. In International Conference on Learning Representations, 2023.
  51. The reduction of a graph to canonical form and the algebra which appears therein. Nauchno-Technicheskaya Informatsia, 2:12–16, 1968.
  52. Expectation-complete graph representations with homomorphisms. In International Conference on Machine Learning, 2023.
  53. A new perspective on ”how graph neural networks go beyond weisfeiler-lehman?”. In International Conference on Learning Representations, 2022.
  54. Handling distribution shifts on graphs: An invariance perspective. In International Conference on Learning Representations, 2022.
  55. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
  56. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33nd International Conference on Machine Learning, volume 48, pp.  40–48, 2016.
  57. A complete expressiveness hierarchy for subgraph GNNs via subgraph Weisfeiler-Lehman tests. In International Conference on Machine Learning, 2023.
  58. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
  59. Nested graph neural networks. In Advances in Neural Information Processing Systems, pp.  15734–15747, 2021.
  60. Fast learning of graph neural networks with guaranteed generalizability: one-hidden-layer case. In International Conference on Machine Learning, 2020.
  61. From stars to subgraphs: Uplifting any GNN with local structure awareness. In International Conference on Learning Representations, 2022.
  62. The generalization error of graph convolutional networks may enlarge with more layers. Neurocomputing, 424:97–106, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Shouheng Li (4 papers)
  2. Dongwoo Kim (63 papers)
  3. Qing Wang (341 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets