Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Graph Neural Networks Use Graphs When They Shouldn't (2309.04332v2)

Published 8 Sep 2023 in cs.LG and cs.AI

Abstract: Predictions over graphs play a crucial role in various domains, including social networks and medicine. Graph Neural Networks (GNNs) have emerged as the dominant approach for learning on graph data. Although a graph-structure is provided as input to the GNN, in some cases the best solution can be obtained by ignoring it. While GNNs have the ability to ignore the graph- structure in such cases, it is not clear that they will. In this work, we show that GNNs actually tend to overfit the given graph-structure. Namely, they use it even when a better solution can be obtained by ignoring it. We analyze the implicit bias of gradient-descent learning of GNNs and prove that when the ground truth function does not use the graphs, GNNs are not guaranteed to learn a solution that ignores the graph, even with infinite data. We examine this phenomenon with respect to different graph distributions and find that regular graphs are more robust to this over-fitting. We also prove that within the family of regular graphs, GNNs are guaranteed to extrapolate when learning with gradient descent. Finally, based on our empirical and theoretical findings, we demonstrate on real-data how regular graphs can be leveraged to reduce graph overfitting and enhance performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. On the bottleneck of graph neural networks and its practical implications, 2021.
  2. Emergence of scaling in random networks. Science, 286(5439):509–512, 1999. doi: 10.1126/science.286.5439.509. URL http://www.sciencemag.org/cgi/content/abstract/286/5439/509.
  3. How attentive are graph attention networks?, 2022.
  4. On graph neural networks versus graph-augmented mlps, 2020.
  5. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330 4:771–83, 2003.
  6. On random graphs i. Publicationes Mathematicae Debrecen, 6:290, 1959.
  7. A fair comparison of graph neural networks for graph classification, 2022.
  8. Neural message passing for quantum chemistry, 2017.
  9. Implicit bias of gradient descent on linear convolutional networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  10. Inductive representation learning on large graphs, 2017. URL https://arxiv.org/abs/1706.02216.
  11. Open graph benchmark: Datasets for machine learning on graphs, 2020. URL https://arxiv.org/abs/2005.00687.
  12. Semi-supervised classification with graph convolutional networks, 2017a.
  13. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017b. URL https://openreview.net/forum?id=SJU4ayYgl.
  14. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pp.  338–348, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10.1145/3394486.3403076. URL https://doi.org/10.1145/3394486.3403076.
  15. Gradient descent maximizes the margin of homogeneous neural networks, 2020.
  16. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020. URL www.graphlearning.io.
  17. Weisfeiler and leman go neural: Higher-order graph neural networks, 2021.
  18. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations, 2019. URL https://api.semanticscholar.org/CorpusID:212859361.
  19. Node feature kernels increase graph convolutional network robustness, 2022.
  20. Weisfeiler-lehman graph kernels. J. Mach. Learn. Res., 12:2539–2561, 2011. URL http://dblp.uni-trier.de/db/journals/jmlr/jmlr12.html#ShervashidzeSLMB11.
  21. Masked label prediction: Unified message passing model for semi-supervised classification, 2021.
  22. The implicit bias of gradient descent on separable data, 2017. URL https://arxiv.org/abs/1710.10345.
  23. Graph attention networks. In International Conference on Learning Representations, 2018.
  24. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ryGs6iA5Km.
  25. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’15, pp.  1365–1374, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336642. doi: 10.1145/2783258.2783417. URL https://doi.org/10.1145/2783258.2783417.
  26. On size generalization in graph neural networks. CoRR, abs/2010.08853, 2020. URL https://arxiv.org/abs/2010.08853.
  27. Deep sets, 2018.
  28. Understanding deep learning requires rethinking generalization, 2017.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now