Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Understanding Oversmoothing in Diffusion-Based GNNs From the Perspective of Operator Semigroup Theory (2402.15326v1)

Published 23 Feb 2024 in cs.LG

Abstract: This paper presents a novel study of the oversmoothing issue in diffusion-based Graph Neural Networks (GNNs). Diverging from extant approaches grounded in random walk analysis or particle systems, we approach this problem through operator semigroup theory. This theoretical framework allows us to rigorously prove that oversmoothing is intrinsically linked to the ergodicity of the diffusion operator. This finding further poses a general and mild ergodicity-breaking condition, encompassing the various specific solutions previously offered, thereby presenting a more universal and theoretically grounded approach to mitigating oversmoothing in diffusion-based GNNs. Additionally, we offer a probabilistic interpretation of our theory, forging a link with prior works and broadening the theoretical horizon. Our experimental results reveal that this ergodicity-breaking term effectively mitigates oversmoothing measured by Dirichlet energy, and simultaneously enhances performance in node classification tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. TIDE: time derivative diffusion for deep learning on graphs. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp.  2015–2030. PMLR, 2023.
  2. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in gnns. Advances in Neural Information Processing Systems, 35:18527–18541, 2022.
  3. Beltrami flow and neural diffusion on graphs. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.  1594–1609, 2021a.
  4. Grand: Graph neural diffusion. In International Conference on Machine Learning, pp.  1407–1418. PMLR, 2021b.
  5. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp.  1725–1735. PMLR, 2020. URL http://proceedings.mlr.press/v119/chen20v.html.
  6. Optimization-induced graph implicit nonlinear diffusion. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp.  3648–3661. PMLR, 2022.
  7. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  8. Gread: Graph neural reaction-diffusion networks. In International Conference on Machine Learning, pp.  5722–5747. PMLR, 2023.
  9. Graph neural networks as gradient flows. arXiv preprint arXiv:2206.10991, 2022.
  10. PDE-GCN: novel architectures for graph neural networks motivated by partial differential equations. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp.  3836–3849, 2021.
  11. Freidlin, M. I. Functional integration and partial differential equations. Number 109. Princeton university press, 1985.
  12. p-laplacian based graph neural networks. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp.  6878–6917. PMLR, 2022. URL https://proceedings.mlr.press/v162/fu22e.html.
  13. A survey of graph neural networks for recommender systems: Challenges, methods, and directions. ACM Transactions on Recommender Systems, 1(1):1–51, 2023a.
  14. Hierarchical graph learning for protein–protein interaction. Nature Communications, 14(1):1093, 2023b.
  15. Markov processes and potential theory: Markov Processes and Potential Theory. Academic press, 2011.
  16. Neural message passing for quantum chemistry. In International conference on machine learning, pp.  1263–1272. PMLR, 2017.
  17. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
  18. Graph neural networks for efficient learning of mechanical properties of polycrystals. Computational Materials Science, 217:111894, 2023.
  19. Matrix analysis. Cambridge university press, 2012.
  20. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openreview.net/forum?id=SJU4ayYgl.
  21. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  22. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127–163, 2000.
  23. Scaling deep learning for materials discovery. Nature, pp.  1–6, 2023.
  24. Norris, J. R. Markov chains. Number 2. Cambridge university press, 1998.
  25. Oksendal, B. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
  26. Graph neural networks exponentially lose expressive power for node classification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=S1ldO2EFPr.
  27. Geom-gcn: Geometric graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=S1e2agrFvS.
  28. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2):cnab014, 2021.
  29. Graph-coupled oscillator networks. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp.  18888–18909. PMLR, 2022. URL https://proceedings.mlr.press/v162/rusch22a.html.
  30. Gradient gating for deep multi-rate learning on graphs. In The Eleventh International Conference on Learning Representations, 2023.
  31. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  32. On the robustness of graph neural diffusion to topology perturbations. Advances in Neural Information Processing Systems, 35:6384–6396, 2022.
  33. Fast and flexible protein design using deep graph neural networks. Cell systems, 11(4):402–411, 2020.
  34. Social influence analysis in large-scale networks. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.  807–816, 2009.
  35. Grand++: Graph neural diffusion with a source term. In International Conference on Learning Representation (ICLR), 2022.
  36. Graph attention networks. CoRR, abs/1710.10903, 2017. URL http://arxiv.org/abs/1710.10903.
  37. Acmp: Allen-cahn message passing with attractive and repulsive forces for graph neural networks. In The Eleventh International Conference on Learning Representations, 2023.
  38. Difformer: Scalable (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=j6zUzrapY3L.
  39. Continuous graph neural networks. In International Conference on Machine Learning, pp.  10432–10441. PMLR, 2020.
  40. Representation learning on graphs with jumping knowledge networks. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp.  5449–5458. PMLR, 2018. URL http://proceedings.mlr.press/v80/xu18c.html.
  41. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pp.  40–48. PMLR, 2016.
  42. Graph learning augmented heterogeneous graph neural network for social recommendation. ACM Transactions on Recommender Systems, 1(4):1–22, 2023a.
  43. Protein representation learning by geometric structure pretraining. In International Conference on Learning Representations, 2023b.
  44. Pairnorm: Tackling oversmoothing in gnns. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://openreview.net/forum?id=rkecl1rtwB.
  45. Analysis of graph neural networks with theory of markov chains. CoRR, abs/2211.06605, 2022. doi: 10.48550/ARXIV.2211.06605. URL https://doi.org/10.48550/arXiv.2211.06605.
  46. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in neural information processing systems, 33:7793–7804, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now