Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Graph Neural Aggregation-diffusion with Metastability (2403.20221v1)

Published 29 Mar 2024 in cs.LG and cs.AI

Abstract: Continuous graph neural models based on differential equations have expanded the architecture of graph neural networks (GNNs). Due to the connection between graph diffusion and message passing, diffusion-based models have been widely studied. However, diffusion naturally drives the system towards an equilibrium state, leading to issues like over-smoothing. To this end, we propose GRADE inspired by graph aggregation-diffusion equations, which includes the delicate balance between nonlinear diffusion and aggregation induced by interaction potentials. The node representations obtained through aggregation-diffusion equations exhibit metastability, indicating that features can aggregate into multiple clusters. In addition, the dynamics within these clusters can persist for long time periods, offering the potential to alleviate over-smoothing effects. This nonlinear diffusion in our model generalizes existing diffusion-based models and establishes a connection with classical GNNs. We prove that GRADE achieves competitive performance across various benchmarks and alleviates the over-smoothing issue in GNNs evidenced by the enhanced Dirichlet energy.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. G. Kour and R. Saabne, “Real-time segmentation of on-line handwritten arabic script,” in Frontiers in Handwriting Recognition (ICFHR), 2014 14th International Conference on.   IEEE, 2014, pp. 417–422.
  2. G. Kour and R. Saabni, “Fast classification of handwritten on-line arabic characters,” in 6th International Conference of Soft Computing and Pattern Recognition, SoCPaR 2014, Tunis, Tunisia, August 11-14, 2014.   IEEE, 2014, pp. 312–318. [Online]. Available: https://doi.org/10.1109/SOCPAR.2014.7008025
  3. G. Hadash, E. Kermany, B. Carmeli, O. Lavi, G. Kour, and A. Jacovi, “Estimate and replace: A novel approach to integrating deep neural networks with existing applications,” arXiv preprint arXiv:1804.09028, 2018.
  4. R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” Advances in neural information processing systems, vol. 31, 2018.
  5. B. Chamberlain, J. Rowbottom, M. I. Gorinova, M. Bronstein, S. Webb, and E. Rossi, “Grand: Graph neural diffusion,” in International Conference on Machine Learning.   PMLR, 2021, pp. 1407–1418.
  6. L.-P. Xhonneux, M. Qu, and J. Tang, “Continuous graph neural networks,” in International Conference on Machine Learning.   PMLR, 2020, pp. 10 432–10 441.
  7. M. Thorpe, T. M. Nguyen, H. Xia, T. Strohmer, A. Bertozzi, S. Osher, and B. Wang, “Grand++: Graph neural diffusion with a source term,” in International Conference on Learning Representation, 2022.
  8. Y. Wang, K. Yi, X. Liu, Y. G. Wang, and S. Jin, “Acmp: Allen-cahn message passing for graph neural networks with particle phase transition,” in International Conference on Learning Representation, 2023.
  9. J. Choi, S. Hong, N. Park, and S.-B. Cho, “Gread: Graph neural reaction-diffusion networks,” in International Conference on Machine Learning.   PMLR, 2023, pp. 5722–5747.
  10. R. DeVore, B. Hanin, and G. Petrova, “Neural network approximation,” Acta Numerica, vol. 30, pp. 327–444, 2021.
  11. M. Belkin and P. Niyogi, “Laplacian eigenmaps for dimensionality reduction and data representation,” Neural computation, vol. 15, no. 6, pp. 1373–1396, 2003.
  12. X. Zhu, Z. Ghahramani, and J. D. Lafferty, “Semi-supervised learning using gaussian fields and harmonic functions,” in Proceedings of the 20th International conference on Machine learning (ICML-03), 2003, pp. 912–919.
  13. D. Zhou, O. Bousquet, T. Lal, J. Weston, and B. Schölkopf, “Learning with local and global consistency,” Advances in neural information processing systems, vol. 16, 2003.
  14. R. R. Coifman, S. Lafon, A. B. Lee, M. Maggioni, B. Nadler, F. Warner, and S. W. Zucker, “Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps,” Proceedings of the national academy of sciences, vol. 102, no. 21, pp. 7426–7431, 2005.
  15. B. Jing, S. Eismann, P. Suriana, R. J. L. Townshend, and R. Dror, “Learning from protein structure with geometric vector perceptrons,” in International Conference on Learning Representations, 2021.
  16. K. Atz, F. Grisoni, and G. Schneider, “Geometric deep learning on molecular representations,” Nature Machine Intelligence, vol. 3, no. 12, pp. 1023–1032, 2021.
  17. A. Merchant, S. Batzner, S. S. Schoenholz, M. Aykol, G. Cheon, and E. D. Cubuk, “Scaling deep learning for materials discovery,” Nature, pp. 1–6, 2023.
  18. Q. Li, Z. Han, and X. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” in Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018.   AAAI Press, 2018, pp. 3538–3545. [Online]. Available: https://doi.org/10.1609/aaai.v32i1.11604
  19. J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. J. Ragotte, L. F. Milles, B. I. Wicky, A. Courbet, R. J. de Haas, N. Bethel et al., “Robust deep learning–based protein sequence design using proteinmpnn,” Science, vol. 378, no. 6615, pp. 49–56, 2022.
  20. Q. Cappart, D. Chételat, E. B. Khalil, A. Lodi, C. Morris, and P. Veličković, “Combinatorial optimization and reasoning with graph neural networks,” Journal of Machine Learning Research, vol. 24, no. 130, pp. 1–61, 2023.
  21. M. J. Schuetz, J. K. Brubaker, and H. G. Katzgraber, “Combinatorial optimization with physics-inspired graph neural networks,” Nature Machine Intelligence, vol. 4, no. 4, pp. 367–377, 2022.
  22. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.
  23. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations, 2017.
  24. J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Neural message passing for quantum chemistry,” in International conference on machine learning.   PMLR, 2017, pp. 1263–1272.
  25. L. Ruthotto and E. Haber, “Deep neural networks motivated by partial differential equations,” Journal of Mathematical Imaging and Vision, vol. 62, pp. 352–364, 2020.
  26. L. Kong, J. Sun, and C. Zhang, “Sde-net: Equipping deep neural networks with uncertainty estimates,” in International conference on machine learning.   PMLR, 2020, pp. 5361–5371.
  27. E. Weinan, “A proposal on machine learning via dynamical systems,” Communications in Mathematics and Statistics, vol. 1, no. 5, pp. 1–11, 2017.
  28. Q. Li, T. Lin, and Z. Shen, “Deep learning via dynamical systems: An approximation perspective,” Journal of the European Mathematical Society, vol. 25, no. 5, pp. 1671–1709, 2022.
  29. Q. Wu, C. Yang, W. Zhao, Y. He, D. Wipf, and J. Yan, “Difformer: Scalable (graph) transformers induced by energy constrained diffusion,” in International Conference on Learning Representations, 2023.
  30. K. Oono and T. Suzuki, “Graph neural networks exponentially lose expressive power for node classification,” in International Conference on Learning Representations, 2020.
  31. K. Hornik, “Approximation capabilities of multilayer feedforward networks,” Neural networks, vol. 4, no. 2, pp. 251–257, 1991.
  32. J. A. Carrillo, K. Craig, and Y. Yao, “Aggregation-diffusion equations: dynamics, asymptotics, and singular limits,” Active Particles, Volume 2: Advances in Theory, Models, and Applications, pp. 65–108, 2019.
  33. C. Mascia and M. Strani, “Metastability for nonlinear parabolic equations with application to scalar viscous conservation laws,” SIAM Journal on Mathematical Analysis, vol. 45, no. 5, pp. 3084–3113, 2013.
  34. X. Li, T.-K. L. Wong, R. T. Chen, and D. Duvenaud, “Scalable gradients for stochastic differential equations,” in International Conference on Artificial Intelligence and Statistics.   PMLR, 2020, pp. 3870–3882.
  35. A. Ghosh, H. Behl, E. Dupont, P. Torr, and V. Namboodiri, “Steer: Simple temporal regularization for neural ode,” Advances in Neural Information Processing Systems, vol. 33, pp. 14 831–14 843, 2020.
  36. T. K. Rusch, M. M. Bronstein, and S. Mishra, “A survey on oversmoothing in graph neural networks,” arXiv preprint arXiv:2303.10993, 2023.
  37. T. K. Rusch, B. P. Chamberlain, M. W. Mahoney, M. M. Bronstein, and S. Mishra, “Gradient gating for deep multi-rate learning on graphs,” 2023. [Online]. Available: https://openreview.net/pdf?id=JpRExTbl1-
  38. Y. Lu, A. Zhong, Q. Li, and B. Dong, “Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations,” in International Conference on Machine Learning.   PMLR, 2018, pp. 3276–3285.
  39. T. K. Rusch, B. Chamberlain, J. Rowbottom, S. Mishra, and M. Bronstein, “Graph-coupled oscillator networks,” in International Conference on Machine Learning.   PMLR, 2022, pp. 18 888–18 909.
  40. A. K. McCallum, K. Nigam, J. Rennie, and K. Seymore, “Automating the construction of internet portals with machine learning,” Information Retrieval, vol. 3, no. 2, pp. 127–163, 2000.
  41. P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad, “Collective classification in network data,” AI Magazine, vol. 29, no. 3, pp. 93–93, 2008.
  42. G. Namata, B. London, L. Getoor, B. Huang, and U. EDU, “Query-driven active surveying for collective classification,” in Proceedings Mining and Learning with Graphs, 2012.
  43. J. McAuley, C. Targett, Q. Shi, and A. Van Den Hengel, “Image-based recommendations on styles and substitutes,” in Proceedings Information Retrieval, 2015.
  44. O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann, “Pitfalls of graph neural network evaluation,” arXiv:1811.05868, 2018.
  45. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” Advances in neural information processing systems, vol. 33, pp. 22 118–22 133, 2020.
  46. W. L. Hamilton, R. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NeurIPS, 2017.
  47. M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park, “Graph neural ordinary differential equations,” pp. 6571–6583, 2019.
  48. J. Zhuang, N. Dvornek, X. Li, and J. S. Duncan, “Ordinary differential equations on graph networks,” Tech. Rep. 1, 2020.
  49. R. Liao, Z. Zhao, R. Urtasun, and R. S. Zemel, “Lanczosnet: Multi-scale deep graph convolutional networks,” in ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.   OpenReview.net, 2019.
  50. F. Wu, T. Zhang, A. H. de Souza, C. Fifty, T. Yu, and K. Q. Weinberger, “Simplifying graph convolutional networks,” in ICML, 2019.
  51. S. Abu-El-Haija, B. Perozzi, A. Kapoor, N. Alipourfard, K. Lerman, H. Harutyunyan, G. Ver Steeg, and A. Galstyan, “Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing,” in ICML, 2019, pp. 21–29.
  52. H. Pei, B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang, “Geom-gcn: Geometric graph convolutional networks,” in ICLR, 2020.
  53. J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra, “Beyond homophily in graph neural networks: Current limitations and effective designs,” in NeurIPS, 2020.
  54. D. Bo, X. Wang, C. Shi, and H. Shen, “Beyond low-frequency information in graph convolutional networks,” in AAAI, 2021.
  55. E. Chien, J. Peng, P. Li, and O. Milenkovic, “Adaptive universal generalized pagerank graph neural network,” in ICLR, 2021.
  56. S. Suresh, V. Budde, J. Neville, P. Li, and J. Ma, “Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns,” in KDD, 2021, p. 1541–1551.
  57. M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li, “Simple and deep graph convolutional networks,” in ICML, 2020.
  58. B. P. Chamberlain, J. Rowbottom, D. Eynard, F. Di Giovanni, D. Xiaowen, and M. M. Bronstein, “Beltrami flow and neural diffusion on graphs,” in NeurIPS, 2021.
  59. C. Bodnar, F. D. Giovanni, B. P. Chamberlain, P. Lio, and M. M. Bronstein, “Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs,” in NeurIPS, 2022.
  60. F. Di Giovanni, J. Rowbottom, B. P. Chamberlain, T. Markovich, and M. M. Bronstein, “Graph neural networks as gradient flows,” arXiv preprint arXiv:2206.10991, 2022.
  61. Y. Yan, M. Hashemi, K. Swersky, Y. Yang, and D. Koutra, “Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks,” in ICDM, 2022.
  62. D. Lim, F. Hohne, X. Li, S. L. Huang, V. Gupta, O. Bhalerao, and S. N. Lim, “Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods,” in NeurIPS, vol. 34.   Curran Associates, Inc., 2021, pp. 20 887–20 902.
  63. X. Li, R. Zhu, Y. Cheng, C. Shan, S. Luo, D. Li, and W. Qian, “Finding global homophily in graph neural networks when meeting heterophily,” in ICML, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, Eds., vol. 162, 2022, pp. 13 242–13 256.
  64. S. Luan, C. Hua, Q. Lu, J. Zhu, M. Zhao, S. Zhang, X.-W. Chang, and D. Precup, “Revisiting heterophily for graph neural networks,” in NeurIPS, 2022.
  65. L. Zhao and L. Akoglu, “PairNorm: Tackling oversmoothing in gnns,” in ICLR, 2020.
  66. K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, and S. Jegelka, “Representation learning on graphs with jumping knowledge networks,” in ICML, 2018, pp. 5453–5462.
  67. T. K. Rusch, B. Chamberlain, J. Rowbottom, S. Mishra, and M. Bronstein, “Graph-coupled oscillator networks,” in ICML, vol. 162, 2022, pp. 18 888–18 909.
  68. B. Rozemberczki, C. Allen, and R. Sarkar, “Multi-Scale Attributed Node Embedding,” Journal of Complex Networks, vol. 9, no. 2, 2021.
  69. J. Tang, J. Sun, C. Wang, and Z. Yang, “Social influence analysis in large-scale networks,” in KDD, 2009, pp. 807–816.
  70. F. Monti, D. Boscaini, J. Masci, E. Rodolà, J. Svoboda, and M. M. Bronstein, “Geometric deep learning on graphs and manifolds using mixture model cnns,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5425–5434, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:301319
  71. O. Penrose and J. L. Lebowitz, “Rigorous treatment of metastable states in the van der waals-maxwell theory,” Journal of Statistical Physics, vol. 3, no. 2, pp. 211–236, 1971.
  72. S. Z. Cheng and A. Keller, “The role of metastable states in polymer phase transitions: Concepts, principles, and experimental observations,” Annual review of materials science, vol. 28, no. 1, pp. 533–562, 1998.
  73. Y. Deshpande, S. Sen, A. Montanari, and E. Mossel, “Contextual stochastic block models,” in NeurIPS, 2018.
  74. S. Motsch and E. Tadmor, “Heterophilious dynamics enhances consensus,” SIAM review, vol. 56, no. 4, pp. 577–621, 2014.
  75. F. K. O. Hoffmann, “Keller-segel-type models and kinetic equations for interacting particles: long-time asymptotic analysis,” Ph.D. dissertation, University of Cambridge, 2017.
  76. S. Fagioli and E. Radici, “Solutions to aggregation–diffusion equations with nonlinear mobility constructed via a deterministic particle approximation,” Mathematical Models and Methods in Applied Sciences, vol. 28, no. 09, pp. 1801–1829, 2018.
  77. G. Biroli and J. Kurchan, “Metastable states in glassy systems,” Physical Review E, vol. 64, no. 1, p. 016101, 2001.
  78. Y. Wang, Y. Wang, J. Yang, and Z. Lin, “Dissecting the diffusion process in linear graph convolutional networks,” Advances in Neural Information Processing Systems, vol. 34, pp. 5758–5769, 2021.
  79. D. Capocaccia, M. Cassandro, and E. Olivieri, “A study of metastability in the ising model,” 1974.
  80. K. Oono and T. Suzuki, “Graph neural networks exponentially lose expressive power for node classification,” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.   OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=S1ldO2EFPr
  81. M. Behmanesh, M. Krahn, and M. Ovsjanikov, “TIDE: time derivative diffusion for deep learning on graphs,” in International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, ser. Proceedings of Machine Learning Research, vol. 202.   PMLR, 2023, pp. 2015–2030.
  82. M. Eliasof, E. Haber, and E. Treister, “PDE-GCN: novel architectures for graph neural networks motivated by partial differential equations,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 2021, pp. 3836–3849.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Kaiyuan Cui (4 papers)
  2. Xinyan Wang (16 papers)
  3. Zicheng Zhang (124 papers)
  4. Weichen Zhao (7 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets