Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
133 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GNN-VPA: A Variance-Preserving Aggregation Strategy for Graph Neural Networks (2403.04747v1)

Published 7 Mar 2024 in cs.LG, cs.AI, and stat.ML

Abstract: Graph neural networks (GNNs), and especially message-passing neural networks, excel in various domains such as physics, drug discovery, and molecular modeling. The expressivity of GNNs with respect to their ability to discriminate non-isomorphic graphs critically depends on the functions employed for message aggregation and graph-level readout. By applying signal propagation theory, we propose a variance-preserving aggregation function (VPA) that maintains expressivity, but yields improved forward and backward dynamics. Experiments demonstrate that VPA leads to increased predictive performance for popular GNN architectures as well as improved learning dynamics. Our results could pave the way towards normalizer-free or self-normalizing GNNs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Layer Normalization. arXiv preprint arXiv:1607.06450, 2016.
  2. Message Passing Neural PDE Solvers. In International Conference on Learning Representations, 2022.
  3. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv preprint arXiv:2104.13478, 2021.
  4. Combinatorial Optimization and Reasoning with Graph Neural Networks. Journal of Machine Learning Research, 24(130):1–61, 2023.
  5. Distribution Knowledge Embedding for Graph Pooling. IEEE Transactions on Knowledge and Data Engineering, 35(8):7898–7908, 2023.
  6. Principal Neighbourhood Aggregation for Graph Nets. In Advances in Neural Information Processing Systems, 2020.
  7. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In Advances in Neural Information Processing Systems, 2016.
  8. Convolutional Networks on Graphs for Learning Molecular Fingerprints. In Advances in Neural Information Processing Systems, 2015.
  9. Genetic Neural Networks: An Artificial Neural Network Architecture for Capturing Gene Expression Relationships. Bioinformatics, 35(13):2226–2234, 11 2018.
  10. Graph Neural Networks for Social Recommendation. In The World Wide Web Conference, 2019.
  11. Fast Graph Representation Learning with PyTorch Geometric. arXiv preprint arXiv:1903.02428, 2019.
  12. Neural Message Passing for Quantum Chemistry. In International Conference on Machine Learning, 2017.
  13. Understanding the Difficulty of Training Deep Feedforward Neural Networks. In International Conference on Artificial Intelligence and Statistics, 2010.
  14. Inductive Representation Learning on Large Graphs. In Advances in Neural Information Processing Systems, 2017.
  15. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In IEEE International Conference on Computer Vision (ICCV), 2015.
  16. Principled Weight Initialisation for Input-Convex Neural Networks. In Advances in Neural Information Processing Systems, 2023.
  17. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In International Conference on Machine Learning, 2015.
  18. Molecular Graph Convolutions: Moving Beyond Fingerprints. Journal of Computer-Aided Molecular Design, 30(8):595–608, 2016.
  19. Keisler, R. Forecasting Global Weather with Graph Neural Networks. arXiv preprint arXiv:2202.07575, 2022.
  20. Semi-Supervised Classification with Graph Convolutional Networks. In International Conference on Learning Representations, 2017.
  21. Self-Normalizing Neural Networks. In Advances in Neural Information Processing Systems, 2017.
  22. GraphCast: Learning Skillful Medium-Range Global Weather Forecasting. arXiv preprint arXiv:2212.12794, 2022.
  23. Efficient BackProp. In Montavon, G., Orr, G. B., and Müller, K.-R. (eds.), Neural Networks: Tricks of the Trade: Second Edition, pp.  9–48. Springer Berlin Heidelberg, 2012.
  24. Deep Neural Networks as Gaussian Processes. In International Conference on Learning Representations, 2018.
  25. Self-Attention Graph Pooling. In International Conference on Machine Learning, 2019.
  26. A reduction of a graph to a canonical form and an algebra arising during this reduction. Nauchno-Technicheskaya Informatsiya, 2(9):12–16, 1968.
  27. Learning Knowledge Graph Embedding With Heterogeneous Relation Attention Networks. IEEE Transactions on Neural Networks and Learning Systems, 33(8):3961–3973, 2022.
  28. Bidirectionally Self-Normalizing Neural Networks. Neural Networks, 167:283–291, 2023.
  29. Rapid Training of Deep Neural Networks without Skip Connections or Normalization Layers using Deep Kernel Shaping. arXiv preprint arXiv:2110.01765, 2021.
  30. Large-Scale Comparison of Machine Learning Methods for Drug Target Prediction on ChEMBL. Chemical Science, 9:5441–5451, 2018.
  31. Boundary Graph Neural Networks for 3D Simulations. In AAAI Conference on Artificial Intelligence, 2023.
  32. Scaling Deep Learning for Materials Discovery. Nature, 624(7990):80–85, 2023.
  33. Fake News Detection on Social Media using Geometric Deep Learning. In International Conference on Learning Representations, 2019.
  34. Weisfeiler and Leman Go Neural: Higher-Order Graph Neural Networks. In AAAI Conference on Artificial Intelligence and Innovative Applications of Artificial Intelligence Conference and AAAI Symposium on Educational Advances in Artificial Intelligence, 2019.
  35. TUDataset: A Collection of Benchmark Datasets for Learning with Graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond, 2020.
  36. Neal, R. M. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
  37. Graph Neural Networks for Materials Science and Chemistry. Communications Materials, 3(1):93, 2022.
  38. Learning to Simulate Complex Physics with Graph Networks. In International Conference on Machine Learning, 2020.
  39. Variational Annealing on Graphs for Combinatorial Optimization. In Advances in Neural Information Processing Systems, 2023.
  40. E(n) Equivariant Graph Neural Networks. In International Conference on Machine Learning, 2021.
  41. The Graph Neural Network Model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009.
  42. Modeling Relational Data with Graph Convolutional Networks. In The Semantic Web, 2018.
  43. Deep Information Propagation. In International Conference on Learning Representations, 2017.
  44. Graph Attention Networks. In International Conference on Learning Representations, 2018.
  45. Inductive Inference of Gene Regulatory Network Using Supervised and Semi-Supervised Graph Neural Networks. Computational and Structural Biotechnology Journal, 18:3335–3343, 2020.
  46. Simplifying Graph Convolutional Networks. In International Conference on Machine Learning, 2019.
  47. A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, 2021.
  48. How Powerful are Graph Neural Networks? In International Conference on Learning Representations, 2019.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets