Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On the power of graph neural networks and the role of the activation function (2307.04661v6)

Published 10 Jul 2023 in cs.LG

Abstract: In this article we present new results about the expressivity of Graph Neural Networks (GNNs). We prove that for any GNN with piecewise polynomial activations, whose architecture size does not grow with the graph input sizes, there exists a pair of non-isomorphic rooted trees of depth two such that the GNN cannot distinguish their root vertex up to an arbitrary number of iterations. In contrast, it was already known that unbounded GNNs (those whose size is allowed to change with the graph sizes) with piecewise polynomial activations can distinguish these vertices in only two iterations. It was also known prior to our work that with ReLU (piecewise linear) activations, bounded GNNs are weaker than unbounded GNNs [ACI+22]. Our approach adds to this result by extending it to handle any piecewise polynomial activation function, which goes towards answering an open question formulated by [2021, Grohe] more completely. Our second result states that if one allows activations that are not piecewise polynomial, then in two iterations a single neuron perceptron can distinguish the root vertices of any pair of nonisomorphic trees of depth two (our results hold for activations like the sigmoid, hyperbolic tan and others). This shows how the power of graph neural networks can change drastically if one changes the activation function of the neural networks. The proof of this result utilizes the Lindemann-Weierstrauss theorem from transcendental number theory.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. Exponentially improving the complexity of simulating the weisfeiler-lehman test with graph neural networks. Advances in Neural Information Processing Systems, 35:27333–27346, 2022.
  2. Neural injective functions for multisets, measures and graphs via a finite witness theorem. Advances in Neural Information Processing Systems, 36, 2024.
  3. Expressive power of invariant and equivariant graph neural networks. In ICLR 2021-International Conference on Learning Representations, 2021.
  4. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18–42, 2017.
  5. The logical expressiveness of graph neural networks. In 8th International Conference on Learning Representations (ICLR 2020), 2020.
  6. On dimensionality of feature vectors in mpnns. arXiv preprint arXiv:2402.03966, 2024.
  7. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
  8. Combinatorial optimization and reasoning with graph neural networks. arXiv preprint arXiv:2102.09544, 2021.
  9. An optimal lower bound on the number of variables for graph identifications. Combinatorica, 12(4):389–410, 1992.
  10. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
  11. Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems, 28, 2015.
  12. Cognitive graph for multi-hop reading comprehension at scale. arXiv preprint arXiv:1905.05460, 2019.
  13. Expressiveness and approximation properties of graph neural networks. arXiv preprint arXiv:2204.04661, 2022.
  14. Martin Grohe. Descriptive complexity, canonisation, and definable graph structure theory, volume 47. Cambridge University Press, 2017.
  15. Martin Grohe. The logic of graph neural networks. In 2021 36th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pages 1–17. IEEE, 2021.
  16. William L Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1–159, 2020.
  17. A short tutorial on the weisfeiler-lehman test and its variants. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8533–8537. IEEE, 2021.
  18. Stefanie Jegelka. Theory of graph neural networks: Representation and learning. arXiv preprint arXiv:2204.07697, 2022.
  19. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
  20. Universal invariant and equivariant graph neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  21. On the universality of invariant networks. In International conference on machine learning, pages 4363–4371. PMLR, 2019.
  22. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4602–4609, 2019.
  23. Learning to simulate complex physics with graph networks. In International conference on machine learning, pages 8459–8468. PMLR, 2020.
  24. The graph neural network model. IEEE transactions on neural networks, 20(1):61–80, 2008.
  25. A deep learning approach to antibiotic discovery. Cell, 180(4):688–702, 2020.
  26. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.
  27. Modeling polypharmacy side effects with graph convolutional networks. Bioinformatics, 34(13):i457–i466, 2018.
  28. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now