Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VC dimension of Graph Neural Networks with Pfaffian activation functions (2401.12362v2)

Published 22 Jan 2024 in stat.ML and cs.LG

Abstract: Graph Neural Networks (GNNs) have emerged in recent years as a powerful tool to learn tasks across a wide range of graph domains in a data-driven fashion; based on a message passing mechanism, GNNs have gained increasing popularity due to their intuitive formulation, closely linked with the Weisfeiler-Lehman (WL) test for graph isomorphism, to which they have proven equivalent. From a theoretical point of view, GNNs have been shown to be universal approximators, and their generalization capability (namely, bounds on the Vapnik Chervonekis (VC) dimension) has recently been investigated for GNNs with piecewise polynomial activation functions. The aim of our work is to extend this analysis on the VC dimension of GNNs to other commonly used activation functions, such as sigmoid and hyperbolic tangent, using the framework of Pfaffian function theory. Bounds are provided with respect to architecture parameters (depth, number of neurons, input size) as well as with respect to the number of colors resulting from the 1-WL test applied on the graph domain. The theoretical analysis is supported by a preliminary experimental study.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI Conference on Artificial Intelligence, pages 4602–4609, 2019.
  2. How powerful are graph neural networks? International Conference on Machine Learning, 2019.
  3. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks, 108:248–259, 2018.
  4. WL meet VC. Proceedings of the 40th International Conference on Machine Learning, 2023.
  5. Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 55(2):1–96, 2022.
  6. Deep learning-based reduced order models in cardiac electrophysiology. PloS one, 15(10):e0239416, 2020.
  7. Learning skillful medium-range global weather forecasting. Science, page eadi2336, 2023.
  8. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  9. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  10. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251–257, 1991.
  11. Barbara Hammer. On the approximation capability of recurrent neural networks. Neurocomputing, 31(1-4):107–123, 2000.
  12. Nonlinear approximation and (deep) relu networks. Constructive Approximation, 55(1):127–172, 2022.
  13. Generalizing outside the training set: When can neural networks learn identity effects? arXiv preprint arXiv:2005.04330, 2020.
  14. Invariance, encodings, and generalization: learning identity effects with neural networks. Neural Computation, 34(8):1756–1789, 2022.
  15. Generalization limits of graph neural networks in identity effects learning. arXiv preprint arXiv:2307.00134, 2023.
  16. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
  17. Towards understanding the role of over-parametrization in generalization of neural networks. arXiv preprint arXiv:1805.12076, 2018.
  18. Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. IEEE Transactions on Information Theory, 47(5):1902–1914, 2001.
  19. The probably approximately correct (PAC) and other learning models. The Mathematics of Generalization, pages 17–36, 2018.
  20. On the uniform convergence of relative frequencies of events to their probabilities. In Doklady Akademii Nauk USSR, volume 181, pages 781–787, 1968.
  21. Vladimir Vapnik. Estimation of dependences based on empirical data. Springer Science & Business Media, 2006.
  22. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009.
  23. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020.
  24. Introduction to graph neural networks. Springer Nature, 2022.
  25. B. Weisfeiler and A. Leman. The reduction of a graph to canonical form and the algebra which appears therein. Nauchno-Technicheskaya Informatsia, 2(9):12–16, 1968. English translation by G. Ryabov is available at https://www.iti.zcu.cz/wl2018/pdf/wl_paper_translation.pdf.
  26. Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networks. In International Conference on Machine Learning, pages 1026–1037. PMLR, 2021.
  27. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 34:2625–2640, 2021.
  28. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning, pages 3419–3430. PMLR, 2020.
  29. Universal covers, color refinement, and two-variable counting logic: Lower bounds for the depth. In 2015 30th Annual ACM/IEEE Symposium on Logic in Computer Science, pages 689–700. IEEE, 2015.
  30. On the approximation capability of GNNs in node classification/regression tasks. arXiv preprint arXiv:2106.08992, 2021.
  31. Learning theory can (sometimes) explain generalisation in graph neural networks. Advances in Neural Information Processing Systems, 34:27043–27056, 2021.
  32. Stability and generalization of graph convolutional neural networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1539–1548, 2019.
  33. A PAC-bayesian approach to generalization bounds for Graph Neural Networks. arXiv preprint arXiv:2012.07690, 2020.
  34. Generalization in Graph Neural Networks: Improved PAC-Bayesian bounds on graph diffusion. In International Conference on Artificial Intelligence and Statistics, pages 6314–6341. PMLR, 2023.
  35. Understanding attention and generalization in graph neural networks. Advances in neural information processing systems, 32, 2019.
  36. Measuring the vc-dimension of a learning machine. Neural computation, 6(5):851–876, 1994.
  37. Eduardo D Sontag et al. Vc dimension of neural networks. NATO ASI Series F Computer and Systems Sciences, 168:69–96, 1998.
  38. Vapnik-chervonenkis dimension of neural nets. The handbook of brain theory and neural networks, pages 1188–1192, 2003.
  39. Vapnik-chervonenkis dimension of recurrent neural networks. Discrete Applied Mathematics, 86(1):63–79, 1998.
  40. Askold G Khovanski. Fewnomials, volume 88. American Mathematical Soc., 1991.
  41. Polynomial bounds for VC dimension of sigmoidal and general pfaffian neural networks. J. Comput. Syst. Sci., 54(1):169–176, 1997.
  42. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE transactions on neural networks and learning systems, 25(8):1553–1565, 2014.
  43. Sandra Kiefer. The Weisfeiler-Leman algorithm: an exploration of its power. ACM SIGLOG News, 7(3):5–27, 2020.
  44. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, 30, 2017.
  45. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  46. VN Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability & Its Applications, 16(2):264–280, 1971.
  47. Barbara Hammer. On the generalization ability of recurrent networks. In Artificial Neural Networks—ICANN 2001: International Conference Vienna, Austria, August 21–25, 2001 Proceedings 11, pages 731–736. Springer, 2001.
  48. Neural networks with quadratic vc dimension. journal of computer and system sciences, 54(1):190–198, 1997.
  49. What size net gives valid generalization? Advances in neural information processing systems, 1, 1988.
  50. Wolfgang Maass. Neural nets with superlinear vc-dimension. Neural Computation, 6(5):877–884, 1994.
  51. Akito Sakurai. On the vc-dimension of depth four threshold circuits and the complexity of boolean-valued functions. Theoretical computer science, 137(1):109–127, 1995.
  52. Bounding the vapnik-chervonenkis dimension of concept classes parameterized by real numbers. In Proceedings of the sixth annual conference on Computational learning theory, pages 361–369, 1993.
  53. Complexity of computations with pfaffian and noetherian functions. Normal forms, bifurcations and finiteness problems in differential equations, 137:211–250, 2004.
  54. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020.
  55. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
  56. A survey on graph diffusion models: Generative ai in science for molecule, protein and material. arXiv preprint arXiv:2304.01565, 2023.
  57. Complexity of stratification of semi-pfaffian sets. Discrete & computational geometry, 14(1):71–91, 1995.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Giuseppe Alessio D'Inverno (11 papers)
  2. Monica Bianchini (14 papers)
  3. Franco Scarselli (15 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now