Papers
Topics
Authors
Recent
Search
2000 character limit reached

DiGRAF: Diffeomorphic Graph-Adaptive Activation Function

Published 2 Jul 2024 in cs.LG | (2407.02013v2)

Abstract: In this paper, we propose a novel activation function tailored specifically for graph data in Graph Neural Networks (GNNs). Motivated by the need for graph-adaptive and flexible activation functions, we introduce DiGRAF, leveraging Continuous Piecewise-Affine Based (CPAB) transformations, which we augment with an additional GNN to learn a graph-adaptive diffeomorphic activation function in an end-to-end manner. In addition to its graph-adaptivity and flexibility, DiGRAF also possesses properties that are widely recognized as desirable for activation functions, such as differentiability, boundness within the domain, and computational efficiency. We conduct an extensive set of experiments across diverse datasets and tasks, demonstrating a consistent and superior performance of DiGRAF compared to traditional and graph-specific activation functions, highlighting its effectiveness as an activation function for GNNs. Our code is available at https://github.com/ipsitmantri/DiGRAF.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (90)
  1. Construction of bayesian deformable models via a stochastic approximation algorithm: a convergence study. Bernoulli, 2010.
  2. Bayesian mixed effect atlas estimation with a diffeomorphic deformation model. SIAM Journal on Imaging Sciences, 8(3):1367–1395, 2015.
  3. A survey on modern trainable activation functions. Neural Networks, 138:14–32, 2021.
  4. Efficient subgraph gnns by learning effective selection policies. arXiv preprint arXiv:2310.20082, 2023.
  5. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning, pages 874–883. PMLR, 2020.
  6. Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com.
  7. How attentive are graph attention networks? In International Conference on Learning Representations, 2022.
  8. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
  9. Simple and deep graph convolutional networks. In International conference on machine learning, pages 1725–1735. PMLR, 2020.
  10. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
  11. Fast and accurate deep network learning by exponential linear units (elus), 2016.
  12. Nonlinear approximation and (deep) relu networks. Constructive Approximation, 55(1):127–172, 2022.
  13. On the approximation of functions by tanh neural networks. Neural Networks, 143:732–750, 2021.
  14. Deep diffeomorphic transformer networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4403–4412, 2018.
  15. Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503:92–108, 2022.
  16. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Representations, 2022.
  17. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1–48, 2023.
  18. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3–11, 2018.
  19. Graph positional encoding via random feature propagation. In International Conference on Machine Learning, pages 9202–9223. PMLR, 2023.
  20. Fast graph representation learning with pytorch geometric, 2019.
  21. Understanding and extending subgraph gnns by rethinking their symmetries. In Advances in Neural Information Processing Systems, 2022.
  22. Highly-expressive spaces of well-behaved transformations: Keeping it simple. In Proceedings of the IEEE International Conference on Computer Vision, pages 2911–2919, 2015.
  23. Transformations based on continuous piecewise-affine velocity fields. IEEE transactions on pattern analysis and machine intelligence, 39(12):2496–2509, 2017.
  24. Kunihiko Fukushima. Visual feature extraction by a multilayered network of analog threshold elements. IEEE Transactions on Systems Science and Cybernetics, 5(4):322–333, 1969.
  25. Graph u-nets. In international conference on machine learning, pages 2083–2092. PMLR, 2019.
  26. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  27. Maxout networks. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1319–1327, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
  28. Thomas Hakon Gronwall. Note on the derivatives with respect to a parameter of the solutions of a system of differential equations. Annals of Mathematics, pages 292–296, 1919.
  29. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4(2):268–276, January 2018. ISSN 2374-7951. doi: 10.1021/acscentsci.7b00572.
  30. Stable architectures for deep neural networks. Inverse problems, 34(1):014004, 2017.
  31. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  32. Gaussian error linear units (gelus), 2023.
  33. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  34. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020a.
  35. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020b.
  36. Graph-adaptive activation functions for graph neural networks. In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2020.
  37. Non-convex optimization for machine learning. Foundations and Trends® in Machine Learning, 10(3-4):142–363, 2017.
  38. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  39. On the power of graph neural networks and the role of the activation function, 2023.
  40. How to find your friendly neighborhood: Graph attention design with self-supervision. In International Conference on Learning Representations, 2021.
  41. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
  42. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:21618–21629, 2021.
  43. Three decades of activations: A comprehensive survey of 400 activation functions for neural networks. arXiv preprint arXiv:2402.09092, 2024.
  44. Self-attention graph pooling. In International conference on machine learning, pages 3734–3743. PMLR, 2019.
  45. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  46. Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing, 216:718–734, 2016. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2016.08.037.
  47. Kan: Kolmogorov-arnold networks, 2024.
  48. Andrew L. Maas. Rectifier nonlinearities improve neural network acoustic models. In International conference on machine learning, 2013.
  49. Ritabrata Maiti. Adact: Learning to optimize activation function choice through adaptive activation modules. In The Second Tiny Papers Track at ICLR 2024, 2024.
  50. Closed-form diffeomorphic transformations for time series alignment. In International Conference on Machine Learning, pages 15122–15158. PMLR, 2022.
  51. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127–163, 2000.
  52. A non-monotonic activation function for neural networks validated on benchmark tasks. In Modern Approaches in Machine Learning and Cognitive Science: A Walkthrough: Latest Trends in AI, Volume 2, pages 319–327. Springer, 2021.
  53. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4602–4609, 2019.
  54. Tudataset: A collection of benchmark datasets for learning with graphs, 2020.
  55. Weisfeiler and leman go machine learning: The story so far. Journal of Machine Learning Research, 24(333):1–59, 2023.
  56. Query-driven active surveying for collective classification. In 10th international workshop on mining and learning with graphs, volume 8, page 1, 2012.
  57. Towards interpretable sparse graph representation learning with laplacian pooling. arXiv preprint arXiv:1905.11577, 2019.
  58. Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378, 2018.
  59. Deep relu networks and high-order finite element methods. Analysis and Applications, 18(05):715–770, 2020.
  60. Effect of activation functions on the training of overparametrized neural nets. arXiv preprint arXiv:1908.05660, 2019.
  61. Pytorch: An imperative style, high-performance deep learning library, 2019.
  62. Deep neural network initialization with sparsity inducing activations. arXiv e-prints, pages arXiv–2402, 2024.
  63. Equivariant polynomials for graph neural networks. In International Conference on Machine Learning, pages 28191–28222. PMLR, 2023.
  64. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
  65. Recipe for a general, powerful, scalable graph transformer. Advances in Neural Information Processing Systems, 35:14501–14515, 2022.
  66. Graph neural networks for materials science and chemistry. Communications Materials, 3(1):93, 2022.
  67. Improving graph convolutional networks with non-parametric activation functions. In 2018 26th European Signal Processing Conference (EUSIPCO), pages 872–876. IEEE, 2018.
  68. The graph neural network model. IEEE transactions on neural networks, 20(1):61–80, 2008.
  69. Deep operator network approximation rates for lipschitz operators. arXiv preprint arXiv:2307.09835, 2023.
  70. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  71. Diffeomorphic temporal alignment nets. Advances in Neural Information Processing Systems, 32, 2019.
  72. ZINC 15 – ligand discovery for everyone. Journal of Chemical Information and Modeling, 55(11):2324–2337, 11 2015. doi: 10.1021/acs.jcim.5b00559.
  73. Variational pdes for acceleration on manifolds and application to diffeomorphisms. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  74. Tomasz Szandała. Review and comparison of commonly used activation functions for deep neural networks. Bio-inspired neurocomputing, pages 203–224, 2021.
  75. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
  76. Continuous piecewise-affine based motion model for image animation. arXiv preprint arXiv:2401.09146, 2024.
  77. Haar graph pooling. In International conference on machine learning, pages 9952–9962. PMLR, 2020.
  78. Regularization-free diffeomorphic temporal alignment nets. In International Conference on Machine Learning, pages 30794–30826. PMLR, 2023.
  79. Graph neural networks in recommender systems: a survey. ACM Computing Surveys, 55(5):1–37, 2022.
  80. Empirical evaluation of rectified activations in convolutional network, 2015.
  81. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
  82. Tafs: Task-aware activation function search for graph neural networks. 2023.
  83. Pane: scalable and effective attributed network embedding. The VLDB Journal, 32(6):1237–1262, March 2023. ISSN 0949-877X. doi: 10.1007/s00778-023-00790-4.
  84. Hierarchical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
  85. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests. In International Conference on Machine Learning, 2023a.
  86. Rethinking the expressive power of gnns via graph biconnectivity. arXiv preprint arXiv:2301.09505, 2023b.
  87. Bayesian statistical shape analysis on the manifold of diffeomorphisms. Algorithmic Advances in Riemannian Geometry and Applications: For Machine Learning, Computer Vision, Statistics, and Optimization, pages 1–23, 2016.
  88. Graph neural networks and their current applications in bioinformatics. Frontiers in genetics, 12:690049, 2021.
  89. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM Web Conference 2022, pages 1331–1339, 2022.
  90. Hierarchical graph pooling with structure learning. arXiv preprint arXiv:1911.05954, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 12 likes about this paper.

Sign Up for Free