Papers
Topics
Authors
Recent
Search
2000 character limit reached

Reversible and irreversible bracket-based dynamics for deep graph neural networks

Published 24 May 2023 in cs.LG | (2305.15616v3)

Abstract: Recent works have shown that physics-inspired architectures allow the training of deep graph neural networks (GNNs) without oversmoothing. The role of these physics is unclear, however, with successful examples of both reversible (e.g., Hamiltonian) and irreversible (e.g., diffusion) phenomena producing comparable results despite diametrically opposed mechanisms, and further complications arising due to empirical departures from mathematical theory. This work presents a series of novel GNN architectures based upon structure-preserving bracket-based dynamical systems, which are provably guaranteed to either conserve energy or generate positive dissipation with increasing depth. It is shown that the theoretically principled framework employed here allows for inherently explainable constructions, which contextualize departures from theory in current architectures and better elucidate the roles of reversibility and irreversibility in network performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729–734. IEEE, 2005.
  2. Graph neural networks: A review of methods and applications. AI open, 1:57–81, 2020.
  3. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
  4. Representation learning on graphs: Methods and applications. arXiv preprint arXiv:1709.05584, 2017.
  5. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3438–3445, 2020.
  6. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021.
  7. Towards deeper graph neural networks with differentiable group normalization. Advances in neural information processing systems, 33:4917–4928, 2020.
  8. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.13318, 2020.
  9. Dissecting the diffusion process in linear graph convolutional networks. Advances in Neural Information Processing Systems, 34:5758–5769, 2021.
  10. Understanding convolution on graphs via energies, 2023.
  11. Enforcing exact physics in scientific machine learning: a data-driven exterior calculus on graphs. arXiv preprint arXiv:2012.11799, 2020.
  12. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
  13. Scalable algorithms for physics-informed neural and graph networks. Data-Centric Engineering, 3:e24, 2022.
  14. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04385, 2020.
  15. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
  16. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6572–6583, 2018.
  17. Torchdyn: Implicit models and neural numerical methods in pytorch. ddd, 2020.
  18. Continuous graph neural networks. In International Conference on Machine Learning, pages 10432–10441. PMLR, 2020.
  19. Implicit graph neural networks. Advances in Neural Information Processing Systems, 33:11984–11995, 2020.
  20. Hamiltonian neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  21. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. Advances in neural information processing systems, 33:13880–13889, 2020.
  22. Data-driven prediction of general hamiltonian dynamics via learning exactly-symplectic maps. In International Conference on Machine Learning, pages 1717–1727. PMLR, 2021.
  23. Deconstructing the inductive biases of hamiltonian neural networks. In International Conference on Learning Representations, 2020.
  24. Hamiltonian generative networks. In International Conference on Learning Representations, 2019.
  25. Symplectic ODE-Net: Learning Hamiltonian dynamics with control. In International Conference on Learning Representations, 2019.
  26. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2018.
  27. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
  28. Partha Guha. Metriplectic structure, leibniz dynamics and dissipative systems. Journal of Mathematical Analysis and Applications, 326(1):121–136, 2007.
  29. Machine learning structure preserving brackets for forecasting irreversible processes. Advances in Neural Information Processing Systems, 34:5696–5707, 2021.
  30. Structure-preserving sparse identification of nonlinear dynamics for data-driven modeling. In Mathematical and Scientific Machine Learning, pages 65–80. PMLR, 2022.
  31. Gfinns: Generic formalism informed neural networks for deterministic and stochastic dynamical systems. Philosophical Transactions of the Royal Society A, 380(2229):20210207, 2022.
  32. Dissipative symoden: Encoding hamiltonian dynamics with dissipation and control into deep learning. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
  33. Port-hamiltonian neural networks for learning explicit time-dependent dynamical systems. Physical Review E, 104(3):034312, 2021.
  34. Miroslav Grmela. Generic guide to the multiscale dynamics and thermodynamics. Journal of Physics Communications, 2(3):032001, 2018.
  35. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  36. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125–141, 2018.
  37. Multiscale modeling of inelastic materials with thermodynamics-based artificial neural networks (tann). Computer Methods in Applied Mechanics and Engineering, 398:115190, 2022.
  38. Thermodynamically consistent physics-informed neural networks for hyperbolic systems. Journal of Computational Physics, 449:110754, 2022.
  39. Structure-preserving neural networks. Journal of Computational Physics, 426:109950, 2021.
  40. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics, 394:136–152, 2019.
  41. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397:108850, 2019.
  42. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021.
  43. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022.
  44. Graph-coupled oscillator networks. In International Conference on Machine Learning, pages 18888–18909. PMLR, 2022.
  45. Gread: Graph neural reaction-diffusion equations. arXiv preprint arXiv:2211.14208, 2022.
  46. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv:1909.12790, 2019.
  47. Enhancing the inductive biases of graph neural ode for modeling physical systems. In The Eleventh International Conference on Learning Representations, 2022.
  48. Thermodynamics-informed graph neural networks. IEEE Transactions on Artificial Intelligence, 1(01):1–1, 2022.
  49. Grand: Graph neural diffusion. In International Conference on Machine Learning, pages 1407–1418. PMLR, 2021.
  50. PJ Morrison. Thoughts on brackets and dissipation: old and new. In Journal of Physics: Conference Series, volume 169, page 012006. IOP Publishing, 2009.
  51. Oliver Knill. The dirac operator of a graph, 2013.
  52. Statistical ranking and combinatorial hodge theory. Mathematical Programming, 127(1):203–244, 2011.
  53. Principles of mimetic discretizations of differential operators. In Compatible spatial discretizations, pages 89–119. Springer, 2006.
  54. Douglas N Arnold. Finite element exterior calculus. SIAM, 2018.
  55. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  56. Ignacio Romero. Thermodynamically consistent time-stepping algorithms for non-linear thermomechanical systems. International Journal for Numerical Methods in Engineering, 79(6):706–732, 2023/05/14 2009.
  57. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012.
  58. Automating the construction of internet portals with machine learning. Information Retrieval, 3:127–163, 2000.
  59. Collective classification in network data. AI magazine, 29(3):93–93, 2008.
  60. Query-driven active surveying for collective classification. In 10th international workshop on mining and learning with graphs, volume 8, page 1, 2012.
  61. Pitfalls of graph neural network evaluation. ArXiv, abs/1811.05868, 2018.
  62. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 43–52, 2015.
  63. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532, 2019.
  64. Statistical ranking and combinatorial hodge theory. Mathematical Programming, 127(1):203–244, nov 2010.
  65. The euler-poincaré equations and double bracket dissipation. Communications in Mathematical Physics, 175(1):1–42, 1996.
  66. The euler–poincaré equations and semidirect products with applications to continuum theories. Advances in Mathematics, 137(1):1–81, 1998.
  67. Hans Christian Oettinger. Irreversible dynamics, onsager-casimir symmetry, and an application to turbulence. Physical Review E, 90(4):042121, 2014.
  68. Energetically consistent model reduction for metriplectic systems. Computer Methods in Applied Mechanics and Engineering, 404:115709, 2023.
  69. Robert J. Renka. A simple explanation of the sobolev gradient method. In A Simple Explanation of the Sobolev Gradient Method, 2006.
  70. Repulsive curves. ACM Trans. Graph., 40(2), may 2021.
  71. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  72. Joe Chen. Chaos from simplicity: an introduction to the double pendulum. Technical report, University of Canterbury, 2008.
  73. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  74. Interconnection and damping assignment passivity-based control of port-controlled hamiltonian systems. Automatica, 38(4):585–596, 2002.
  75. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016.
  76. Lukas Biewald. Experiment tracking with weights and biases, 2020. Software available from wandb.com.
Citations (9)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now