Papers
Topics
Authors
Recent
Search
2000 character limit reached

HAMLET: Graph Transformer Neural Operator for Partial Differential Equations

Published 5 Feb 2024 in cs.LG, cs.NA, and math.NA | (2402.03541v2)

Abstract: We present a novel graph transformer framework, HAMLET, designed to address the challenges in solving partial differential equations (PDEs) using neural networks. The framework uses graph transformers with modular input encoders to directly incorporate differential equation information into the solution process. This modularity enhances parameter correspondence control, making HAMLET adaptable to PDEs of arbitrary geometries and varied input formats. Notably, HAMLET scales effectively with increasing data complexity and noise, showcasing its robustness. HAMLET is not just tailored to a single type of physical simulation, but can be applied across various domains. Moreover, it boosts model resilience and performance, especially in scenarios with limited data. We demonstrate, through extensive experiments, that our framework is capable of outperforming current techniques for PDEs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. Graph element networks: adaptive, structured computation and memory. In International Conference on Machine Learning, pp.  212–222. PMLR, 2019.
  2. Layer normalization. In Advances in neural information processing systems, pp.  198–206, 2016.
  3. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
  4. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
  5. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64:525–545, 2019.
  6. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7:121–157, 2021.
  7. Specformer: Spectral graph neural networks meet transformers. arXiv preprint arXiv:2303.01028, 2023.
  8. Magnet: Mesh agnostic neural pde solver, 2022.
  9. Clifford neural layers for pde modeling. arXiv preprint arXiv:2209.04934, 2022a.
  10. Lie point symmetry data augmentation for neural pde solvers. In International Conference on Machine Learning, pp.  2241–2256. PMLR, 2022b.
  11. Message passing neural pde solvers. arXiv preprint arXiv:2202.03376, 2022c.
  12. Machine learning for fluid mechanics. Annual review of fluid mechanics, 52:477–508, 2020.
  13. Cao, S. Choose a transformer: Fourier or galerkin. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  24924–24940. Curran Associates, Inc., 2021a.
  14. Cao, S. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924–24940, 2021b.
  15. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
  16. The deep ritz method: A deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018.
  17. Spectral neural operators. arXiv preprint arXiv:2205.10573, 2022.
  18. Limitations of physics informed machine learning for nonlinear two-phase transport in porous media. Journal of Machine Learning for Modeling and Computing, 1(1), 2020.
  19. Transformers for modeling physical systems. Neural Networks, 146:272–289, 2022.
  20. Neural message passing for quantum chemistry. In International conference on machine learning, pp.  1263–1272. PMLR, 2017.
  21. Adaptive fourier neural operators: Efficient token mixers for transformers, 2022.
  22. Multiwavelet-based operator learning for differential equations. arXiv preprint arXiv:2109.13459, 2021.
  23. Predicting physics in mesh-reduced space with temporal attention. arXiv preprint arXiv:2201.09113, 2022.
  24. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  25. Physics-embedded neural networks: Graph neural pde solvers with mixed boundary conditions. Advances in Neural Information Processing Systems, 35:23218–23229, 2022.
  26. Learning continuous-time pdes from sparse data with graph neural networks. arXiv preprint arXiv:2006.08956, 2020.
  27. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
  28. Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. In AAAI Spring Symposium: MLPS, pp.  2002–2041, 2021.
  29. Sparse identification of nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 474, 2018.
  30. Physics-informed machine learning. Nature Reviews Physics, pp.  1–19, 2021. doi: 10.1038/s42254-021-00314-5.
  31. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4d flow mri data using physics-informed neural networks. Computer Methods in Applied Mechanics and Engineering, 358:112623, 2020.
  32. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1–63, 2022.
  33. Neural operator: Learning maps between function spaces, 2021.
  34. Graph neural network-accelerated lagrangian fluid simulation. Computers & Graphics, 103:201–211, 2022.
  35. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020a.
  36. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020b.
  37. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020c.
  38. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.
  39. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209, 2022a.
  40. Fourier Neural Operator with Learned Deformations for PDEs on General Geometries, 2022b.
  41. Transformer for partial differential equations’ operator learning. Transactions on Machine Learning Research, 2023.
  42. Multi-fidelity physics-constrained neural network and its application in materials modeling. Journal of Mechanical Design, 141(12), 2019.
  43. Pde-net 2.0: Learning pdes from data with a numeric-symbolic hybrid deep network. Journal of Computational Physics, 399:108925, 2019.
  44. Learning the solution operator of boundary value problems using graph neural networks. arXiv preprint arXiv:2206.14092, 2022.
  45. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. Nature Machine Intelligence, 3:218–229, March 2021. arXiv:1910.03193 [cs, stat].
  46. Transformer for graphs: An overview from architecture perspective. arXiv preprint arXiv:2202.08455, 2022.
  47. Iae-net: Integral autoencoders for discretization-invariant learning. arXiv preprint arXiv:2203.05142, 2022.
  48. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
  49. Learning time-dependent pde solver using message passing graph neural networks. arXiv preprint arXiv:2204.07651, 2022.
  50. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, February 2019a. ISSN 00219991. doi: 10.1016/j.jcp.2018.10.045.
  51. Deep learning of vortex-induced vibrations. Journal of Fluid Mechanics, 861:119–137, 2019b.
  52. U-net: Convolutional networks for biomedical image segmentation. arXiv preprint arXiv:1505.04597, 2015.
  53. Physics-informed neural networks for cardiac activation mapping. Frontiers in Physics, 8:42, 2020.
  54. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp.  4470–4479. PMLR, 2018.
  55. Sit: Simulation transformer for particle-based physics simulation. International Conference on Learning Representations, 2022.
  56. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339–1364, 2018.
  57. M2n: mesh movement networks for pde solvers. Advances in Neural Information Processing Systems, 35:7199–7210, 2022.
  58. Learned simulators for turbulence. In International Conference on Learning Representations, 2022.
  59. RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv e-prints, art. arXiv:2104.09864, April 2021. doi: 10.48550/arXiv.2104.09864.
  60. Surrogate modeling for fluid flows based on physics-constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:112732, 2020.
  61. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1596–1611, 2022.
  62. Fourier features let networks learn high frequency functions in low dimensional domains. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  7537–7547. Curran Associates, Inc., 2020.
  63. Factorized fourier neural operators. arXiv preprint arXiv:2111.13802, 2021.
  64. Wavelet neural operator: A neural operator for parametric partial differential equations. arXiv preprint arXiv:2205.02191, 2022.
  65. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.  1457–1466, 2020.
  66. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021a.
  67. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021b.
  68. Graph transformer networks. Advances in neural information processing systems, 32, 2019.
  69. Incremental spectral learning in fourier neural operator. arXiv preprint arXiv:2211.15188, 2022.
  70. Physics-informed autoencoder for solving nonlinear partial differential equations. Proceedings of the National Academy of Sciences, 116(45):22252–22263, 2019.
  71. Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415–447, 2018.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free