Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equations (2402.17853v1)

Published 27 Feb 2024 in cs.LG, cs.AI, and math.AP

Abstract: Neural networks have shown promising potential in accelerating the numerical simulation of systems governed by partial differential equations (PDEs). Different from many existing neural network surrogates operating on high-dimensional discretized fields, we propose to learn the dynamics of the system in the latent space with much coarser discretizations. In our proposed framework - Latent Neural PDE Solver (LNS), a non-linear autoencoder is first trained to project the full-order representation of the system onto the mesh-reduced space, then a temporal model is trained to predict the future state in this mesh-reduced space. This reduction process simplifies the training of the temporal model by greatly reducing the computational cost accompanying a fine discretization. We study the capability of the proposed framework and several other popular neural PDE solvers on various types of systems including single-phase and multi-phase flows along with varying system parameters. We showcase that it has competitive accuracy and efficiency compared to the neural PDE solver that operates on full-order space.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Li, Z.; Kovachki, N. B.; Azizzadenesheli, K.; liu, B.; Bhattacharya, K.; Stuart, A.; Anandkumar, A. Fourier Neural Operator for Parametric Partial Differential Equations. International Conference on Learning Representations. 2021
  2. Sanchez-Gonzalez, A.; Godwin, J.; Pfaff, T.; Ying, R.; Leskovec, J.; Battaglia, P. Learning to simulate complex physics with graph networks. International conference on machine learning. 2020; pp 8459–8468
  3. Pfaff, T.; Fortunato, M.; Sanchez-Gonzalez, A.; Battaglia, P. Learning Mesh-Based Simulation with Graph Networks. International Conference on Learning Representations. 2021
  4. Brandstetter, J.; Worrall, D. E.; Welling, M. Message Passing Neural PDE Solvers. International Conference on Learning Representations. 2022
  5. Stachenfeld, K.; Fielding, D. B.; Kochkov, D.; Cranmer, M.; Pfaff, T.; Godwin, J.; Cui, C.; Ho, S.; Battaglia, P.; Sanchez-Gonzalez, A. Learned Simulators for Turbulence. International Conference on Learning Representations. 2022
  6. Cao, S. Choose a Transformer: Fourier or Galerkin. Advances in Neural Information Processing Systems. 2021; pp 24924–24940
  7. Gupta, J. K.; Brandstetter, J. Towards Multi-spatiotemporal-scale Generalized PDE Modeling. Transactions on Machine Learning Research 2023,
  8. Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Stuart, A.; Bhattacharya, K.; Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems. 2020; pp 6755–6766
  9. Wang, R.; Kashinath, K.; Mustafa, M.; Albert, A.; Yu, R. Towards physics-informed deep learning for turbulent flow prediction. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020; pp 1457–1466
  10. Rahman, M. A.; Ross, Z. E.; Azizzadenesheli, K. U-NO: U-shaped Neural Operators. Transactions on Machine Learning Research 2023,
  11. Ho, J.; Jain, A.; Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems. 2020; pp 6840–6851
  12. Han, X.; Gao, H.; Pfaff, T.; Wang, J.-X.; Liu, L. Predicting Physics in Mesh-reduced Space with Temporal Attention. International Conference on Learning Representations. 2022
  13. Ummenhofer, B.; Prantl, L.; Thuerey, N.; Koltun, V. Lagrangian Fluid Simulation with Continuous Convolutions. International Conference on Learning Representations. 2020
  14. Prantl, L.; Ummenhofer, B.; Koltun, V.; Thuerey, N. Guaranteed conservation of momentum for learning particle-based fluid dynamics. Advances in Neural Information Processing Systems. 2022; pp 6901–6913
  15. Lötzsch, W.; Ohler, S.; Otterbach, J. Learning the Solution Operator of Boundary Value Problems using Graph Neural Networks. ICML 2022 2nd AI for Science Workshop. 2022
  16. JANNY, S.; Bénéteau, A.; Nadri, M.; Digne, J.; THOME, N.; Wolf, C. EAGLE: Large-scale Learning of Turbulent Fluid Dynamics with Mesh Transformers. International Conference on Learning Representations. 2023
  17. Rasp, S.; Dueben, P. D.; Scher, S.; Weyn, J. A.; Mouatadid, S.; Thuerey, N. WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting. Journal of Advances in Modeling Earth Systems 2020, 12
  18. Nguyen, T.; Brandstetter, J.; Kapoor, A.; Gupta, J. K.; Grover, A. ClimaX: A foundation model for weather and climate. Proceedings of the 40th International Conference on Machine Learning. 2023; pp 25904–25938
  19. Kurth, T.; Subramanian, S.; Harrington, P.; Pathak, J.; Mardani, M.; Hall, D.; Miele, A.; Kashinath, K.; Anandkumar, A. Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators. Proceedings of the Platform for Advanced Scientific Computing Conference. 2023; pp 1–11
  20. Brandstetter, J.; van den Berg, R.; Welling, M.; Gupta, J. K. Clifford Neural Layers for PDE Modeling. International Conference on Learning Representations. 2023
  21. Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Bhattacharya, K.; Stuart, A.; Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485 2020,
  22. Brandstetter, J.; Welling, M.; Worrall, D. E. Lie Point Symmetry Data Augmentation for Neural PDE Solvers. Proceedings of the 39th International Conference on Machine Learning. 2022; pp 2241–2256
  23. Gupta, G.; Xiao, X.; Bogdan, P. Multiwavelet-based Operator Learning for Differential Equations. Advances in Neural Information Processing Systems. 2021
  24. Li, Z.; Meidani, K.; Farimani, A. B. Transformer for Partial Differential Equations’ Operator Learning. Transactions on Machine Learning Research 2023,
  25. Hao, Z.; Wang, Z.; Su, H.; Ying, C.; Dong, Y.; Liu, S.; Cheng, Z.; Song, J.; Zhu, J. GNOT: A General Neural Operator Transformer for Operator Learning. Proceedings of the 40th International Conference on Machine Learning. 2023; pp 12556–12569
  26. Ovadia, O.; Kahana, A.; Stinis, P.; Turkel, E.; Karniadakis, G. E. ViTO: Vision Transformer-Operator. arXiv preprint arXiv:2303.08891 2023,
  27. Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Bhattacharya, K.; Stuart, A.; Anandkumar, A. Neural Operator: Graph Kernel Network for Partial Differential Equations. 2020,
  28. Tran, A.; Mathews, A.; Xie, L.; Ong, C. S. Factorized Fourier Neural Operators. The Eleventh International Conference on Learning Representations. 2023
  29. Guibas, J.; Mardani, M.; Li, Z.; Tao, A.; Anandkumar, A.; Catanzaro, B. Efficient Token Mixing for Transformers via Adaptive Fourier Neural Operators. International Conference on Learning Representations. 2022
  30. Li, Z.; Zheng, H.; Kovachki, N.; Jin, D.; Chen, H.; Liu, B.; Azizzadenesheli, K.; Anandkumar, A. Physics-Informed Neural Operator for Learning Partial Differential Equations. 2023,
  31. Lorsung, C.; Farimani, A. B. PICL: Physics Informed Contrastive Learning for Partial Differential Equations. 2024
  32. Lorsung, C.; Li, Z.; Farimani, A. B. Physics Informed Token Transformer for Solving Partial Differential Equations. 2024
  33. Razavi, A.; Van den Oord, A.; Vinyals, O. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems. 2019
  34. Esser, P.; Rombach, R.; Ommer, B. Taming transformers for high-resolution image synthesis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021; pp 12873–12883
  35. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; Ommer, B. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022; pp 10684–10695
  36. Zeng, X.; Vahdat, A.; Williams, F.; Gojcic, Z.; Litany, O.; Fidler, S.; Kreis, K. LION: Latent Point Diffusion Models for 3D Shape Generation. Advances in Neural Information Processing Systems. 2022
  37. Hsieh, J.-T.; Zhao, S.; Eismann, S.; Mirabella, L.; Ermon, S. Learning Neural PDE Solvers with Convergence Guarantees. International Conference on Learning Representations. 2019
  38. Lee, K.; Carlberg, K. T. Deep Conservation: A Latent-Dynamics Model for Exact Satisfaction of Physical Conservation Laws. 2021; pp 277–285
  39. Wiewel, S.; Becher, M.; Thürey, N. Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow. Computer Graphics Forum 2019, 38
  40. Morton, J.; Jameson, A.; Kochenderfer, M. J.; Witherden, F. Deep Dynamical Modeling and Control of Unsteady Fluid Flows. Advances in Neural Information Processing Systems. 2018
  41. Li, Y.; He, H.; Wu, J.; Katabi, D.; Torralba, A. Learning Compositional Koopman Operators for Model-Based Control. International Conference on Learning Representations. 2020
  42. Takeishi, N.; Kawahara, Y.; Yairi, T. Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition. Advances in Neural Information Processing Systems. 2017
  43. Hemmasian, A.; Barati Farimani, A. Reduced-order modeling of fluid flows with transformers. Physics of Fluids 2023, 35
  44. Hemmasian, A.; Farimani, A. B. Multi-scale Time-stepping of Partial Differential Equations with Transformers. 2023
  45. van den Oord, A.; Vinyals, O.; kavukcuoglu, k. Neural Discrete Representation Learning. Advances in Neural Information Processing Systems. 2017
  46. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L. u.; Polosukhin, I. Attention is All you Need. Advances in Neural Information Processing Systems. 2017
  47. Zhang, H.; Goodfellow, I.; Metaxas, D.; Odena, A. Self-Attention Generative Adversarial Networks. Proceedings of the 36th International Conference on Machine Learning. 2019; pp 7354–7363
  48. Guo, R.; Cao, S.; Chen, L. Transformer Meets Boundary Value Inverse Problems. International Conference on Learning Representations. 2023
  49. Li, Z.; Shu, D.; Farimani, A. B. Scalable Transformer for PDE Surrogate Modeling. Thirty-seventh Conference on Neural Information Processing Systems. 2023
  50. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016
  51. Ulyanov, D.; Vedaldi, A.; Lempitsky, V. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 2016,
  52. Ba, J. L.; Kiros, J. R.; Hinton, G. E. Layer normalization. arXiv preprint arXiv:1607.06450 2016,
  53. Trockman, A.; Kolter, J. Z. Patches Are All You Need? Transactions on Machine Learning Research 2023, Featured Certification
  54. Tolstikhin, I.; Houlsby, N.; Kolesnikov, A.; Beyer, L.; Zhai, X.; Unterthiner, T.; Yung, J.; Steiner, A. P.; Keysers, D.; Uszkoreit, J.; Lucic, M.; Dosovitskiy, A. MLP-Mixer: An all-MLP Architecture for Vision. Advances in Neural Information Processing Systems. 2021
  55. Nichol, A. Q.; Dhariwal, P. Improved Denoising Diffusion Probabilistic Models. Proceedings of the 38th International Conference on Machine Learning. 2021; pp 8162–8171
  56. Reynolds, O. IV. On the dynamical theory of incompressible viscous fluids and the determination of the criterion. Philosophical transactions of the royal society of london.(a.) 1895, 123–164
  57. Launder, B. E.; Spalding, D. B. The numerical computation of turbulent flows. 1983, 96–116
  58. Rognebakke, O. F.; Faltinsen, O. M. Sloshing induced impact with air cavity in rectangular tank with a high filling ratio. 20th international workshop on water waves and floating bodies. 2005; pp 217–20
  59. Wu, Y.; He, K. Group normalization. Proceedings of the European conference on computer vision (ECCV). 2018; pp 3–19
  60. Vahdat, A.; Kreis, K.; Kautz, J. Score-based Generative Modeling in Latent Space. Advances in Neural Information Processing Systems. 2021
  61. Patankar, S. V. Numerical methods in heat transfer. International Heat Transfer Conference Digital Library. 1982
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now