Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Improving PINNs By Algebraic Inclusion of Boundary and Initial Conditions (2407.20741v1)

Published 30 Jul 2024 in cs.LG, cs.NA, math.DS, and math.NA

Abstract: "AI for Science" aims to solve fundamental scientific problems using AI techniques. As most physical phenomena can be described as Partial Differential Equations (PDEs) , approximating their solutions using neural networks has evolved as a central component of scientific-ML. Physics-Informed Neural Networks (PINNs) is the general method that has evolved for this task but its training is well-known to be very unstable. In this work we explore the possibility of changing the model being trained from being just a neural network to being a non-linear transformation of it - one that algebraically includes the boundary/initial conditions. This reduces the number of terms in the loss function than the standard PINN losses. We demonstrate that our modification leads to significant performance gains across a range of benchmark tasks, in various dimensions and without having to tweak the training algorithm. Our conclusions are based on conducting hundreds of experiments, in the fully unsupervised setting, over multiple linear and non-linear PDEs set to exactly solvable scenarios, which lends to a concrete measurement of our performance gains in terms of order(s) of magnitude lower fractional errors being achieved, than by standard PINNs. The code accompanying this manuscript is publicly available at, https://github.com/MorganREN/Improving-PINNs-By-Algebraic-Inclusion-of-Boundary-and-Initial-Conditions

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Ravi P Agarwal. Burgers’ equation (viscous). Indian Institute of Space Science and Technology, 2011.
  2. Active training of physics-informed neural networks to aggregate and interpolate parametric solutions to the navier-stokes equations. Journal of Computational Physics, 438:110364, 2021. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2021.110364. URL https://www.sciencedirect.com/science/article/pii/S002199912100259X.
  3. Solving the kolmogorov PDE by means of deep learning. Journal of Scientific Computing, 88(3), jul 2021. doi: 10.1007/s10915-021-01590-0. URL https://doi.org/10.1007%2Fs10915-021-01590-0.
  4. Saida Bendaas. Periodic wave shock solutions of burgers equations. Cogent Mathematics & Statistics, 5(1):1463597, 2018.
  5. Numerically solving parametric families of high-dimensional kolmogorov partial differential equations via deep learning. Advances in Neural Information Processing Systems, 33:16615–16627, 2020.
  6. Three ways to solve partial differential equations with neural networks—a review. GAMM-Mitteilungen, 44(2):e202100006, 2021.
  7. Radial basis functions, multi-variable functional interpolation and adaptive networks. ROYAL SIGNALS AND RADAR ESTABLISHMENT MALVERN (UNITED KINGDOM), RSRE-MEMO-4148, 03 1988.
  8. EM De Jager. On the origin of the korteweg-de vries equation. arXiv preprint math/0602661, 2006.
  9. Uniform convergence guarantees for the deep ritz method for nonlinear problems. Advances in Continuous and Discrete Models, 2022(1):1–19, 2022.
  10. Solitons: An Introduction. Cambridge Texts in Applied Mathematics. Cambridge University Press, 2 edition, 1989. doi: 10.1017/CBO9781139172059.
  11. Algorithms for solving high dimensional PDEs: from nonlinear monte carlo to machine learning. Nonlinearity, 35(1):278–310, dec 2021. doi: 10.1088/1361-6544/ac337f. URL https://doi.org/10.1088%2F1361-6544%2Fac337f.
  12. Physics-informed neural networks for solving reynolds-averaged navier–stokes equations. Physics of Fluids, 34(7):075117, jul 2022. doi: 10.1063/5.0095270. URL https://doi.org/10.1063%2F5.0095270.
  13. Physics-informed autoencoders for lyapunov-stable fluid flow prediction. arXiv preprint arXiv:1905.10866, 2019.
  14. Numerical solution of the parametric diffusion equation by deep neural networks. Journal of Scientific Computing, 88(1):22, 2021.
  15. Training machine learning models on climate model output yields skillful interpretable seasonal precipitation forecasts. Communications Earth and Environment, 2(1):159, December 2021. doi: 10.1038/s43247-021-00225-4.
  16. Global convergence of SGD for logistic loss on two layer neural nets. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
  17. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  18. When do extended physics-informed neural networks (XPINNs) improve generalization? SIAM Journal on Scientific Computing, 44(5):A3158–A3182, sep 2022. doi: 10.1137/21m1447039. URL https://doi.org/10.1137%2F21m1447039.
  19. Bias-variance trade-off in physics-informed neural networks with randomized smoothing for high-dimensional pdes. arXiv preprint arXiv:2311.15283, 2023.
  20. Towards neural earth system modelling by integrating artificial intelligence in earth system science. Nature Machine Intelligence, 3:667–674, 08 2021. doi: 10.1038/s42256-021-00374-3.
  21. Global convergence of deep galerkin and pinns methods for solving partial differential equations, 2023.
  22. Data-driven discovery of koopman eigenfunctions for control. Machine Learning: Science and Technology, 2(3):035023, 2021.
  23. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021.
  24. Characterizing possible failure modes in physics-informed neural networks. Advances in neural information processing systems, 34:26548–26560, 2021.
  25. Investigating the ability of pinns to solve burgers’ pde near finite-time blowup. Machine Learning: Science and Technology, 5(2):025063, jun 2024. doi: 10.1088/2632-2153/ad51cd. URL https://dx.doi.org/10.1088/2632-2153/ad51cd.
  26. A theoretical analysis of deep neural networks and parametric pdes. Constructive Approximation, 55(1):73–125, 2022.
  27. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987–1000, 1998. doi: 10.1109/72.712178. URL https://doi.org/10.1109%2F72.712178.
  28. Physics-informed neural network (pinn) evolution and beyond: A systematic literature review and bibliometric analysis. Big Data and Cognitive Computing, 11 2022. doi: 10.3390/bdcc6040140.
  29. Chapter 12 - nonlinear differential equations. In Brent J. Lewis, E. Nihan Onder, and Andrew A. Prudil (eds.), Advanced Mathematics for Engineering Students, pp.  329–347. Butterworth-Heinemann, 2022. ISBN 978-0-12-823681-9. doi: https://doi.org/10.1016/B978-0-12-823681-9.00020-4. URL https://www.sciencedirect.com/science/article/pii/B9780128236819000204.
  30. Learning dissipative dynamics in chaotic systems, 2022.
  31. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, mar 2021a. doi: 10.1038/s42256-021-00302-5. URL https://doi.org/10.1038%2Fs42256-021-00302-5.
  32. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021b.
  33. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022. ISSN 0045-7825. doi: https://doi.org/10.1016/j.cma.2022.114778. URL https://www.sciencedirect.com/science/article/pii/S0045782522001207.
  34. A priori generalization analysis of the deep ritz method for solving high dimensional elliptic partial differential equations. In Conference on learning theory, pp.  3196–3241. PMLR, 2021c.
  35. Artificial neural network method for solution of boundary value problems with exact satisfaction of arbitrary boundary conditions. IEEE Transactions on Neural Networks, 20(8):1221–1233, 2009. doi: 10.1109/TNN.2009.2020735.
  36. Estimates on the generalization error of physics-informed neural networks for approximating PDEs. IMA Journal of Numerical Analysis, 43(1):1–43, 01 2022. ISSN 0272-4979. doi: 10.1093/imanum/drab093. URL https://doi.org/10.1093/imanum/drab093.
  37. Error estimates for the deep ritz method with boundary penalty. In Mathematical and Scientific Machine Learning, pp. 215–230. PMLR, 2022.
  38. Space-fractional advection–dispersion equations by the kansa method. Journal of Computational Physics, 293:280–296, 2015. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2014.07.020. URL https://www.sciencedirect.com/science/article/pii/S0021999114005130. Fractional PDEs.
  39. Computational fluid mechanics and heat transfer. CRC press, 2012.
  40. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  41. A Salih. Inviscid burgers’ equation. A university lecture, Indian Institute of Space Science and Technology, 2015.
  42. Neural stochastic pdes: Resolution-invariant learning of continuous spatiotemporal dynamics. Advances in Neural Information Processing Systems, 35:1333–1344, 2022.
  43. What do the navier–stokes equations mean? European Journal of Physics, 35(1):015020, dec 2013. doi: 10.1088/0143-0807/35/1/015020. URL https://dx.doi.org/10.1088/0143-0807/35/1/015020.
  44. Modified cubic b-spline differential quadrature method for numerical solution of three-dimensional coupled viscous burger equation. Modern Physics Letters B, 30(11):1650110, 2016.
  45. Ya G Sinai. Statistics of shocks in solutions of inviscid burgers equation. Communications in Mathematical Physics, 148(3):601–621, 1992.
  46. N Sukumar and Ankit Srivastava. Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks. Computer Methods in Applied Mechanics and Engineering, 389:114333, 2022.
  47. Teaching the incompressible navier–stokes equations to fast neural surrogate models in three dimensions. Physics of Fluids, 33(4):047117, apr 2021. doi: 10.1063/5.0047428. URL https://doi.org/10.1063%2F5.0047428.
  48. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  1457–1466, 2020.
  49. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021a.
  50. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 7(40):eabi8605, 2021b. doi: 10.1126/sciadv.abi8605. URL https://www.science.org/doi/abs/10.1126/sciadv.abi8605.
  51. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022.
  52. Asymptotic self-similar blow-up profile for three-dimensional axisymmetric euler equations using neural networks. Physical Review Letters, 130(24):244002, 2023.
  53. Understanding loss landscapes of neural network models in solving partial differential equations, 2021.
  54. Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018.
  55. Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions. Nature Communications, 11(1), jul 2020. doi: 10.1038/s41467-020-17142-3. URL https://doi.org/10.1038%2Fs41467-020-17142-3.
  56. Interaction of "solitons" in a collisionless plasma and the recurrence of initial states. Phys. Rev. Lett., 15:240–243, Aug 1965. doi: 10.1103/PhysRevLett.15.240. URL https://link.aps.org/doi/10.1103/PhysRevLett.15.240.
  57. Numerical solutions of two-dimensional burgers’ equations by discrete adomian decomposition method. Computers & Mathematics with Applications, 60(3):840–848, 2010.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets