Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks (2304.13807v2)

Published 26 Apr 2023 in cs.NE

Abstract: Ordinary and partial differential equations (DE) are used extensively in scientific and mathematical domains to model physical systems. Current literature has focused primarily on deep neural network (DNN) based methods for solving a specific DE or a family of DEs. Research communities with a history of using DE models may view DNN-based differential equation solvers (DNN-DEs) as a faster and transferable alternative to current numerical methods. However, there is a lack of systematic surveys detailing the use of DNN-DE methods across physical application domains and a generalized taxonomy to guide future research. This paper surveys and classifies previous works and provides an educational tutorial for senior practitioners, professionals, and graduate students in engineering and computer science. First, we propose a taxonomy to navigate domains of DE systems studied under the umbrella of DNN-DE. Second, we examine the theory and performance of the Physics Informed Neural Network (PINN) to demonstrate how the influential DNN-DE architecture mathematically solves a system of equations. Third, to reinforce the key ideas of solving and discovery of DEs using DNN, we provide a tutorial using DeepXDE, a Python package for developing PINNs, to develop DNN-DEs for solving and discovering a classic DE, the linear transport equation.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Solving nonlinear and high-dimensional partial differential equations via deep learning. arXiv (nov 2018). arXiv:1811.08782 http://arxiv.org/abs/1811.08782
  2. Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? Advances in neural information processing systems 27 (2014).
  3. Workshop report on basic research needs for scientific machine learning: Core technologies for artificial intelligence. Technical Report. USDOE Office of Science (SC), Washington, DC (United States).
  4. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research 18 (2018), 1–43.
  5. Deep Splitting Method for Parabolic PDEs. SIAM Journal on Scientific Computing 43, 5 (Jan. 2021), A3135–A3154. https://doi.org/10.1137/19M1297919
  6. Jens Berg and Kaj Nyström. 2018. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing 317 (nov 2018), 28–41. https://doi.org/10.1016/j.neucom.2018.06.056 arXiv:1711.06464
  7. Numerically solving parametric families of high-dimensional Kolmogorov partial differential equations via deep learning. arXiv (nov 2020). arXiv:2011.04602 http://arxiv.org/abs/2011.04602
  8. Michael Betancourt. 2018. A geometric theory of higher-order automatic differentiation. arXiv preprint arXiv:1812.11592 (2018).
  9. Taylor-mode automatic differentiation for higher-order derivatives in JAX. (2019).
  10. Avrim L Blum and Ronald L Rivest. 1992. Training a 3-node neural network is NP-complete. Neural Networks 5, 1 (1992), 117–127.
  11. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing 16, 5 (1995), 1190–1208.
  12. Comsol. 2019. Physics, PDEs, Mathematical and Numerical Modeling. Comsol. https://www.comsol.com/multiphysics/introduction-to-physics-pdes-and-numerical-modeling
  13. Chen Debao. 1993. Degree of approximation by superpositions of a sigmoidal function. Approximation Theory and its Applications 9, 3 (1993), 17–28. https://doi.org/10.1007/BF02836480
  14. Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations. Communications in Mathematics and Statistics 5, 4 (Dec. 2017), 349–380. https://doi.org/10.1007/s40304-017-0117-6
  15. Weinan E and Bing Yu. 2017. The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. (9 2017). http://arxiv.org/abs/1710.00211
  16. PhyGeoNet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state PDEs on irregular domain. J. Comput. Phys. 428 (mar 2021). https://doi.org/10.1016/j.jcp.2020.110079 arXiv:2004.13145
  17. A derivative-free method for solving elliptic partial differential equations with deep neural networks. J. Comput. Phys. 419 (oct 2020), 109672. https://doi.org/10.1016/j.jcp.2020.109672 arXiv:2001.06145
  18. Physics-informed neural networks for multiphysics data assimilation with application to subsurface transport. Advances in Water Resources 141, December 2019 (2020). https://doi.org/10.1016/j.advwatres.2020.103610 arXiv:1912.02968
  19. J. S. Hesthaven and S. Ubbiali. 2018. Non-intrusive reduced order modeling of nonlinear problems using neural networks. J. Comput. Phys. 363 (jun 2018), 55–78. https://doi.org/10.1016/j.jcp.2018.02.037
  20. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
  21. Multilayer feedforward networks are universal approximators. Neural Networks 2, 5 (1989), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8
  22. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. J. Comput. Phys. 404 (2020), 109136.
  23. hp-VPINNs: Variational physics-informed neural networks with domain decomposition. Computer Methods in Applied Mechanics and Engineering 374 (feb 2021). https://doi.org/10.1016/j.cma.2020.113547 arXiv:2003.05385
  24. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  25. Fourier Neural Operator for Parametric Partial Differential Equations. (oct 2020). arXiv:2010.08895 http://arxiv.org/abs/2010.08895
  26. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 (2020).
  27. Multipole graph neural operator for parametric partial differential equations. arXiv preprint arXiv:2006.09535 (2020).
  28. Shiyu Liang and Rayadurgam Srikant. 2016. Why deep neural networks for function approximation? arXiv preprint arXiv:1610.04161 (2016).
  29. Deepxde: A deep learning library for solving differential equations. arXiv (jul 2019). arXiv:1907.04502 http://arxiv.org/abs/1907.04502
  30. DeepXDE: A deep learning library for solving differential equations. SIAM Rev. 63, 1 (2021), 208–228.
  31. Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 3276–3285. https://proceedings.mlr.press/v80/lu18d.html
  32. Deep lagrangian networks: Using physics as model prior for deep learning. arXiv (jul 2019). arXiv:1907.04490 http://arxiv.org/abs/1907.04490
  33. Deep learning observables in computational fluid dynamics. J. Comput. Phys. 410 (2020), 1–57. https://doi.org/10.1016/j.jcp.2020.109339 arXiv:1903.03040
  34. Charles C Margossian. 2019. A review of automatic differentiation and its efficient implementation. Wiley interdisciplinary reviews: data mining and knowledge discovery 9, 4 (2019), e1305.
  35. Solving differential equations using deep neural networks. Neurocomputing 399 (jul 2020), 193–212. https://doi.org/10.1016/j.neucom.2020.02.015
  36. Siddhartha Mishra. 2018. A machine learning framework for data driven acceleration of computations of differential equations. arXiv (jul 2018). https://doi.org/10.3934/mine.2018.1.118 arXiv:1807.09519
  37. Siddhartha Mishra and Roberto Molinaro. 2021. Estimates on the generalization error of physics-informed neural networks for approximating a class of inverse problems for PDEs. IMA J. Numer. Anal. (2021), 1–35. https://doi.org/10.1093/imanum/drab032 arXiv:2007.01138
  38. Physics-informed neural networks for power systems. In 2020 IEEE Power & Energy Society General Meeting (PESGM). IEEE, 1–5.
  39. John C. Nash. 2000. The (Dantzig) simplex method for linear programming. Computing in Science and Engineering 2, 1 (2000), 29–31. https://doi.org/10.1109/5992.814654
  40. Arkadi Nemirovski. 2004. Interior Point Polynomial Time Methods in Convex Programming. Spring 42, 16 (2004), 3215–3224. arXiv:arXiv:1011.1669v3 http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.160.6909&rep=rep1&type=pdf
  41. Convex Optimization. 1965. 1. Introduction. The Mathematical Gazette 49 (3 1965), 1–15. Issue 369. http://www.jstor.org/stable/3612902?origin=crossref
  42. fPINNs: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing 41, 4 (2019), A2603–A2626.
  43. Turbulence forecasting via Neural ODE. arXiv (nov 2019). arXiv:1911.05180 http://arxiv.org/abs/1911.05180
  44. Maziar Raissi. 2018. Forward-Backward Stochastic Neural Networks: Deep Learning of High-dimensional Partial Differential Equations. arXiv (apr 2018). arXiv:1804.07010 http://arxiv.org/abs/1804.07010
  45. Maziar Raissi and George Em Karniadakis. 2018. Hidden physics models: Machine learning of nonlinear partial differential equations. J. Comput. Phys. 357 (mar 2018), 125–141. https://doi.org/10.1016/j.jcp.2017.11.039 arXiv:1708.00588
  46. Physics informed deep learning (Part II): Data-driven discovery of nonlinear partial differential equations. arXiv (nov 2017). arXiv:1711.10566 http://arxiv.org/abs/1711.10561
  47. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378 (feb 2019), 686–707. https://doi.org/10.1016/j.jcp.2018.10.045
  48. An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications. Computer Methods in Applied Mechanics and Engineering 362 (apr 2020), 112790. https://doi.org/10.1016/j.cma.2019.112790 arXiv:1908.10407
  49. Justin Sirignano and Konstantinos Spiliopoulos. 2018. DGM: A deep learning algorithm for solving partial differential equations. J. Comput. Phys. 375 (dec 2018), 1339–1364. https://doi.org/10.1016/j.jcp.2018.08.029 arXiv:1708.07469
  50. Learning without Data: Physics-Informed Neural Networks for Fast Time-Domain Simulation. 2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids, SmartGridComm 2021, 438–443. https://doi.org/10.1109/SmartGridComm51999.2021.9631995
  51. Belinda Tzen and Maxim Raginsky. 2019. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv (may 2019). arXiv:1905.09883 http://arxiv.org/abs/1905.09883
  52. Data-driven forecasting of high-dimensional chaotic systems with long short-Term memory networks. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 474, 2213 (may 2018). https://doi.org/10.1098/rspa.2017.0844 arXiv:1802.07486
  53. Yufei Wang. 2020. Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning. Communications in Computational Physics 28, 5 (June 2020), 2158–2179. https://doi.org/10.4208/cicp.OA-2020-0194
  54. Kailiang Wu and Dongbin Xiu. 2020. Data-driven deep learning of partial differential equations in modal space. J. Comput. Phys. 408 (may 2020), 109307. https://doi.org/10.1016/j.jcp.2020.109307 arXiv:1910.06948
  55. Hao Xu. 2021. DL-PDE: Deep-Learning Based Data-Driven Discovery of Partial Differential Equations from Discrete and Noisy Data. Communications in Computational Physics 29, 3 (June 2021), 698–728. https://doi.org/10.4208/cicp.OA-2020-0142
  56. On Robustness of Neural Ordinary Differential Equations. arXiv:1910.05513 [cs, stat] (Nov. 2021). http://arxiv.org/abs/1910.05513 arXiv: 1910.05513.
  57. Haibo Yi. 2020. Efficient architecture for improving differential equations based on normal equation method in deep learning. Alexandria Engineering Journal 59, 4 (Aug. 2020), 2491–2502. https://doi.org/10.1016/j.aej.2020.04.014
  58. Towards real-time in-flight ice detection systems via computational aeroacoustics and machine learning. In AIAA Aviation 2019 Forum. 3103.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now