Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hutchinson Trace Estimation for High-Dimensional and High-Order Physics-Informed Neural Networks (2312.14499v2)

Published 22 Dec 2023 in cs.LG, cs.AI, cs.NA, math.DS, math.NA, and stat.ML

Abstract: Physics-Informed Neural Networks (PINNs) have proven effective in solving partial differential equations (PDEs), especially when some data are available by seamlessly blending data and physics. However, extending PINNs to high-dimensional and even high-order PDEs encounters significant challenges due to the computational cost associated with automatic differentiation in the residual loss. Herein, we address the limitations of PINNs in handling high-dimensional and high-order PDEs by introducing Hutchinson Trace Estimation (HTE). Starting with the second-order high-dimensional PDEs ubiquitous in scientific computing, HTE transforms the calculation of the entire Hessian matrix into a Hessian vector product (HVP). This approach alleviates the computational bottleneck via Taylor-mode automatic differentiation and significantly reduces memory consumption from the Hessian matrix to HVP. We further showcase HTE's convergence to the original PINN loss and its unbiased behavior under specific conditions. Comparisons with Stochastic Dimension Gradient Descent (SDGD) highlight the distinct advantages of HTE, particularly in scenarios with significant variance among dimensions. We further extend HTE to higher-order and higher-dimensional PDEs, specifically addressing the biharmonic equation. By employing tensor-vector products (TVP), HTE efficiently computes the colossal tensor associated with the fourth-order high-dimensional biharmonic equation, saving memory and enabling rapid computation. The effectiveness of HTE is illustrated through experimental setups, demonstrating comparable convergence rates with SDGD under memory and speed constraints. Additionally, HTE proves valuable in accelerating the Gradient-Enhanced PINN (gPINN) version as well as the Biharmonic equation. Overall, HTE opens up a new capability in scientific machine learning for tackling high-order and high-dimensional PDEs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Deep splitting method for parabolic pdes. SIAM Journal on Scientific Computing, 43(5):A3135–A3154, 2021.
  2. Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations. Journal of Nonlinear Science, 29:1563–1619, 2019.
  3. Overcoming the curse of dimensionality in the numerical approximation of high-dimensional semilinear elliptic partial differential equations. arXiv preprint arXiv:2003.00596, 2020.
  4. Overcoming the curse of dimensionality in the numerical approximation of allen–cahn partial differential equations via truncated full-history recursive multilevel picard approximations. Journal of Numerical Mathematics, 28(4):197–222, 2020.
  5. Numerical simulations for full history recursive multilevel picard approximations for systems of high-dimensional partial differential equations. arXiv preprint arXiv:2005.10206, 2020.
  6. Solving high-dimensional optimal stopping problems using deep learning. European Journal of Applied Mathematics, 32(3):470–514, 2021.
  7. Taylor-mode automatic differentiation for higher-order derivatives in JAX. In Program Transformations for ML Workshop at NeurIPS 2019, 2019.
  8. JAX: composable transformations of Python+NumPy programs, 2018.
  9. Physics-informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica, 37(12):1727–1738, 2021.
  10. Machine learning for semi linear pdes. Journal of scientific computing, 79(3):1667–1712, 2019.
  11. Solving inverse stochastic problems from discrete particle observations using the fokker–planck equation and physics-informed neural networks. SIAM Journal on Scientific Computing, 43(3):B811–B830, 2021.
  12. Separable pinn: Mitigating the curse of dimensionality in physics-informed neural networks. arXiv preprint arXiv:2211.08761, 2022.
  13. Overcoming the curse of dimensionality for some hamilton–jacobi partial differential equations via neural network architectures. Research in the Mathematical Sciences, 7:1–50, 2020.
  14. Algorithms for overcoming the curse of dimensionality for certain hamilton–jacobi equations arising in control theory and elsewhere. Research in the Mathematical Sciences, 3(1):19, 2016.
  15. A physics-informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379:113741, 2021.
  16. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  17. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in mathematics and statistics, 5(4):349–380, 2017.
  18. Learning physics-informed neural networks without stacked back-propagation. In International Conference on Artificial Intelligence and Statistics, pages 3034–3047. PMLR, 2023.
  19. Pierre Henry-Labordere. Deep primal-dual algorithm for bsdes: Applications of machine learning to cva and im. Available at SSRN 3071506, 2017.
  20. Augmented physics-informed neural networks (apinns): A gating network-based soft domain decomposition methodology. arXiv preprint arXiv:2211.08939, 2022.
  21. When do extended physics-informed neural networks (xpinns) improve generalization? SIAM Journal on Scientific Computing, 44(5):A3158–A3182, 2022.
  22. Tackling the curse of dimensionality with physics-informed neural networks. arXiv preprint arXiv:2307.12306, 2023.
  23. Bias-variance trade-off in physics-informed neural networks with randomized smoothing for high-dimensional pdes. arXiv preprint arXiv:2311.15283, 2023.
  24. Deep backward schemes for high-dimensional nonlinear pdes. Mathematics of Computation, 89(324):1547–1579, 2020.
  25. Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulation and Computation, 18(3):1059–1076, 1989.
  26. Overcoming the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations. Proceedings of the Royal Society A, 476(2244):20190630, 2020.
  27. Multilevel picard iterations for solving smooth semilinear parabolic heat equations. Partial Differential Equations and Applications, 2(6):1–31, 2021.
  28. Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communications in Computational Physics, 28(5):2002–2041, 2020.
  29. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404:109136, 2020.
  30. Deep learning of inverse water waves problems using multi-fidelity data: Application to serre–green–naghdi equations. Ocean Engineering, 248:110775, 2022.
  31. Three algorithms for solving high-dimensional fully coupled fbsdes through deep learning. IEEE Intelligent Systems, 35(3):71–84, 2020.
  32. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951, 2021.
  33. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021.
  34. Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems (NeurIPS), pages 586–594, 2016.
  35. How does information bottleneck help deep learning? arXiv preprint arXiv:2305.18887, 2023.
  36. Generalization in deep learning. Cambridge University Press, 2022.
  37. Adam: A method for stochastic optimization. ICLR, 2015.
  38. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021.
  39. Hutch++: Optimal stochastic trace estimation. In Symposium on Simplicity in Algorithms (SOSA), pages 142–155. SIAM, 2021.
  40. Estimates on the generalization error of physics informed neural networks (pinns) for approximating pdes. arXiv preprint arXiv:2006.16144, 2020.
  41. Randomized automatic differentiation. In International Conference on Learning Representations, 2021.
  42. Improved variants of the hutch++ algorithm for trace estimation. SIAM Journal on Matrix Analysis and Applications, 43(3):1162–1185, 2022.
  43. Meta-learning pinn loss functions. Journal of computational physics, 458:111121, 2022.
  44. Maziar Raissi. Forward-backward stochastic neural networks: Deep learning of high-dimensional partial differential equations. arXiv preprint arXiv:1804.07010, 2018.
  45. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
  46. Improved bounds on sample size for implicit matrix trace estimators. Foundations of Computational Mathematics, 15(5):1187–1212, 2015.
  47. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. arXiv preprint arXiv:2004.01806, 2020.
  48. Maciej Skorski. Modern analysis of hutchinson’s trace estimator. In 2021 55th Annual Conference on Information Sciences and Systems (CISS), pages 1–5. IEEE, 2021.
  49. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  50. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  51. Is $l^2$ physics informed loss always suitable for training physics informed neural network? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  52. Tensor neural network and its numerical integration. arXiv preprint arXiv:2207.02754, 2022.
  53. Solving schr\\\backslash\”{{\{{o}}\}} dinger equation using tensor neural network. arXiv preprint arXiv:2209.12572, 2022.
  54. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics, 394:136–152, 2019.
  55. Gradient-enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:114823, 2022.
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now