Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Bias-Variance Trade-off in Physics-Informed Neural Networks with Randomized Smoothing for High-Dimensional PDEs (2311.15283v1)

Published 26 Nov 2023 in cs.LG, cs.AI, cs.NA, math.DS, math.NA, and stat.ML

Abstract: While physics-informed neural networks (PINNs) have been proven effective for low-dimensional partial differential equations (PDEs), the computational cost remains a hurdle in high-dimensional scenarios. This is particularly pronounced when computing high-order and high-dimensional derivatives in the physics-informed loss. Randomized Smoothing PINN (RS-PINN) introduces Gaussian noise for stochastic smoothing of the original neural net model, enabling Monte Carlo methods for derivative approximation, eliminating the need for costly auto-differentiation. Despite its computational efficiency in high dimensions, RS-PINN introduces biases in both loss and gradients, negatively impacting convergence, especially when coupled with stochastic gradient descent (SGD). We present a comprehensive analysis of biases in RS-PINN, attributing them to the nonlinearity of the Mean Squared Error (MSE) loss and the PDE nonlinearity. We propose tailored bias correction techniques based on the order of PDE nonlinearity. The unbiased RS-PINN allows for a detailed examination of its pros and cons compared to the biased version. Specifically, the biased version has a lower variance and runs faster than the unbiased version, but it is less accurate due to the bias. To optimize the bias-variance trade-off, we combine the two approaches in a hybrid method that balances the rapid convergence of the biased version with the high accuracy of the unbiased version. In addition, we present an enhanced implementation of RS-PINN. Extensive experiments on diverse high-dimensional PDEs, including Fokker-Planck, HJB, viscous Burgers', Allen-Cahn, and Sine-Gordon equations, illustrate the bias-variance trade-off and highlight the effectiveness of the hybrid RS-PINN. Empirical guidelines are provided for selecting biased, unbiased, or hybrid versions, depending on the dimensionality and nonlinearity of the specific PDE problem.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (39)
  1. Deep splitting method for parabolic pdes. SIAM Journal on Scientific Computing, 43(5):A3135–A3154, 2021.
  2. Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations. Journal of Nonlinear Science, 29:1563–1619, 2019.
  3. Overcoming the curse of dimensionality in the numerical approximation of high-dimensional semilinear elliptic partial differential equations. arXiv preprint arXiv:2003.00596, 2020.
  4. Overcoming the curse of dimensionality in the numerical approximation of allen–cahn partial differential equations via truncated full-history recursive multilevel picard approximations. Journal of Numerical Mathematics, 28(4):197–222, 2020.
  5. Numerical simulations for full history recursive multilevel picard approximations for systems of high-dimensional partial differential equations. arXiv preprint arXiv:2005.10206, 2020.
  6. Taylor-mode automatic differentiation for higher-order derivatives in JAX. In Program Transformations for ML Workshop at NeurIPS 2019, 2019.
  7. Machine learning for semi linear pdes. Journal of scientific computing, 79(3):1667–1712, 2019.
  8. Can-pinn: A fast physics-informed neural network based on coupled-automatic–numerical differentiation method. Computer Methods in Applied Mechanics and Engineering, 395:114909, 2022.
  9. Separable pinn: Mitigating the curse of dimensionality in physics-informed neural networks. arXiv preprint arXiv:2211.08761, 2022.
  10. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1310–1320. PMLR, 09–15 Jun 2019.
  11. Does federated learning really need backpropagation? arXiv preprint arXiv:2301.12195, 2023.
  12. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
  13. Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in mathematics and statistics, 5(4):349–380, 2017.
  14. Learning physics-informed neural networks without stacked back-propagation. In International Conference on Artificial Intelligence and Statistics, pages 3034–3047. PMLR, 2023.
  15. Pierre Henry-Labordere. Deep primal-dual algorithm for bsdes: Applications of machine learning to cva and im. Available at SSRN 3071506, 2017.
  16. When do extended physics-informed neural networks (xpinns) improve generalization? SIAM Journal on Scientific Computing, 44(5):A3158–A3182, 2022.
  17. Tackling the curse of dimensionality with physics-informed neural networks. arXiv preprint arXiv:2307.12306, 2023.
  18. Deep backward schemes for high-dimensional nonlinear pdes. Mathematics of Computation, 89(324):1547–1579, 2020.
  19. Overcoming the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations. Proceedings of the Royal Society A, 476(2244):20190630, 2020.
  20. Multilevel picard iterations for solving smooth semilinear parabolic heat equations. Partial Differential Equations and Applications, 2(6):1–31, 2021.
  21. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404:109136, 2020.
  22. Three algorithms for solving high-dimensional fully coupled fbsdes through deep learning. IEEE Intelligent Systems, 35(3):71–84, 2020.
  23. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021.
  24. Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems (NeurIPS), pages 586–594, 2016.
  25. How does information bottleneck help deep learning? In International Conference on Machine Learning (ICML), 2023.
  26. Generalization in deep learning. Cambridge University Press, 2022.
  27. Adam: A method for stochastic optimization. ICLR, 2015.
  28. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), pages 656–672. IEEE, 2019.
  29. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1105–B1132, 2021.
  30. A hybrid physics-informed neural network for nonlinear partial differential equation. arXiv preprint arXiv:2112.01696, 2021.
  31. Estimates on the generalization error of physics informed neural networks (pinns) for approximating pdes. arXiv preprint arXiv:2006.16144, 2020.
  32. fpinns: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41(4):A2603–A2626, 2019.
  33. Maziar Raissi. Forward-backward stochastic neural networks: Deep learning of high-dimensional partial differential equations. arXiv preprint arXiv:1804.07010, 2018.
  34. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
  35. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. arXiv preprint arXiv:2004.01806, 2020.
  36. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339–1364, 2018.
  37. Is $l^2$ physics informed loss always suitable for training physics informed neural network? In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  38. Tensor neural network and its numerical integration. arXiv preprint arXiv:2207.02754, 2022.
  39. Solving schr\\\backslash\”{{\{{o}}\}} dinger equation using tensor neural network. arXiv preprint arXiv:2209.12572, 2022.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now