Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Operator Learning Lessens the Curse of Dimensionality for PDEs (2301.12227v3)

Published 28 Jan 2023 in cs.LG

Abstract: Deep neural networks (DNNs) have achieved remarkable success in numerous domains, and their application to PDE-related problems has been rapidly advancing. This paper provides an estimate for the generalization error of learning Lipschitz operators over Banach spaces using DNNs with applications to various PDE solution operators. The goal is to specify DNN width, depth, and the number of training samples needed to guarantee a certain testing error. Under mild assumptions on data distributions or operator structures, our analysis shows that deep operator learning can have a relaxed dependence on the discretization resolution of PDEs and, hence, lessen the curse of dimensionality in many PDE-related problems including elliptic equations, parabolic equations, and Burgers equations. Our results are also applied to give insights about discretization-invariance in operator learning.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Near-optimal learning of banach-valued, high-dimensional functions via deep neural networks. arXiv preprint arXiv:2211.12633, 2022.
  2. Neural network learning: Theoretical foundations, volume 9. cambridge university press Cambridge, 1999.
  3. Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930–945, 1993.
  4. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285–2301, 2019.
  5. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. The Annals of Statistics, 47(4):2261–2285, 2019.
  6. Deep generative models that solve pdes: Distributed computing for training large data-free models. In 2020 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC) and Workshop on Artificial Intelligence and Machine Learning for Scientific Applications (AI4S), pages 50–63. IEEE, 2020.
  7. Deepm&mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436:110296, 2021.
  8. Efficient approximation of deep relu networks for functions on low dimensional manifolds. Advances in neural information processing systems, 32, 2019.
  9. Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA, 11(4):1203–1253, 2022.
  10. Breaking the curse of dimensionality in sparse polynomial approximation of parametric pdes. Journal de Mathématiques Pures et Appliquées, 103(2):400–428, 2015.
  11. Relu nets adapt to intrinsic dimensionality beyond the target domain. arXiv preprint arXiv:2008.02545, 2020.
  12. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314, 1989.
  13. Convergence rates for learning linear operators from noisy data. arXiv preprint arXiv:2108.12515, 2021.
  14. Learning to synthesize: robust phase retrieval at low photon counts. Light: Science & Applications, 9(1):1–16, 2020.
  15. Lawrence C Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
  16. Deep neural networks for estimation and inference. Econometrica, 89(1):181–213, 2021.
  17. 2d & 3d shepp-logan phantom standards for mri. In 2008 19th International Conference on Systems Engineering, pages 521–526. IEEE, 2008.
  18. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 6645–6649. Ieee, 2013.
  19. Smooth solutions of the vector burgers equation in nonsmooth domains. Diferential and Integral Equations, 10(5):961–974, 1997.
  20. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2):251–257, 1991.
  21. A framework for data-driven solution and parameter estimation of pdes using conditional generative adversarial networks. Nature Computational Science, 1(12):819–829, 2021.
  22. Switchnet: a neural network model for forward and inverse scattering problems. SIAM Journal on Scientific Computing, 41(5):A3182–A3201, 2019.
  23. Adaptive regression estimation with multilayer feedforward neural networks. Nonparametric Statistics, 17(8):891–913, 2005.
  24. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art–No, 2021.
  25. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
  26. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
  27. l∞superscript𝑙l^{\infty}italic_l start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT-error bounds for multivariate lagrange approximation. SIAM Journal on Numerical Analysis, 11(2):363–381, 1974.
  28. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
  29. Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10):104118, 2021.
  30. Besov function approximation and binary classification on low-dimensional manifolds using convolutional residual networks. In International Conference on Machine Learning, pages 6770–6780. PMLR, 2021.
  31. Deep nonparametric estimation of operators between infinite dimensional spaces. arXiv preprint arXiv:2201.00217, 2022.
  32. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465–5506, 2021.
  33. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, 2021.
  34. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
  35. The barron space and the flow-induced function spaces for neural network models. Constructive Approximation, 55(1):369–406, 2022.
  36. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 19(6):1236–1246, 2018.
  37. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. J. Mach. Learn. Res., 21(174):1–38, 2020.
  38. The random feature model for input-output maps between banach spaces. SIAM Journal on Scientific Computing, 43(5):A3212–A3243, 2021.
  39. Finding the homology of submanifolds with high confidence from random samples. Discrete & Computational Geometry, 39(1):419–441, 2008.
  40. Integral autoencoder network for discretization-invariant learning. The Journal of Machine Learning Research, 23(1):12996–13040, 2022.
  41. Data-driven operator inference for nonintrusive projection-based model reduction. Computer Methods in Applied Mechanics and Engineering, 306:196–215, 2016.
  42. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nature Methods, 18(2):194–202, 2021.
  43. Generative adversarial neural operators. arXiv preprint arXiv:2205.03017, 2022.
  44. Johannes Schmidt-Hieber. Deep relu network approximation of functions on a manifold. arXiv preprint arXiv:1908.00695, 2019.
  45. Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with relu activation function. The Annals of Statistics, 48(4):1875–1897, 2020.
  46. Martin H Schultz. l∞superscript𝑙l^{\infty}italic_l start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT-multivariate approximation theory. SIAM Journal on Numerical Analysis, 6(2):161–183, 1969.
  47. Deep network approximation characterized by number of neurons. arXiv preprint arXiv:1906.05497, 2019.
  48. Deep network with approximation error being reciprocal of width to power of square root of depth. Neural Computation, 33(4):1005–1036, 2021.
  49. Charles J Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics, pages 1040–1053, 1982.
  50. Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint arXiv:1810.08033, 2018.
  51. Deep learning on image denoising: An overview. Neural Networks, 131:251–275, 2020.
  52. Generative diffusion learning for parametric partial differential equations. arXiv preprint arXiv:2305.14703, 2023.
  53. Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017.
  54. Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. In Conference on learning theory, pages 639–649. PMLR, 2018.
  55. Dmitry Yarotsky. Elementary superexpressive activations. In International Conference on Machine Learning, pages 11932–11940. PMLR, 2021.
Citations (12)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now