Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Residual Multi-Fidelity Neural Network Computing (2310.03572v3)

Published 5 Oct 2023 in cs.LG, cs.NA, and math.NA

Abstract: In this work, we consider the general problem of constructing a neural network surrogate model using multi-fidelity information. Motivated by error-complexity estimates for ReLU neural networks, we formulate the correlation between an inexpensive low-fidelity model and an expensive high-fidelity model as a possibly non-linear residual function. This function defines a mapping between 1) the shared input space of the models along with the low-fidelity model output, and 2) the discrepancy between the outputs of the two models. The computational framework proceeds by training two neural networks to work in concert. The first network learns the residual function on a small set of high- and low-fidelity data. Once trained, this network is used to generate additional synthetic high-fidelity data, which is used in the training of the second network. The trained second network then acts as our surrogate for the high-fidelity quantity of interest. We present four numerical examples to demonstrate the power of the proposed framework, showing that significant savings in computational cost may be achieved when the output predictions are desired to be accurate within small tolerances.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. The examination and analysis of residuals. Technometrics, 5:141–160, 1963.
  2. General multi-fidelity framework for training artificial neural networks with computational models. Frontiers in Materials, 6:1–14, 2019.
  3. A. R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39:930–945, 1993.
  4. Y. Bengio. Practical recommendations for gradient-based training of deep architectures. In Müller KR. Montavon G., Orr G.B., editor, Neural Networks: Tricks of the Trades, pages 437–478. Springer, Berlin, 2012.
  5. F. Chollet et al. Keras. https://keras.io, 2015.
  6. Multilevel Monte Carlo methods and applications to elliptic PDEs with random coefficients. Comput. Visual Sci., 14:3–15, 2011.
  7. On systems of differential equations with extrinsic oscillation. Discrete & Continuous Dynamical Systems - A, 28:1345–1367, 2010.
  8. Multi-fidelity surrogate modeling using long short-term memory networks. Computer methods in applied mechanics and engineering, 404:115811, 2023.
  9. Nonlinear approximation and (deep) relu networks. Constructive Approximation, 55(1):127–172, 2022.
  10. Approximation error and complexity bounds for ReLU networks on low regular function spaces. submitted, 2024.
  11. Deep neural network approximation theory. IEEE Transactions on Information Theory, 67(5):2581–2623, 2021.
  12. Review of multi-fidelity models. arXiv:1609.07196, 2016.
  13. G. S. Fishman. Monte Carlo: Concepts, Algorithms, and Applications. Springer- Verlag, New York, 1996.
  14. Multi-fidelity optimization via surrogate modelling. Proceedings of the royal society a: mathematical, physical and engineering sciences, 463(2088):3251–3269, 2007.
  15. Deep Learning. MIT Press, Cambridge, MA, USA, 2016.
  16. P. Grohs and L. Herrmann. Deep neural network approximation for high-dimensional elliptic pdes with boundary conditions. IMA Journal of Numerical Analysis, 42(3):2055–2082, 2022.
  17. A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. arXiv preprint arXiv:1809.02362, 2018.
  18. Error bounds for approximations with deep ReLU neural networks in Ws,psuperscript𝑊𝑠𝑝W^{s,p}italic_W start_POSTSUPERSCRIPT italic_s , italic_p end_POSTSUPERSCRIPT norms. Analysis and Applications, 18:803–859, 2020.
  19. Multi-fidelity regression using artificial neural networks: Efficient approximation of parameter-dependent output quantities. Computer methods in applied mechanics and engineering, 389:114378, 2022.
  20. Multi-index Monte Carlo: when sparsity meets sampling. Numerische Mathematik, 132:767–806, 2016.
  21. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  22. Multilayer feedforward networks are universal approximators. Journal Neural Networks, 2:359–366, 1989.
  23. Multifidelity deep operator networks for data-driven and physics-informed problems. Journal of Computational Physics, 493:112462, 2023.
  24. A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations. SN Partial Differential Equations and Applications volume, 1, 2020.
  25. A proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients. arxiv.org/abs/1809.07321, 2018.
  26. M. C. Kennedy and A. O’Hagan. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87:1–13, 2000.
  27. A theoretical analysis of deep neural networks and parametric pdes. Constructive Approximation, 55(1):73–125, 2022.
  28. D. Liu and Y. Wang. Multi-fidelity physics-constrained neural network and its application in materials modeling. Journal of Mechanical Design, 141:121403, 2019.
  29. Multi-fidelity bayesian neural networks: Algorithms and applications. Journal of Computational Physics, 438:110361, 2021.
  30. X. Meng and G.E. Karniadakis. A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems. Journal of Computational Physics, 401: 20160751, 2020.
  31. M. Motamed. A multi-fidelity neural network surrogate sampling method for uncertainty quantification. International Journal for Uncertainty Quantification, 10:315–332, 2020.
  32. M. Motamed. Approximation power of deep neural networks: an explanatory mathematical survey. arXiv:2207.095511, 2022.
  33. M. Motamed and D. Appelö. A multi-order discontinuous Galerkin Monte Carlo method for hyperbolic problems with stochastic parameters. SIAM J. Numer. Anal., 56:448–468, 2018.
  34. F. Nobile and F. Tesei. A multi level Monte Carlo method with control variate for elliptic PDEs with log-normal coefficients. Stoch PDE: Anal Comp, 3:398–444, 2015.
  35. Multifidelity data fusion in convolutional encoder/decoder networks. Journal of Computational Physics, 472:111666, 2023.
  36. A. Paszke et al. Pytorch: An imperative style, high-performance deep learning library, 2019.
  37. Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proc. R. Soc. A, 473: 20160751, 2017.
  38. P. Petersen and F. Voigtländer. Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Networks, 108:296–330, 2018.
  39. A. Pinkus. Approximation theory of the MLP model in neural networks. Acta Numer., 8:143–195, 1999.
  40. O. San and R. Maulik. Neural network closures for nonlinear model order reduction. Advances in Computational Mathematics, 44:1717–1750, 2018.
  41. J. Schmidhuber. Deep learning in neural networks: An overview. Neural netw., 61:85–117, 2015.
  42. J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339–1364, 2018.
  43. Transfer learning on multifidelity data. Journal of Machine Learning for Modeling and Computing, 3(1), 2022.
  44. Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning. Nonlinearity, 35(1):278, 2021.
  45. E. Weinan and B. Yu. The deep Ritz method: A deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat., 6:1–12, 2018.
  46. Integrating physics-based modeling with machine learning: A survey. arXiv:2003.04919, 2020.
  47. D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Owen Davis (4 papers)
  2. Mohammad Motamed (13 papers)
  3. Raul Tempone (61 papers)
Citations (1)
View on Semantic Scholar