Papers
Topics
Authors
Recent
Search
2000 character limit reached

Differentiable Multi-Fidelity Fusion: Efficient Learning of Physics Simulations with Neural Architecture Search and Transfer Learning

Published 12 Jun 2023 in cs.LG and cs.AI | (2306.06904v1)

Abstract: With rapid progress in deep learning, neural networks have been widely used in scientific research and engineering applications as surrogate models. Despite the great success of neural networks in fitting complex systems, two major challenges still remain: i) the lack of generalization on different problems/datasets, and ii) the demand for large amounts of simulation data that are computationally expensive. To resolve these challenges, we propose the differentiable \mf (DMF) model, which leverages neural architecture search (NAS) to automatically search the suitable model architecture for different problems, and transfer learning to transfer the learned knowledge from low-fidelity (fast but inaccurate) data to high-fidelity (slow but accurate) model. Novel and latest machine learning techniques such as hyperparameters search and alternate learning are used to improve the efficiency and robustness of DMF. As a result, DMF can efficiently learn the physics simulations with only a few high-fidelity training samples, and outperform the state-of-the-art methods with a significant margin (with up to 58%\% improvement in RMSE) based on a variety of synthetic and practical benchmark problems.

Authors (3)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. {{\{{TensorFlow}}\}}: a system for {{\{{Large-Scale}}\}} machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283, 2016.
  2. Theano: a cpu and gpu math expression compiler. In Proceedings of the Python for scientific computing conference (SciPy), volume 4, pages 1–7. Austin, TX, 2010.
  3. Souvik Chakraborty. Transfer learning based multi-fidelity physics informed deep neural network. Journal of Computational Physics, 426:109942, 2021.
  4. Numerical methods for engineers, volume 1221. Mcgraw-hill New York, 2011.
  5. Probing scattering resonances of vogel’s spirals with the green’s matrix spectral method. Optics letters, 41(9):1933–1936, 2016.
  6. T J Chung et al. Computational fluid dynamics. Cambridge university press, 2002.
  7. A statistical method for tuning a computer code to a data base. Computational statistics & data analysis, 37(1):77–92, 2001.
  8. Deep gaussian processes for multi-fidelity modeling. arXiv preprint arXiv:1903.07320, 2019.
  9. Andreas Damianou. Deep Gaussian processes and variational propagation of uncertainty. PhD thesis, University of Sheffield, 2015.
  10. Effective-medium theory for finite-size aggregates. JOSA A, 23(2):349–358, 2006.
  11. Multi-fidelity regression using artificial neural networks: efficient approximation of parameter-dependent output quantities. Computer methods in applied mechanics and engineering, 389:114378, 2022.
  12. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4340–4349, 2019.
  13. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  14. Extraction of material properties through multi-fidelity deep learning from molecular dynamics simulation. Computational Materials Science, 188:110187, 2021.
  15. Face detection with the faster r-cnn. In 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017), pages 650–657. IEEE, 2017.
  16. Gaussian process bandit optimisation with multi-fidelity evaluations. Advances in neural information processing systems, 29, 2016.
  17. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1–13, 2000.
  18. A hybrid multi-fidelity approach to the optimal design of warm forming processes using a knowledge-based artificial neural network. International Journal of Machine Tools and Manufacture, 47(2):211–222, 2007.
  19. Parallel architecture and hyperparameter search via successive halving and classification. arXiv preprint arXiv:1805.10255, 2018.
  20. June Gunn Lee. Computational materials science: an introduction. CRC press, 2016.
  21. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765–6816, 2017.
  22. Deep multi-fidelity active learning of high-dimensional outputs. arXiv preprint arXiv:2012.00901, 2020.
  23. Multi-fidelity bayesian optimization via deep neural networks. Advances in Neural Information Processing Systems, 33:8521–8531, 2020.
  24. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.
  25. Multi-fidelity bayesian neural networks: Algorithms and applications. Journal of Computational Physics, 438:110361, 2021.
  26. A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse pde problems. Journal of Computational Physics, 401:109020, 2020.
  27. Deep convolutional encoder-decoder networks for uncertainty quantification of dynamic multiphase flow in heterogeneous media. Water Resources Research, 55(1):703–728, 2019.
  28. Kai Nagel. Particle hopping models and traffic flow theory. Physical review E, 53(5):4655, 1996.
  29. Deep relu networks and high-order finite element methods. Analysis and Applications, 18(05):715–770, 2020.
  30. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009.
  31. Automatic differentiation in pytorch. 2017.
  32. Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2198):20160751, 2017.
  33. S Persides. The laplace and poisson equations in schwarzschild’s space-time. Journal of Mathematical Analysis and Applications, 43(3):571–578, 1973.
  34. Multi-information source optimization. Advances in neural information processing systems, 30, 2017.
  35. Physically informed artificial neural networks for atomistic modeling of materials. Nature communications, 10(1):1–10, 2019.
  36. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  37. Machine learning of linear differential equations using gaussian processes. Journal of Computational Physics, 348:683–693, 2017.
  38. Optimization of large-scale vogel spiral arrays of plasmonic nanoparticles. Plasmonics, 14(1):253–261, 2019.
  39. Transfer learning on multi-fidelity data. arXiv preprint arXiv:2105.00856, 2021.
  40. A general framework for multi-fidelity bayesian optimization with gaussian processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3158–3167. PMLR, 2019.
  41. Nobumasa Sugimoto. Burgers equation with a fractional derivative; hereditary effects on nonlinear acoustic waves. Journal of fluid mechanics, 225:631–653, 1991.
  42. Surrogate modeling of computer experiments with different mesh densities. Technometrics, 56(3):372–380, 2014.
  43. Gar: Generalized autoregression for multi-fidelity fusion. arXiv preprint arXiv:2301.05729, 2023.
  44. Reduced dimensional gaussian process emulators of parametrized partial differential equations based on isomap. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 471(2174):20140697, 2015.
  45. Deep coregionalization for the emulation of simulation-based spatial-temporal fields. Journal of Computational Physics, 428:109984, 2021.
  46. Information-based multi-fidelity bayesian optimization. In NIPS Workshop on Bayesian Optimization, 2017.
  47. Heterogeneous transfer learning for image classification. In Twenty-fifth aaai conference on artificial intelligence, 2011.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now