Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow (2206.03198v1)

Published 7 Jun 2022 in physics.flu-dyn, cs.LG, and physics.comp-ph

Abstract: Transfer learning (TL) is becoming a powerful tool in scientific applications of neural networks (NNs), such as weather/climate prediction and turbulence modeling. TL enables out-of-distribution generalization (e.g., extrapolation in parameters) and effective blending of disparate training sets (e.g., simulations and observations). In TL, selected layers of a NN, already trained for a base system, are re-trained using a small dataset from a target system. For effective TL, we need to know 1) what are the best layers to re-train? and 2) what physics are learned during TL? Here, we present novel analyses and a new framework to address (1)-(2) for a broad range of multi-scale, nonlinear systems. Our approach combines spectral analyses of the systems' data with spectral analyses of convolutional NN's activations and kernels, explaining the inner-workings of TL in terms of the system's nonlinear physics. Using subgrid-scale modeling of several setups of 2D turbulence as test cases, we show that the learned kernels are combinations of low-, band-, and high-pass filters, and that TL learns new filters whose nature is consistent with the spectral differences of base and target systems. We also find the shallowest layers are the best to re-train in these cases, which is against the common wisdom guiding TL in machine learning literature. Our framework identifies the best layer(s) to re-train beforehand, based on physics and NN theory. Together, these analyses explain the physics learned in TL and provide a framework to guide TL for wide-ranging applications in science and engineering, such as climate change modeling.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Earth system modeling 2.0: A blueprint for models that learn from observations and targeted high-resolution simulations. Geophysical Research Letters, 44(24):12–396, 2017.
  2. Prognostic validation of a neural network unified physics parameterization. Geophysical Research Letters, 45(12):6289–6298, 2018.
  3. Deep learning to represent subgrid processes in climate models. Proceedings of the National Academy of Sciences, 115(39):9684–9689, 2018.
  4. Applications of deep learning to ocean data inference and subgrid parameterization. Journal of Advances in Modeling Earth Systems, 11(1):376–399, 2019.
  5. Deep neural networks for data-driven LES closure models. Journal of Computational Physics, 398:108910, 2019.
  6. Deep learning for multi-year ENSO forecasts. Nature, 573(7775):568–572, September 2019.
  7. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  8. Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere. Journal of Advances in Modeling Earth Systems, 12(9):e2020MS002109, 2020.
  9. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477–508, 2020.
  10. Stable machine-learning parameterization of subgrid processes for climate modeling at a range of resolutions. Nature communications, 11(1):1–10, 2020.
  11. Machine learning–accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021.
  12. Automating turbulence modelling by multi-agent reinforcement learning. Nature Machine Intelligence, 3(1):87–96, 2021.
  13. FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  14. How transferable are features in deep neural networks? arXiv:1411.1792 [cs], November 2014. arXiv: 1411.1792.
  15. Understanding the failure modes of out-of-distribution generalization. arXiv preprint arXiv:2010.15775, 2020.
  16. Data-driven super-parameterization using deep learning: Experimentation with multiscale Lorenz 96 systems and transfer learning. Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002084, 2020.
  17. Enforcing analytic constraints in neural networks emulating physical systems. Physical Review Letters, 126(9):098302, 2021.
  18. Data-driven subgrid-scale modeling of forced Burgers turbulence using deep learning with generalization to higher Reynolds numbers via transfer learning. Physics of Fluids, 33(3):031702, March 2021.
  19. Interpretable data-driven methods for subgrid-scale closure in LES for transcritical LOX/GCH4 combustion. Combustion and Flame, page 111758, 2021.
  20. Physical invariance in neural networks for subgrid-scale scalar flux modeling. Physical Review Fluids, 6(2):024607, 2021.
  21. Turbulence closure modeling with data-driven techniques: physical compatibility and consistency considerations. New Journal of Physics, 22(9):093023, 2020.
  22. Stable a posteriori LES of 2D turbulence using convolutional neural networks: Backscattering analysis and generalization to higher Re via transfer learning. Journal of Computational Physics (in press), February 2021. arXiv: 2102.11400.
  23. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270–279. Springer, 2018.
  24. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43–76, 2020.
  25. Transfer learning for nonlinear dynamics and its application to fluid turbulence. Physical Review E, 102(4):043301, 2020.
  26. Convolutional-network models to predict wall-bounded turbulence from wall quantities. Journal of Fluid Mechanics, 928, 2021.
  27. High-fidelity reconstruction of turbulent flow from spatially limited data using enhanced super-resolution generative adversarial network. Physics of Fluids, 33(12):125119, 2021.
  28. Data-driven medium-range weather prediction with a ResNet pretrained on climate simulations: A new model for weatherbench. Journal of Advances in Modeling Earth Systems, 13(2):e2020MS002405, 2021.
  29. Transfer learning of deep neural networks for predicting thermoacoustic instabilities in combustion systems. Energy and AI, 5:100085, September 2021.
  30. Long-term stability and generalization of observationally-constrained stochastic data-driven models for geophysical turbulence. arXiv preprint arXiv:2205.04601, 2022.
  31. Deep residual convolutional neural network combining dropout and transfer learning for enso forecasting. Geophysical Research Letters, 48(24):e2021GL093531, 2021.
  32. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021.
  33. Souvik Chakraborty. Transfer learning based multi-fidelity physics informed deep neural network. Journal of Computational Physics, 426:109942, 2021.
  34. A study on cnn transfer learning for image classification. In UK Workshop on computational Intelligence, pages 191–202. Springer, 2018.
  35. Application of deep transfer learning for automated brain abnormality classification using mr images. Cognitive Systems Research, 54:176–188, 2019.
  36. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014.
  37. Subgrid modelling for two-dimensional turbulence using neural networks. Journal of Fluid Mechanics, 858:122–144, 2019.
  38. Revealing the state space of turbulence using machine learning. Physical Review Fluids, 6(3):034402, 2021.
  39. Frame invariant neural network closures for kraichnan turbulence. arXiv preprint arXiv:2201.02928, 2022.
  40. Learning physics-constrained subgrid-scale closures in the small-data regime for stable and accurate les. arXiv preprint arXiv:2201.07347, 2022.
  41. What is being transferred in transfer learning? arXiv:2008.11687 [cs, stat], January 2021. arXiv: 2008.11687.
  42. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
  43. Visualizing the Loss Landscape of Neural Nets. arXiv:1712.09913 [cs, stat], November 2018. arXiv: 1712.09913.
  44. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34, 2021.
  45. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301–5310. PMLR, 2019.
  46. Andrey Nikolaevich Kolmogorov. The local structure of turbulence in incompressible viscous fluid for very large Reynolds numbers. Cr Acad. Sci. URSS, 30:301–305, 1941.
  47. Stephen B Pope. Turbulent Flows. IOP Publishing, 2001.
  48. Adaptive wavelet distillation from neural networks through interpretations. Advances in Neural Information Processing Systems, 34, 2021.
  49. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203, 2013.
  50. Climate-invariant machine learning. arXiv preprint arXiv:2112.08440, 2021.
  51. Physics-informed machine learning: case studies for weather and climate modelling. Philosophical Transactions of the Royal Society A, 379(2194):20200093, 2021.
  52. Pierre Sagaut. Large eddy simulation for incompressible flows: An introduction. Springer Science & Business Media, 2006.
  53. Data-driven equation discovery of ocean mesoscale closures. Geophysical Research Letters, 47(17):e2020GL088376, 2020.
  54. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851, 2013.
  55. Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks. arXiv preprint arXiv:2205.02902, 2022.
  56. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581–590. IEEE, 2020.
Citations (35)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now