Papers
Topics
Authors
Recent
Search
2000 character limit reached

Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods

Published 20 Nov 2023 in cs.LG | (2311.11798v4)

Abstract: Data-driven modeling techniques have been explored in the spatial-temporal modeling of complex dynamical systems for many engineering applications. However, a systematic approach is still lacking to leverage the information from different types of data, e.g., with different spatial and temporal resolutions, and the combined use of short-term trajectories and long-term statistics. In this work, we build on the recent progress of neural operator and present a data-driven modeling framework called neural dynamical operator that is continuous in both space and time. A key feature of the neural dynamical operator is the resolution-invariance with respect to both spatial and temporal discretizations, without demanding abundant training data in different temporal resolutions. To improve the long-term performance of the calibrated model, we further propose a hybrid optimization scheme that leverages both gradient-based and derivative-free optimization methods and efficiently trains on both short-term time series and long-term statistics. We investigate the performance of the neural dynamical operator with three numerical examples, including the viscous Burgers' equation, the Navier-Stokes equations, and the Kuramoto-Sivashinsky equation. The results confirm the resolution-invariance of the proposed modeling framework and also demonstrate stable long-term simulations with only short-term time series data. In addition, we show that the proposed model can better predict long-term statistics via the hybrid optimization scheme with a combined use of short-term and long-term data.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. Adrian E Gill. Atmosphere-ocean dynamics, volume 30. Academic press, 1982.
  2. Harindra JS Fernando. Turbulent mixing in stratified fluids. Annual Review of Fluid Mechanics, 23(1):455–493, 1991.
  3. Geoffrey K Vallis. Atmospheric and oceanic fluid dynamics. Cambridge University Press, 2017.
  4. Paul E Dimotakis. Turbulent mixing. Annu. Rev. Fluid Mech., 37:329–356, 2005.
  5. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference on Robotics and Automation (ICRA), pages 9784–9790. IEEE, 2019.
  6. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.
  7. Dynamic mode decomposition: data-driven modeling of complex systems. SIAM, 2016.
  8. Physics-informed machine learning approach for reconstructing Reynolds stress modeling discrepancies based on DNS data. Physical Review Fluids, 2(3):034603, 2017.
  9. Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework. Physical Review Fluids, 3(7):074602, 2018.
  10. Turbulence modeling in the age of data. Annual Review of Fluid Mechanics, 51:357–377, 2019.
  11. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477–508, 2020.
  12. Learning stochastic closures using ensemble Kalman inversion. Transactions of Mathematics and Its Applications, 5(1):tnab003, 2021.
  13. Neural closure models for dynamical systems. Proceedings of the Royal Society A, 477(2252):20201004, 2021.
  14. Data-driven science and engineering: Machine learning, dynamical systems, and control. Cambridge University Press, 2022.
  15. A causality-based learning approach for discovering the underlying dynamics of complex systems from partial observations with stochastic parameterization. Physica D: Nonlinear Phenomena, 449:133743, 2023.
  16. CEBoosting: Online sparse identification of dynamical systems with regime switching by causation entropy boosting. arXiv preprint arXiv:2304.07863, 2023.
  17. Handwritten digit recognition with a back-propagation network. Advances in Neural Information Processing Systems, 2, 1989.
  18. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
  19. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6):801–806, 1993.
  20. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
  21. Herbert Jaeger. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13, 2001.
  22. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14(11):2531–2560, 2002.
  23. A baseline for global weather and climate simulations at 1 km resolution. Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002192, 2020.
  24. Large-eddy simulation of atmospheric boundary layer flow through wind turbines and wind farms. Journal of Wind Engineering and Industrial Aerodynamics, 99(4):154–168, 2011.
  25. Large-eddy simulation of a very large wind farm in a stable atmospheric boundary layer. Physics of Fluids, 23(6):065101, 2011.
  26. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
  27. Neural operator: Learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89):1–97, 2023.
  28. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
  29. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020.
  30. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
  31. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, 2021.
  32. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems, 34:24048–24062, 2021.
  33. Variationally mimetic operator networks. arXiv preprint arXiv:2209.12871, 2022.
  34. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
  35. Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problems. Journal of Computational Physics, 486:112104, 2023.
  36. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.
  37. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science Advances, 7(40):eabi8605, 2021.
  38. Derivate informed neural operator: An efficient framework for high-dimensional parametric derivative learning. arXiv preprint arXiv:2206.10745, 2022.
  39. Fourier neural operator with learned deformations for PDEs on general geometries. arXiv preprint arXiv:2207.05209, 2022.
  40. Learning the dynamical response of nonlinear non-autonomous dynamical systems with deep operator neural networks. Engineering Applications of Artificial Intelligence, 125:106689, 2023.
  41. In-context operator learning with data prompts for differential equation problems. Proceedings of the National Academy of Sciences, 120(39):e2310142120, 2023.
  42. Prose: Predicting operators and symbolic expressions using multimodal transformers. arXiv preprint arXiv:2309.16816, 2023.
  43. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.
  44. When neural ODEs meet neural operators. 2022.
  45. W. Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1–11, 2017.
  46. Time-series learning of latent-space dynamics for reduced-order model closure. Physica D: Nonlinear Phenomena, 405:132368, 2020.
  47. Turbulence forecasting via neural ODE. arXiv preprint arXiv:1911.05180, 2019.
  48. Data-driven reduced-order modeling of spatiotemporal chaos with neural ordinary differential equations. Chaos: An Interdisciplinary Journal of Nonlinear Science, 32(7):073110, 2022.
  49. Neural networks with physics-informed architectures and constraints for dynamical systems modeling. In Learning for Dynamics and Control Conference, pages 263–277. PMLR, 2022.
  50. Stabilized neural ordinary differential equations for long-time forecasting of dynamical systems. Journal of Computational Physics, 474:111838, 2023.
  51. Automatic differentiation in pytorch. 2017.
  52. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1–43, 2018.
  53. Charles C Margossian. A review of automatic differentiation and its efficient implementation. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 9(4):e1305, 2019.
  54. Least squares shadowing sensitivity analysis of chaotic limit cycle oscillations. Journal of Computational Physics, 267:210–224, 2014.
  55. Ensemble Kalman methods for inverse problems. Inverse Problems, 29(4):045001, 2013.
  56. Learning about structural errors in models of complex dynamical systems. arXiv preprint arXiv:2401.00035, 2023.
  57. Analysis of the ensemble Kalman filter for inverse problems. SIAM Journal on Numerical Analysis, 55(3):1264–1290, 2017.
  58. Ensemble Kalman inversion: mean-field limit and convergence analysis. Statistics and Computing, 31:1–21, 2021.
  59. Ensemble Kalman methods: A mean field perspective. arXiv preprint arXiv:2209.11371, 2022.
  60. Ensemble Kalman methods with constraints. Inverse Problems, 35(9):095007, 2019.
  61. Tikhonov regularization within ensemble Kalman inversion. SIAM Journal on Numerical Analysis, 58(2):1263–1294, 2020.
  62. Ensemble Kalman inversion for sparse learning of dynamical systems from time-averaged data. Journal of Computational Physics, 470:111559, 2022.
  63. Yoonsang Lee. lpsubscript𝑙𝑝l_{p}italic_l start_POSTSUBSCRIPT italic_p end_POSTSUBSCRIPT regularization for ensemble Kalman inversion. SIAM Journal on Scientific Computing, 43(5):A3417–A3437, 2021.
  64. Regularized ensemble Kalman methods for inverse problems. Journal of Computational Physics, 416:109517, 2020.
  65. Interacting Langevin diffusions: Gradient structure and ensemble Kalman sampler. SIAM Journal on Applied Dynamical Systems, 19(1):412–441, 2020.
  66. Iterated Kalman methodology for inverse problems. Journal of Computational Physics, 463:111262, 2022.
  67. Lucas Böttcher. Gradient-free training of neural ODEs for system identification and control using ensemble Kalman inversion. arXiv preprint arXiv:2307.07882, 2023.
  68. Geir Evensen. Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics. Journal of Geophysical Research: Oceans, 99(C5):10143–10162, 1994.
  69. Geir Evensen. The ensemble Kalman filter: Theoretical formulation and practical implementation. Ocean Dynamics, 53:343–367, 2003.
  70. Geir Evensen et al. Data assimilation: the ensemble Kalman filter, volume 2. Springer, 2009.
  71. Ensemble Kalman inversion: a derivative-free technique for machine learning tasks. Inverse Problems, 35(9):095005, 2019.
  72. Johannes Martinus Burgers. A mathematical model illustrating the theory of turbulence. Advances in Applied Mechanics, 1:171–199, 1948.
  73. Eberhard Hopf. The partial differential equation ut+u⁢ux=μx⁢xsubscript𝑢𝑡𝑢subscript𝑢𝑥subscript𝜇𝑥𝑥u_{t}+uu_{x}=\mu_{xx}italic_u start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT + italic_u italic_u start_POSTSUBSCRIPT italic_x end_POSTSUBSCRIPT = italic_μ start_POSTSUBSCRIPT italic_x italic_x end_POSTSUBSCRIPT. Communications on Pure and Applied mathematics, 3(3):201–230, 1950.
  74. Burgers turbulence. Physics Reports, 447(1-2):1–66, 2007.
  75. Dynamic scaling of growing interfaces. Physical Review Letters, 56(9):889, 1986.
  76. Invariant measures for Burgers equation with stochastic forcing. Annals of Mathematics, pages 877–960, 2000.
  77. Dirk Helbing. Traffic and related self-driven many-particle systems. Reviews of Modern Physics, 73(4):1067, 2001.
  78. Takashi Nagatani. The physics of traffic jams. Reports on Progress in Physics, 65(9):1331, 2002.
  79. Claude Navier. Mémoire sur les lois du mouvement des fluides. éditeur inconnu, 1822.
  80. Alexandre Joel Chorin. Numerical solution of the Navier-Stokes equations. Mathematics of Computation, 22(104):745–762, 1968.
  81. David J Acheson. Elementary fluid dynamics, 1991.
  82. Roger Temam. Navier-Stokes equations: theory and numerical analysis, volume 343. American Mathematical Soc., 2001.
  83. Persistent propagation of concentration waves in dissipative media far from thermal equilibrium. Progress of Theoretical Physics, 55(2):356–369, 1976.
  84. Gi Siv Ashinsky. Nonlinear analysis of hydrodynamic instability in laminar flames—I. Derivation of basic equations. In Dynamics of Curved Fronts, pages 459–488. Elsevier, 1988.
  85. The Kuramoto-Sivashinsky equation: a bridge between PDE’S and dynamical systems. Physica D: Nonlinear Phenomena, 18(1-3):113–126, 1986.
  86. Back in the saddle again: a computer assisted study of the Kuramoto–Sivashinsky equation. SIAM Journal on Applied Mathematics, 50(3):760–790, 1990.
  87. Divergence estimation for multidimensional densities via k𝑘kitalic_k-Nearest-Neighbor distances. IEEE Transactions on Information Theory, 55(5):2392–2405, 2009.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Authors (2)

  1. Chuanqi Chen 
  2. Jin-Long Wu 

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free