Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Smooth and Sparse Latent Dynamics in Operator Learning with Jerk Regularization (2402.15636v1)

Published 23 Feb 2024 in cs.LG, cs.CE, cs.NA, math-ph, math.MP, and math.NA

Abstract: Spatiotemporal modeling is critical for understanding complex systems across various scientific and engineering disciplines, but governing equations are often not fully known or computationally intractable due to inherent system complexity. Data-driven reduced-order models (ROMs) offer a promising approach for fast and accurate spatiotemporal forecasting by computing solutions in a compressed latent space. However, these models often neglect temporal correlations between consecutive snapshots when constructing the latent space, leading to suboptimal compression, jagged latent trajectories, and limited extrapolation ability over time. To address these issues, this paper introduces a continuous operator learning framework that incorporates jerk regularization into the learning of the compressed latent space. This jerk regularization promotes smoothness and sparsity of latent space dynamics, which not only yields enhanced accuracy and convergence speed but also helps identify intrinsic latent space coordinates. Consisting of an implicit neural representation (INR)-based autoencoder and a neural ODE latent dynamics model, the framework allows for inference at any desired spatial or temporal resolution. The effectiveness of this framework is demonstrated through a two-dimensional unsteady flow problem governed by the Navier-Stokes equations, highlighting its potential to expedite high-fidelity simulations in various scientific and engineering applications.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (68)
  1. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 619(7970):533–538, 2023.
  2. Machine learning for fluid mechanics. Annual review of fluid mechanics, 52:477–508, 2020.
  3. Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences. NPJ digital medicine, 2(1):115, 2019.
  4. Mechanistic artificial intelligence (mechanistic-ai) for modeling, design, and control of advanced manufacturing processes: Current state and perspectives. Journal of Materials Processing Technology, 302:117485, 2022.
  5. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15):3932–3937, 2016.
  6. Data-driven discovery of dimensionless numbers and governing laws from scarce measurements. Nature communications, 13(1):7562, 2022.
  7. Phycrnet: Physics-informed convolutional-recurrent network for solving spatiotemporal pdes. Computer Methods in Applied Mechanics and Engineering, 389:114399, 2022.
  8. Encoding physics to learn reaction-diffusion processes. Nature Machine Intelligence, pages 1–15, 2023.
  9. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
  10. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
  11. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021.
  12. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  13. Nomad: Nonlinear manifold decoders for operator learning. Advances in Neural Information Processing Systems, 35:5601–5613, 2022.
  14. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 404:115783, 2023.
  15. Learning operators with coupled attention. The Journal of Machine Learning Research, 23(1):9636–9698, 2022.
  16. Message passing neural pde solvers. arXiv preprint arXiv:2202.03376, 2022.
  17. Approximation of high-dimensional parametric pdes. Acta Numerica, 24:1–159, 2015.
  18. A survey of projection-based model reduction methods for parametric dynamical systems. SIAM review, 57(4):483–531, 2015.
  19. Model reduction for flow analysis and control. Annual Review of Fluid Mechanics, 49:387–417, 2017.
  20. Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020.
  21. Crom: Continuous reduced-order modeling of pdes using implicit neural representations. In The Eleventh International Conference on Learning Representations, 2022.
  22. Continuous pde dynamics forecasting with implicit neural representations. In The Eleventh International Conference on Learning Representations, 2023.
  23. Physics-informed neural ode (pinode): embedding physics into models using collocation points. Scientific Reports, 13(1):10166, 2023.
  24. Glasdi: Parametric physics-informed greedy latent space dynamics identification. Journal of Computational Physics, page 112267, 2023.
  25. Evolve smoothly, fit consistently: Learning smooth latent dynamics for advection-dominated systems. arXiv preprint arXiv:2301.10391, 2023.
  26. Operator learning with neural fields: Tackling pdes on general geometries. arXiv preprint arXiv:2306.07266, 2023.
  27. Arousal as a universal embedding for spatiotemporal brain dynamics. bioRxiv, pages 2023–11, 2023.
  28. Learning nonlinear projections for reduced-order modeling of dynamical systems using constrained autoencoders. Chaos: An Interdisciplinary Journal of Nonlinear Science, 33(11), 2023.
  29. A nonlinear-manifold reduced-order model and operator learning for partial differential equations with sharp solution gradients. Computer Methods in Applied Mechanics and Engineering, 419:116684, 2024.
  30. Bruce Moore. Principal component analysis in linear systems: Controllability, observability, and model reduction. IEEE transactions on automatic control, 26(1):17–32, 1981.
  31. Sivaguru S Ravindran. A reduced-order approach for optimal control of fluids using proper orthogonal decomposition. International journal for numerical methods in fluids, 34(5):425–448, 2000.
  32. Turbulence, coherent structures, dynamical systems and symmetry. Cambridge university press, 2012.
  33. Balanced model reduction via the proper orthogonal decomposition. AIAA journal, 40(11):2323–2330, 2002.
  34. Clarence W Rowley. Model reduction for fluids, using balanced proper orthogonal decomposition. International Journal of Bifurcation and Chaos, 15(03):997–1013, 2005.
  35. An overview of the proper generalized decomposition with applications in computational rheology. Journal of Non-Newtonian Fluid Mechanics, 166(11):578–592, 2011.
  36. Allan Pinkus. N-widths in Approximation Theory, volume 7. Springer Science & Business Media, 2012.
  37. Benjamin Peherstorfer. Breaking the kolmogorov barrier with nonlinear model reduction. Notices of the American Mathematical Society, 69(5):725–733, 2022.
  38. Neural-network-based approximations for solving partial differential equations. communications in Numerical Methods in Engineering, 10(3):195–201, 1994.
  39. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987–1000, 1998.
  40. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  41. Physics-informed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica, 37(12):1727–1738, 2021.
  42. Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, 2021.
  43. Optimal control of pdes using physics-informed neural networks. Journal of Computational Physics, 473:111731, 2023.
  44. Physics-informed convolutional neural networks for temperature field prediction of heat source layout without labeled data. Engineering Applications of Artificial Intelligence, 117:105516, 2023.
  45. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.
  46. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021.
  47. Physics-informed deep neural operator networks. In Machine Learning in Modeling and Simulation: Methods and Applications, pages 219–254. Springer, 2023.
  48. Physics informed wno. Computer Methods in Applied Mechanics and Engineering, 418:116546, 2024.
  49. Machine learning strategies for systems with invariance properties. Journal of Computational Physics, 318:22–35, 2016.
  50. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155–166, 2016.
  51. Scalars are universal: Equivariant machine learning, structured like classical physics. Advances in Neural Information Processing Systems, 34:28848–28863, 2021.
  52. Learning lie group symmetry transformations with neural networks. In Topological, Algebraic and Geometric Learning Workshops 2023, pages 50–59. PMLR, 2023.
  53. Symmetry-informed surrogates with data-free constraint for real-time acoustic wave propagation. Applied Acoustics, 214:109686, 2023.
  54. Learning physical models that can respect conservation laws. arXiv preprint arXiv:2302.11002, 2023.
  55. Trade-offs in the latent representation of microstructure evolution. Acta Materialia, 263:119514, 2024.
  56. Neural implicit flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal data. Journal of Machine Learning Research, 24(41):1–60, 2023.
  57. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616, 2022.
  58. Self-supervised learning with lie symmetries for partial differential equations. arXiv preprint arXiv:2307.05432, 2023.
  59. An unsupervised latent/output physics-informed convolutional-lstm network for solving partial differential equations using peridynamic differential operator. Computer Methods in Applied Mechanics and Engineering, 407:115944, 2023.
  60. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):22445–22451, 2019.
  61. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  62. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  63. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  64. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  65. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3–11, 2018.
  66. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  67. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462–7473, 2020.
  68. Machine learning refined: Foundations, algorithms, and applications. Cambridge University Press, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now