Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers (2306.12077v1)

Published 21 Jun 2023 in cs.LG and cs.AI

Abstract: We propose a method for learning dynamical systems from high-dimensional empirical data that combines variational autoencoders and (spatio-)temporal attention within a framework designed to enforce certain scientifically-motivated invariances. We focus on the setting in which data are available from multiple different instances of a system whose underlying dynamical model is entirely unknown at the outset. The approach rests on a separation into an instance-specific encoding (capturing initial conditions, constants etc.) and a latent dynamics model that is itself universal across all instances/realizations of the system. The separation is achieved in an automated, data-driven manner and only empirical data are required as inputs to the model. The approach allows effective inference of system behaviour at any continuous time but does not require an explicit neural ODE formulation, which makes it efficient and highly scalable. We study behaviour through simple theoretical analyses and extensive experiments on synthetic and real-world datasets. The latter investigate learning the dynamics of complex systems based on finite data and show that the proposed approach can outperform state-of-the-art neural-dynamical models. We study also more general inductive bias in the context of transfer to data obtained under entirely novel system interventions. Overall, our results provide a promising new framework for efficiently learning dynamical models from heterogeneous data with potential applications in a wide range of fields including physics, medicine, biology and engineering.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
  2. The lattice boltzmann equation: theory and applications. Physics Reports, 222(3):145–197, 1992.
  3. A model for collision processes in gases. i. small amplitude processes in charged and neutral one-component systems. Physical review, 94(3):511, 1954.
  4. Which priors matter? benchmarking models for learning latent dynamics. Advances in Neural Information Processing Systems, 34, 2021.
  5. High-speed tomographic PIV measurements in a DISI engine. Experiments in Fluids, 60(9):146, 2019.
  6. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932–3937, 2016.
  7. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):22445–22451, 2019.
  8. Learning neural event functions for ordinary differential equations. International Conference on Learning Representations, 2021.
  9. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.
  10. Learning quantum dynamics with latent neural ordinary differential equations. Physical Review A, 105(4):042403, 2022.
  11. Lagrangian neural networks. arXiv preprint arXiv:2003.04630, 2020.
  12. Hamiltonian-based Neural ODE Networks on the SE(3) Manifold For Dynamics Learning and Control. In Proceedings of Robotics: Science and Systems, 2021.
  13. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
  14. Weinan E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1–11, 2017.
  15. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983–1049, 2016.
  16. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference on machine learning, pages 3154–3164. PMLR, 2020.
  17. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. Advances in Neural Information Processing Systems, 33, 2020.
  18. Vortices enable the complex aerobatics of peregrine falcons. Communications biology, 1(1):27, 2018.
  19. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  20. Stable architectures for deep neural networks. Inverse Problems, 34(1):014004, 2017.
  21. Dieter Hänel. Molekulare Gasdynamik: Einführung in die kinetische Theorie der Gase und Lattice-Boltzmann-Methoden. Springer-Verlag, 2006.
  22. Learning unknown ODE models with Gaussian processes. In International Conference on Machine Learning, pages 1959–1968. PMLR, 2018.
  23. Latent neural ODEs with sparse bayesian multiple shooting. In The Eleventh International Conference on Learning Representations, 2023.
  24. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
  25. SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 476(2242), 2020.
  26. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33, 2020.
  27. Inferring latent dynamics underlying neural population activity via neural differential equations. In International Conference on Machine Learning, pages 5551–5561. PMLR, 2021.
  28. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  29. Generalization of deep recurrent optical flow estimation for particle-image velocimetry data. Measurement Science and Technology, 2022.
  30. Fourier neural operator for parametric partial differential equations. In The Eighth International Conference on Learning Representations, 2020.
  31. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  32. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3276–3285. PMLR, 2018.
  33. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2019.
  34. Differentiable multiple shooting layers. Advances in Neural Information Processing Systems, 34:16532–16544, 2021.
  35. How spanwise travelling transversal surface waves change the near-wall flow. Journal of Fluid Mechanics, 957:A30, 2023.
  36. Analysis of spatiotemporal inner-outer large-scale interactions in turbulent channel flow by multivariate empirical mode decomposition. Physical Review Fluids, 7(3):034603, 2022.
  37. Neural rough differential equations for long time series. In International Conference on Machine Learning, pages 7829–7838. PMLR, 2021.
  38. Pytorch: An imperative style, high-performance deep learning library, 2019.
  39. Lattice BGK models for Navier-Stokes equation. Europhysics letters, 17(6):479, 1992.
  40. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278–1286. PMLR, 2014.
  41. Neural Conservation Laws: A Divergence-Free Perspective. In Advances in Neural Information Processing Systems, volume 35, 2022.
  42. Latent ordinary differential equations for irregularly-sampled time series. In Advances in Neural Information Processing Systems, volume 32, 2019.
  43. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62:352–364, 2020.
  44. ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
  45. Continuous PDE Dynamics Forecasting with Implicit Neural Representations. In International Conference on Learning Representations, 2023.
  46. Learning efficient and robust ordinary differential equations via invertible neural networks. In International Conference on Machine Learning, pages 27060–27074. PMLR, 2022.
  47. Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control. In International Conference on Learning Representations, 2020.
  48. Benchmarking energy-conserving neural networks for learning dynamics from data. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, pages 1218–1229. PMLR, 2021.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now