Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
143 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning invariant representations of time-homogeneous stochastic dynamical systems (2307.09912v3)

Published 19 Jul 2023 in cs.LG

Abstract: We consider the general class of time-homogeneous stochastic dynamical systems, both discrete and continuous, and study the problem of learning a representation of the state that faithfully captures its dynamics. This is instrumental to learning the transfer operator or the generator of the system, which in turn can be used for numerous tasks, such as forecasting and interpreting the system dynamics. We show that the search for a good representation can be cast as an optimization problem over neural networks. Our approach is supported by recent results in statistical learning theory, highlighting the role of approximation error and metric distortion in the learning problem. The objective function we propose is associated with projection operators from the representation space to the data space, overcomes metric distortion, and can be empirically estimated from data. In the discrete-time setting, we further derive a relaxed objective function that is differentiable and numerically well-conditioned. We compare our method against state-of-the-art approaches on different datasets, showing better performance across the board.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques. Physica D: Nonlinear Phenomena, 409:132520.
  2. Allen, E. (2007). Modeling with Itô stochastic differential equations, volume 22. Springer Science & Business Media.
  3. Computation of derivatives of repeated eigenvalues and the corresponding eigenvectors of symmetric matrix pencils. SIAM Journal on Matrix Analysis and Applications, 20(1):78–100.
  4. Deep canonical correlation analysis. In International Conference on Machine Learning, pages 1247–1255. PMLR.
  5. Arnold, L. (1974). Stochastic Differential Equations: Theory and Applications, volume 2. John Wiley & Sons.
  6. Aronszajn, N. (1950). Theory of reproducing kernels. Transactions of the American Mathematical Society, 68(3):337–404.
  7. Forecasting sequential data using consistent koopman autoencoders. In International Conference on Machine Learning, pages 475–485. PMLR.
  8. KoopmanizingFlows: Diffeomorphically Learning Stable Koopman Operators. arXiv preprint arXiv.2112.04085.
  9. Deep learning the slow modes for rare events sampling. Proceedings of the National Academy of Sciences, 118(44).
  10. Kernel Methods for the Approximation of Nonlinear Systems. SIAM Journal on Control and Optimization, 55(4):2460–2492.
  11. Modern Koopman Theory for Dynamical Systems. SIAM Review, 64(2):229–340.
  12. Mathematical Approach to Climate Change and Its Impacts: MAC2I, volume 38. Springer.
  13. Open catalyst 2020 (OC20) dataset and community challenges. ACS Catalysis, 11(10):6059–6072.
  14. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303–314.
  15. Koopman spectra in reproducing kernel Hilbert spaces. Applied and Computational Harmonic Analysis, 49(2):573–607.
  16. Learning Stable Koopman Embeddings. arXiv preprint arXiv.2110.06509.
  17. Latent representation and simulation of markov processes via time-lagged information bottleneck. In The Twelfth International Conference on Learning Representations.
  18. Data assimilation in weather forecasting: a case study in pde-constrained optimization. Optimization and Engineering, 10(3):409–426.
  19. Statistical consistency of kernel canonical correlation analysis. Journal of Machine Learning Research, 8(14):361–383.
  20. GraphVAMPNet, using graph neural networks and variational approach to markov processes for dynamical modeling of biomolecules. The Journal of Chemical Physics, 156(18):184103.
  21. The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM Journal on Numerical Analysis, 10(2):413–432.
  22. Matrix Computations. JHU press.
  23. Deep Learning. MIT press.
  24. Completely positive dynamical semigroups of N-level systems. Journal of Mathematical Physics, 17(5):821.
  25. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic Learning Theory: 16th International Conference, ALT 2005, Singapore, October 8-11, 2005. Proceedings 16, pages 63–77. Springer.
  26. Canonical correlation analysis: An overview with application to learning methods. Neural Computation, 16(12):2639–2664.
  27. Harold, H. (1936). Relations between two sets of variates. Biometrika, 28(3/4):321.
  28. Sparse learning of dynamical systems in rkhs: An operator-theoretic approach. ICML2023.
  29. Kato, T. (1976). Perturbation theory for linear operators; 2nd ed. Grundlehren der mathematischen Wissenschaften : a series of comprehensive studies in mathematics. Springer, Berlin.
  30. Kawahara, Y. (2016). Dynamic Mode Decomposition with Reproducing Kernels for Koopman Spectral Analysis. In Advances in Neural Information Processing Systems, volume 29.
  31. Kernel-based approximation of the koopman generator and schrödinger operator. Entropy, 22(7):722.
  32. Eigendecompositions of transfer operators in reproducing kernel Hilbert spaces. Journal of Nonlinear Science, 30(1):283–315.
  33. On Convergence of Extended Dynamic Mode Decomposition to the Koopman operator. Journal of Nonlinear Science, 28(2):687–710.
  34. Sharp spectral rates for koopman operator learning. Advances in Neural Information Processing Systems, 36.
  35. Learning dynamical systems via Koopman operator regression in reproducing kernel hilbert spaces. In Advances in Neural Information Processing Systems.
  36. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1–97.
  37. Kramers, H. (1940). Brownian motion in a field of force and the diffusion model of chemical reactions. Physica, 7(4):284–304.
  38. Dynamic Mode Decomposition. Society for Industrial and Applied Mathematics.
  39. Chaos, Fractals, and Noise, volume 97 of Applied Mathematical Sciences. Springer New York.
  40. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27(10):103111.
  41. Optimal rates for regularized conditional mean embedding learning. In Advances in Neural Information Processing Systems.
  42. Lindblad, G. (1976). On the generators of quantum dynamical semigroups. Communications in Mathematical Physics, 48(2):119–130.
  43. How fast-folding proteins fold. Science, 334(6055):517–520.
  44. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9(1).
  45. Deep learning markov and koopman models with physical constraints. In Mathematical and Scientific Machine Learning.
  46. Vampnets for deep learning of molecular kinetics. Nature Communications, 9(1):5.
  47. A variational conformational dynamics approach to the selection of collective variables in metadynamics. The Journal of Chemical Physics, 147(20):204109.
  48. Estimating koopman operators with sketching to provably learn large scale dynamical systems. Advances in Neural Information Processing Systems, 36.
  49. Minsker, S. (2017). On some extensions of Bernstein’s inequality for self-adjoint operators. Statistics & Probability Letters, 127:111–119.
  50. Deep dynamical modeling and control of unsteady fluid flows. Advances in Neural Information Processing Systems, 31.
  51. Characterizing metastable states with the help of machine learning. Journal of Chemical Theory and Computation, 18(9):5195–5202.
  52. Finite-data error bounds for koopman-based prediction and control. Journal of Nonlinear Science, 33(1):14.
  53. Dynamical entropy for systems with stochastic perturbation. Physical Review E, 62(2):2018–2029.
  54. Linearly recurrent autoencoder networks for learning dynamics. SIAM Journal on Applied Dynamical Systems, 18(1):558–593.
  55. Pascucci, A. (2011). PDE and Martingale Methods in Option Pricing. Springer Milan.
  56. Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems, 15(1):142–161.
  57. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481):1026–1030.
  58. Ross, S. M. (1995). Stochastic Processes. John Wiley & Sons.
  59. Schmid, P. J. (2010). Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 656:5–28.
  60. SchNetPack 2.0: A neural network toolbox for atomistic machine learning. The Journal of Chemical Physics, 158(14):144801.
  61. SchNetPack: A Deep Learning Toolbox For Atomistic Systems. Journal of Chemical Theory and Computation, 15(1):448–455.
  62. Transfer Operator Approach to Conformational Dynamics in Biomolecular Systems. In Ergodic Theory, Analysis, and Efficient Simulation of Dynamical Systems, pages 191–223. Springer Berlin Heidelberg.
  63. Modeling Molecular Kinetics with tICA and the Kernel Trick. Journal of Chemical Theory and Computation, 11(2):600–608.
  64. Support Vector Machines. Springer New York.
  65. Kernel embedding based variational approach for low-dimensional approximation of dynamical systems. Computational Methods in Applied Mathematics, 21(3):635–659.
  66. Spectra and Pseudospectra: The Behavior of Nonnormal Matrices and Operators. Princeton University Press.
  67. Tropp, J. A. (2012). User-friendly tail bounds for sums of random matrices. Technical report.
  68. Tuckerman, M. (2010). Statistical Mechanics: Theory and Molecular Simulation. Oxford university press.
  69. Generalization bounds for kernel canonical correlation analysis. Transactions on Machine Learning Research.
  70. A data–driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25(6):1307–1346.
  71. A kernel-based method for data-driven Koopman spectral analysis. Journal of Computational Dynamics, 2(2):247–265.
  72. Variational Approach for Learning Markov Processes from Time Series Data. Journal of Nonlinear Science, 30(1):23–66.
  73. Learning deep neural network representations for koopman operators of nonlinear dynamical systems. In 2019 American Control Conference (ACC). IEEE.
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now