Neural Koopman prior for data assimilation (2309.05317v3)
Abstract: With the increasing availability of large scale datasets, computational power and tools like automatic differentiation and expressive neural network architectures, sequential data are now often treated in a data-driven way, with a dynamical model trained from the observation data. While neural networks are often seen as uninterpretable black-box architectures, they can still benefit from physical priors on the data and from mathematical knowledge. In this paper, we use a neural network architecture which leverages the long-known Koopman operator theory to embed dynamical systems in latent spaces where their dynamics can be described linearly, enabling a number of appealing features. We introduce methods that enable to train such a model for long-term continuous reconstruction, even in difficult contexts where the data comes in irregularly-sampled time series. The potential for self-supervised learning is also demonstrated, as we show the promising use of trained dynamical models as priors for variational data assimilation techniques, with applications to e.g. time series interpolation and forecasting.
- M. Rußwurm, C. Pelletier, M. Zollner, S. Lefèvre, and M. Körner, “Breizhcrops: a time series dataset for crop type mapping,” ISPRS-International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. 43, pp. 1545–1551, 2020.
- G. Weikmann, C. Paris, and L. Bruzzone, “Timesen2crop: A million labeled samples dataset of Sentinel 2 image time series for crop-type classification,” IEEE J-STARS, vol. 14, pp. 4699–4708, 2021.
- X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, “Self-supervised learning: Generative or contrastive,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 1, pp. 857–876, 2021.
- C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in IEEE ICCV, 2015, pp. 1422–1430.
- S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” in ICLR, 2018.
- T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in ICML. PMLR, 2020, pp. 1597–1607.
- Y. Wang, C. Albrecht, N. A. A. Braham, L. Mou, and X. Zhu, “Self-supervised learning in remote sensing: A review,” IEEE GRSM, 2022.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” NeurIPS, vol. 33, pp. 1877–1901, 2020.
- R. A. Borsoi, T. Imbiriba, P. Closas, J. C. M. Bermudez, and C. Richard, “Kalman filtering and expectation maximization for multitemporal spectral unmixing,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2020.
- R. A. Borsoi, T. Imbiriba, and P. Closas, “Dynamical hyperspectral unmixing with variational recurrent neural networks,” IEEE TIP, 2023.
- J. M. Lewis and J. C. Derber, “The use of adjoint equations to solve a variational adjustment problem with advective constraints,” Tellus A, vol. 37, no. 4, pp. 309–322, 1985.
- M. Nonnenmacher and D. S. Greenberg, “Deep emulators for differentiation, forecasting, and parametrization in earth science simulators,” Journal of Advances in Modeling Earth Systems, vol. 13, no. 7, p. e2021MS002554, 2021.
- A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
- B. O. Koopman, “Hamiltonian systems and transformation in Hilbert space,” Proceedings of the National Academy of Sciences, vol. 17, no. 5, pp. 315–318, 1931.
- S. L. Brunton, M. Budišić, E. Kaiser, and J. N. Kutz, “Modern Koopman theory for dynamical systems,” arXiv preprint arXiv:2102.12086, 2021.
- S. L. Brunton, B. W. Brunton, J. L. Proctor, and J. N. Kutz, “Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control,” PloS one, vol. 11, no. 2, p. e0150171, 2016.
- A. Frion, L. Drumetz, M. Dalla Mura, G. Tochon, and A. Aïssa-El-Bey, “Leveraging neural Koopman operators to learn continuous representations of dynamical systems from scarce data,” in ICASSP. IEEE, 2023, pp. 1–5.
- P. J. Schmid, “Dynamic mode decomposition of numerical and experimental data,” Journal of fluid mechanics, vol. 656, pp. 5–28, 2010.
- C. W. Rowley, I. Mezić, S. Bagheri, P. Schlatter, and D. S. Henningson, “Spectral analysis of nonlinear flows,” Journal of fluid mechanics, vol. 641, pp. 115–127, 2009.
- K. K. Chen, J. H. Tu, and C. W. Rowley, “Variants of dynamic mode decomposition: boundary condition, Koopman, and Fourier analyses,” Journal of nonlinear science, vol. 22, pp. 887–915, 2012.
- J. H. Tu, C. W. Rowley, D. M. Luchtenburg, S. L. Brunton, and J. N. Kutz, “On dynamic mode decomposition: Theory and applications,” Journal of Computational Dynamics, vol. 1, no. 2, pp. 391–421, 2014.
- J. N. Kutz, X. Fu, and S. L. Brunton, “Multiresolution dynamic mode decomposition,” SIAM Journal on Applied Dynamical Systems, vol. 15, no. 2, pp. 713–735, 2016.
- M. O. Williams, I. G. Kevrekidis, and C. W. Rowley, “A data–driven approximation of the Koopman operator: Extending dynamic mode decomposition,” Journal of Nonlinear Science, vol. 25, pp. 1307–1346, 2015.
- S. Klus, F. Nüske, S. Peitz, J.-H. Niemann, C. Clementi, and C. Schütte, “Data-driven approximation of the koopman generator: Model reduction, system identification, and control,” Physica D: Nonlinear Phenomena, vol. 406, p. 132416, 2020.
- Y. Kawahara, “Dynamic mode decomposition with reproducing kernels for Koopman spectral analysis,” NeurIPS, vol. 29, 2016.
- V. Kostic, P. Novelli, A. Maurer, C. Ciliberto, L. Rosasco, and M. Pontil, “Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces,” NeurIPS, vol. 35, pp. 4017–4031, 2022.
- P. Bevanda, M. Beier, A. Lederer, S. Sosnowski, E. Hüllermeier, and S. Hirche, “Koopman kernel regression,” arXiv:2305.16215, 2023.
- Q. Li, F. Dietrich, E. M. Bollt, and I. G. Kevrekidis, “Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the Koopman operator,” Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 27, no. 10, p. 103111, 2017.
- E. Yeung, S. Kundu, and N. Hodas, “Learning deep neural network representations for Koopman operators of nonlinear dynamical systems,” in American Control Conference (ACC). IEEE, 2019, pp. 4832–4839.
- B. Lusch, J. N. Kutz, and S. L. Brunton, “Deep learning for universal linear embeddings of nonlinear dynamics,” Nature communications, vol. 9, no. 1, p. 4950, 2018.
- S. E. Otto and C. W. Rowley, “Linearly recurrent autoencoder networks for learning dynamics,” SIAM Journal on Applied Dynamical Systems, vol. 18, no. 1, pp. 558–593, 2019.
- J. Morton, A. Jameson, M. J. Kochenderfer, and F. Witherden, “Deep dynamical modeling and control of unsteady fluid flows,” NeurIPS, vol. 31, 2018.
- Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba, “Learning compositional Koopman operators for model-based control,” in ICLR, 2019.
- O. Azencot, N. B. Erichson, V. Lin, and M. Mahoney, “Forecasting sequential data using consistent Koopman autoencoders,” in ICML. PMLR, 2020, pp. 475–485.
- N. B. Erichson, M. Muehlebach, and M. W. Mahoney, “Physics-informed autoencoders for lyapunov-stable fluid flow prediction,” arXiv preprint arXiv:1905.10866, 2019.
- S. Pan and K. Duraisamy, “Physics-informed probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability,” SIAM Journal on Applied Dynamical Systems, vol. 19, no. 1, pp. 480–509, 2020.
- F. Fan, B. Yi, D. Rye, G. Shi, and I. R. Manchester, “Learning stable koopman embeddings,” in 2022 American Control Conference (ACC). IEEE, 2022, pp. 2742–2747.
- P. Bevanda, M. Beier, S. Kerz, A. Lederer, S. Sosnowski, and S. Hirche, “Diffeomorphically learning stable koopman operators,” IEEE Control Systems Letters, vol. 6, pp. 3427–3432, 2022.
- K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE CVPR, 2016, pp. 770–778.
- A. Saxe, J. McClelland, and S. Ganguli, “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,” in ICLR, 2014.
- D. Xie, J. Xiong, and S. Pu, “All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation,” in IEEE CVPR, 2017, pp. 6176–6185.
- A. Krogh and J. Hertz, “A simple weight decay can improve generalization,” NeurIPS, vol. 4, 1991.
- N. Bansal, X. Chen, and Z. Wang, “Can we gain more from orthogonality regularizations in training deep networks?” NeurIPS, vol. 31, 2018.
- I. M. Navon, “Data assimilation for numerical weather prediction: a review,” Data assimilation for atmospheric, oceanic and hydrologic applications, pp. 21–65, 2009.
- X. Jin, L. Kumar, Z. Li, H. Feng, X. Xu, G. Yang, and J. Wang, “A review of data assimilation of remote sensing and crop models,” European Journal of Agronomy, vol. 92, pp. 141–152, 2018.
- K. H. Cho, Y. Pachepsky, M. Ligaray, Y. Kwon, and K. H. Kim, “Data assimilation in surface water quality modeling: A review,” Water Research, vol. 186, p. 116307, 2020.
- Q. Xu, B. Li, R. E. McRoberts, Z. Li, and Z. Hou, “Harnessing data assimilation and spatial autocorrelation for forest inventory,” Remote Sensing of Environment, vol. 288, p. 113488, 2023.
- R. N. Bannister, “A review of operational methods of variational and ensemble-variational data assimilation,” Quarterly Journal of the Royal Meteorological Society, vol. 143, no. 703, pp. 607–633, 2017.
- R. Fablet, L. Drumetz, and F. Rousseau, “Joint learning of variational representations and solvers for inverse problems with partially-observed data,” arXiv:2006.03653, 2020.
- A. Farchi, P. Laloyaux, M. Bonavita, and M. Bocquet, “Using machine learning to correct model error in data assimilation and forecast applications,” Quarterly Journal of the Royal Meteorological Society, vol. 147, no. 739, pp. 3067–3084, 2021.
- S. Cheng et al., “Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review,” arXiv:2303.10462, 2023.
- F. Takens, “Detecting strange attractors in turbulence,” in Dynamical Systems and Turbulence: proceedings of a symposium held at the University of Warwick 1979/80. Springer, 2006, pp. 366–381.
- W. J. Culver, “On the existence and uniqueness of the real logarithm of a matrix,” Proceedings of the American Mathematical Society, vol. 17, no. 5, pp. 1146–1151, 1966.
- R. A. Willoughby, “Solutions of Ill-Posed Problems (A. N. Tikhonov and V. Y. Arsenin),” SIAM Review, vol. 21, no. 2, pp. 266–267, 1979.
- L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: nonlinear phenomena, vol. 60, no. 1-4, pp. 259–268, 1992.
- J. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks,” PNAS, vol. 114, no. 13, pp. 3521–3526, 2017.
- C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune BERT for text classification?” in Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18. Springer, 2019, pp. 194–206.
- B. R. Noack, K. Afanasiev, M. Morzyński, G. Tadmor, and F. Thiele, “A hierarchy of low-dimensional models for the transient and post-transient cylinder wake,” Journal of Fluid Mechanics, vol. 497, pp. 335–363, 2003.
- S. Jutz and M. Milagro-Pérez, “Copernicus: the european Earth observation programme,” Revista de Teledetección, no. 56, pp. V–XI, 2020.
- A. Frion, L. Drumetz, G. Tochon, M. D. Mura, and A. A. E. Bey, “Learning Sentinel-2 reflectance dynamics for data-driven assimilation and forecasting,” in EUSIPCO 2023, 2023, pp. 1–5.
- S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
- S. Le Clainche and J. M. Vega, “Higher order dynamic mode decomposition,” SIAM Journal on Applied Dynamical Systems, vol. 16, no. 2, pp. 882–925, 2017.