Probabilistic Forecasting with Stochastic Interpolants and Föllmer Processes (2403.13724v2)
Abstract: We propose a framework for probabilistic forecasting of dynamical systems based on generative modeling. Given observations of the system state over time, we formulate the forecasting problem as sampling from the conditional distribution of the future system state given its current state. To this end, we leverage the framework of stochastic interpolants, which facilitates the construction of a generative model between an arbitrary base distribution and the target. We design a fictitious, non-physical stochastic dynamics that takes as initial condition the current system state and produces as output a sample from the target conditional distribution in finite time and without bias. This process therefore maps a point mass centered at the current state onto a probabilistic ensemble of forecasts. We prove that the drift coefficient entering the stochastic differential equation (SDE) achieving this task is non-singular, and that it can be learned efficiently by square loss regression over the time-series data. We show that the drift and the diffusion coefficients of this SDE can be adjusted after training, and that a specific choice that minimizes the impact of the estimation error gives a F\"oLLMer process. We highlight the utility of our approach on several complex, high-dimensional forecasting problems, including stochastically forced Navier-Stokes and video prediction on the KTH and CLEVRER datasets.
- Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2022.
- Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023.
- Operator-theoretic framework for forecasting nonlinear time series with kernel analog techniques. Physica D: Nonlinear Phenomena, 409:132520, 2020.
- Nonparametric forecasting of low-dimensional dynamical systems. Physical Review E, 91(3):032915, 2015.
- Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
- Forecasting small-scale dynamics of fluid turbulence using deep neural networks. Proceedings of the National Academy of Sciences, 120(30), July 2023. URL https://www.pnas.org/doi/10.1073/pnas.2305765120. Publisher: Proceedings of the National Academy of Sciences.
- Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting. NeurIPS, 2023.
- Stochastic control liaisons: Richard sinkhorn meets gaspard monge on a schrodinger bridge. Siam Review, 63(2):249–313, 2021.
- Efficient video prediction via sparsely conditioned flow matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23263–23274, 2023.
- Normalizing kalman filters for multivariate time series analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2995–3007. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1f47cef5e38c952f94c5d61726027439-Paper.pdf.
- Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:17695–17709, 2021.
- On the approximation of complicated dynamical behavior. SIAM Journal on Numerical Analysis, 36(2):491–515, 1999.
- Learning to correct spectral methods for simulating turbulent flows. arXiv preprint arXiv:2207.00556, 2022.
- Regularization under diffusion and anticoncentration of the information content. Duke Mathematical Journal, 167(5):969–993, 2018.
- Stability of the logarithmic sobolev inequality via the follmer process. In Annales De L Institut Henri Poincare-Probabilites Et Statistiques, volume 56, pp. 2253–2269. Institute of Mathematical Statistics, 2020.
- Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873–12883, 2021.
- Unsupervised Learning for Physical Interaction through Video Prediction. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
- Föllmer, H. Time reversal on wiener space. Stochastic Processes—Mathematics and Physics, pp. 119–129, 1986.
- Learning to Forecast Dynamical Systems from Streaming Data. SIAM Journal on Applied Dynamical Systems, 22(2):527–558, June 2023. ISSN 1536-0040. doi: 10.1137/21M144983X. URL https://epubs.siam.org/doi/10.1137/21M144983X.
- Probabilistic forecasting. Annual Review of Statistics and Its Application, 1(1):125–151, 2014. doi: 10.1146/annurev-statistics-062713-085831.
- Calibrated probabilistic forecasting using ensemble model output statistics and minimum crps estimation. Monthly Weather Review, 133(5):1098 – 1118, 2005. doi: 10.1175/MWR2904.1.
- Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
- Ergodicity of the 2d navier-stokes equations with degenerate stochastic forcing. Annals of Mathematics, pp. 993–1032, 2006.
- Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
- Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
- Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
- Schrödinger-Föllmer sampler: sampling without ergodicity. arXiv preprint arXiv:2106.10880, 2021.
- Training neural operators to preserve invariant measures of chaotic attractors. arXiv preprint arXiv:2306.01187, 2023.
- Convergence analysis of Schrödinger-Föllmer sampler without convexity. arXiv preprint arXiv:2107.04766, 2021.
- Data-driven discovery of koopman eigenfunctions for control. Machine Learning: Science and Technology, 2(3):035023, 2021.
- Neural sdes as infinite-dimensional gans. In International conference on machine learning, pp. 5453–5463. PMLR, 2021.
- Dynamic mode decomposition: data-driven modeling of complex systems. SIAM, 2016.
- Stochastic Adversarial Video Prediction, April 2018. URL http://arxiv.org/abs/1804.01523. arXiv:1804.01523 [cs].
- Lehec, J. Representation formula for the entropy and functional inequalities. In Annales de l’IHP Probabilités et statistiques, volume 49, pp. 885–899, 2013.
- Léonard, C. A survey of the schrödinger problem and some of its connections with optimal transport. Discrete and Continuous Dynamical Systems-Series A, 34(4):1533–1574, 2014.
- Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
- Learning dissipative dynamics in chaotic systems. arXiv preprint arXiv:2106.06898, 2021.
- From zero to turbulence: Generative modeling for 3d flow simulation. In The Twelfth International Conference on Learning Representations, 2023.
- Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2194):20200209, February 2021. doi: 10.1098/rsta.2020.0209.
- Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2022.
- I2SBsuperscript𝐼2𝑆𝐵I^{2}SBitalic_I start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT italic_S italic_B: Image-to-image Schrödinger bridge. arXiv preprint arXiv:2302.05872, 2023.
- Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2022.
- Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
- Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021.
- Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:2401.08740, 2024.
- Machine learning advances for time series forecasting. Journal of Economic Surveys, 37(1):76–111, 2023. ISSN 1467-6419. doi: 10.1111/joes.12429.
- Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
- A Review on Deep Learning Techniques for Video Prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6):2806–2826, June 2022. ISSN 1939-3539. doi: 10.1109/TPAMI.2020.3045007.
- Ensemble prediction, 07/1992 1992. URL https://www.ecmwf.int/node/11560.
- Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, 2022.
- Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205, 2023.
- Peluchetti, S. Non-denoising forward-time diffusions. arXiv preprint arXiv:2312.14589, 2023.
- Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pp. 8857–8868. PMLR, 2021.
- High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684–10695, 2022.
- Schrödinger, E. Sur la théorie relativiste de l’électron et l’interprétation de la mécanique quantique. In Annales de l’institut Henri Poincaré, volume 3, pp. 269–310, 1932.
- Recognizing human actions: a local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pp. 32–36. IEEE, 2004.
- Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36, 2024.
- Smagorinsky, J. General circulation experiments with the primitive equations. Monthly Weather Review, 91(3):99 – 164, 1963. doi: 10.1175/1520-0493(1963)091¡0099:GCEWTP¿2.3.CO;2. URL https://journals.ametsoc.org/view/journals/mwre/91/3/1520-0493_1963_091_0099_gcewtp_2_3_co_2.xml.
- Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
- Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
- Theoretical guarantees for sampling and inference in generative models with latent diffusions. In Conference on Learning Theory, pp. 3084–3114. PMLR, 2019.
- Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018.
- Score-based generative modeling in latent space. arxiv, 2021.
- Bayesian learning via neural schrödinger–föllmer flows. Statistics and Computing, 33(1):3, 2023.
- Deep generative learning via schrödinger bridge. In International Conference on Machine Learning, pp. 10794–10804. PMLR, 2021.
- Robust approximation of the stochastic koopman operator. SIAM Journal on Applied Dynamical Systems, 21(3):1930–1951, 2022.
- Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
- Path integral sampler: A stochastic control approach for sampling. In International Conference on Learning Representations, 2021.
- Data-driven probability density forecast for stochastic dynamical systems. Journal of Computational Physics, 492:112422, 2023.