Particle-based algorithm for stochastic optimal control (2311.06906v4)
Abstract: The solution to a stochastic optimal control problem can be determined by computing the value function from a discretization of the associated Hamilton-Jacobi-BeLLMan equation. Alternatively, the problem can be reformulated in terms of a pair of forward-backward SDEs, which makes Monte-Carlo techniques applicable. More recently, the problem has also been viewed from the perspective of forward and reverse time SDEs and their associated Fokker-Planck equations. This approach is closely related to techniques used in diffusion-based generative models. Forward and reverse time formulations express the value function as the ratio of two probability density functions; one stemming from a forward McKean-Vlasov SDE and another one from a reverse McKean-Vlasov SDE. In this paper, we extend this approach to a more general class of stochastic optimal control problems and combine it with ensemble Kalman filter type and diffusion map approximation techniques in order to obtain efficient and robust particle-based algorithms.
- Ensemble transform Kalman-Bucy filters. Q.J.R. Meteor. Soc., 140:995–1004, 2014.
- Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes Applications, 12:313–326, 1982.
- An optimal control perspective on diffusion-based generative modeling. preprint arXiv:2211.01364, 2023.
- Ensemble Kalman methods: A mean field perspective. preprint arXiv:2209.11371, 2022.
- René Carmona. Lectures on BSDEs, Stochastic Control, and Stochastic Differential Games with Financial Applications. SIAM, Philadelphia, 2016.
- Numerical methods for backward stochastic differential equations: A survey. preprint arXiv:2101.08936, 2021.
- Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the National Academy of Sciences, 102(21):7426–7431, 2005.
- Diffusion maps. Applied and Computational Harmonic Analysis, 21(1):5–30, 2006. Special Issue: Diffusion Maps and Wavelets.
- Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communications in Mathematics and Statistics, 5:349–380, 2017.
- Algorithms for solving high-dimensional PDEs: From nonlinear Monte Carlo to machine learning. Nonlinearity, 35:278, 2021.
- Data Assimilation Fundamentals: A unified Formulation of the State and Parameter Estimation Problem. Springer Nature Switzerland AG, Cham, Switzerland, 2022.
- Stable generative modeling using diffusion maps. preprint arXiv:2401.04372, 2024.
- Controlled interacting particle algorithms for simulation-based reinforcement learning. Systems & Control Letters, 170:105392, 2022.
- Numerical methods for stochastic differential equations. Springer, New York, 1991.
- Deterministic particle flows for constraining stochastic nonlinear systems. Phys. Rev. Res., 4:043035, 2022.
- Interacting particle solutions of Fokker–Planck equations through gradient-log-density estimation. Entropy, 22(8), 2020.
- Sean Meyn. Control Systems and Reinforcement Learning. Cambridge University Press, Cambridge, 2022.
- Grigorios A. Pavliotis. Stochastic Processes and Applications. Springer Verlag, New York, 2016.
- Sebastian Reich. A dynamical systems framework for intermittent data assimilation. BIT Numerical Mathematics, 51(1):235–249, 2011.
- Sebastian Reich. Data assimilation: The Schrödinger perspective. Acta Numerica, 28:635–711, 2019.
- Sebastian Reich. Data assimilation: A dynamic homotopy-based coupling approach. In Bertrand Chapron, Dan Crisan, Darryl Holm, Etienne Mémin, and Anna Radomska, editors, Stochastic Transport in Upper Ocean Dynamics II, pages 261–280, Cham, 2024. Springer Nature Switzerland.
- Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
- Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=PxTIG12RRHS.
- C.L. Wormell and S. Reich. Spectral convergence of diffusion maps: Improved error bounds and an alternative normalisation. SIAM J. Numer. Anal., 59:1687–1734, 2021.