Time-Reversal of Stochastic Maximum Principle (2403.02044v1)
Abstract: Stochastic maximum principle (SMP) specifies a necessary condition for the solution of a stochastic optimal control problem. The condition involves a coupled system of forward and backward stochastic differential equations (FBSDE) for the state and the adjoint processes. Numerical solution of the FBSDE is challenging because the boundary condition of the adjoint process is specified at the terminal time, while the solution should be adaptable to the forward in time filtration of a Wiener process. In this paper, a "time-reversal" of the FBSDE system is proposed that involves integration with respect to a backward in time Wiener process. The time-reversal is used to propose an iterative Monte-Carlo procedure to solves the FBSDE system and its time-reversal simultaneously. The procedure involves approximating the {F\"oLLMer's drift} and solving a regression problem between the state and its adjoint at each time. The procedure is illustrated for the linear quadratic (LQ) optimal control problem with a numerical example.
- S. Peng, “Probabilistic interpretation for systems of quasilinear parabolic partial differential equations,” Stochastics and stochastics reports (Print), vol. 37, no. 1-2, pp. 61–74, 1991.
- J. Ma, P. Protter, and J. Yong, “Solving forward-backward stochastic differential equations explicitly—a four step scheme,” Probability theory and related fields, vol. 98, no. 3, pp. 339–359, 1994.
- J. Ma, P. Protter, J. San Martín, and S. Torres, “Numerical method for backward stochastic differential equations,” Annals of Applied Probability, pp. 302–316, 2002.
- J. Zhang, “A numerical scheme for bsdes,” The annals of applied probability, vol. 14, no. 1, pp. 459–488, 2004.
- B. Bouchard and N. Touzi, “Discrete-time approximation and monte-carlo simulation of backward stochastic differential equations,” Stochastic Processes and their applications, vol. 111, no. 2, pp. 175–206, 2004.
- W. Zhao, L. Chen, and S. Peng, “A new kind of accurate numerical method for backward stochastic differential equations,” SIAM Journal on Scientific Computing, vol. 28, no. 4, pp. 1563–1581, 2006.
- C. Bender and R. Denk, “A forward scheme for backward sdes,” Stochastic processes and their applications, vol. 117, no. 12, pp. 1793–1812, 2007.
- G. Zhang, M. Gunzburger, and W. Zhao, “A sparse-grid method for multi-dimensional backward stochastic differential equations,” Journal of Computational Mathematics, pp. 221–248, 2013.
- B. D. Anderson, “Reverse-time diffusion equation models,” Stochastic Processes and their Applications, vol. 12, no. 3, pp. 313–326, 1982.
- U. G. Haussmann and E. Pardoux, “Time reversal of diffusions,” The Annals of Probability, pp. 1188–1205, 1986.
- A. Millet, D. Nualart, and M. Sanz, “Integration by parts and time reversal for diffusion processes,” The Annals of Probability, pp. 208–238, 1989.
- H. Föllmer, “An entropy approach to the time reversal of diffusion processes,” in Stochastic Differential Systems Filtering and Control: Proceedings of the IFIP-WG 7/1 Working Conference Marseille-Luminy, France, March 12–17, 1984. Springer, 2005, pp. 156–163.
- ——, “Time reversal on wiener space,” in Stochastic Processes—Mathematics and Physics: Proceedings of the 1st BiBoS-Symposium held in Bielefeld, West Germany, September 10–15, 1984. Springer, 2006, pp. 119–129.
- P. Cattiaux, G. Conforti, I. Gentil, and C. Léonard, “Time reversal of diffusion processes under a finite entropy condition,” arXiv preprint arXiv:2104.07708, 2021.
- M. Fornasier and F. Solombrino, “Mean-field optimal control,” ESAIM: Control, Optimisation and Calculus of Variations, vol. 20, no. 4, pp. 1123–1152, 2014.
- I. Exarchos and E. A. Theodorou, “Learning optimal control via forward and backward stochastic differential equations,” in 2016 American Control Conference (ACC). IEEE, 2016, pp. 2155–2161.
- J. Han, A. Jentzen, and W. E, “Solving high-dimensional partial differential equations using deep learning,” Proceedings of the National Academy of Sciences, vol. 115, no. 34, pp. 8505–8510, 2018.
- I. Exarchos and E. A. Theodorou, “Stochastic optimal control via forward and backward stochastic differential equations and importance sampling,” Automatica, vol. 87, pp. 159–165, 2018.
- R. Archibald, “A stochastic gradient descent approach for stochastic optimal control,” East Asian Journal on Applied Mathematics, vol. 10, no. 4, 2020.
- M. Pereira, Z. Wang, T. Chen, E. Reed, and E. Theodorou, “Feynman-kac neural network architectures for stochastic control using second-order fbsde theory,” in Learning for Dynamics and Control. PMLR, 2020, pp. 728–738.
- E. Pardoux and P. Protter, “A two-sided stochastic integral and its calculus,” Probability theory and related fields, vol. 76, no. 1, pp. 15–49, 1987.