Pontryagin Neural Operator for Solving Parametric General-Sum Differential Games (2401.01502v2)
Abstract: The values of two-player general-sum differential games are viscosity solutions to Hamilton-Jacobi-Isaacs (HJI) equations. Value and policy approximations for such games suffer from the curse of dimensionality (CoD). Alleviating CoD through physics-informed neural networks (PINN) encounters convergence issues when differentiable values with large Lipschitz constants are present due to state constraints. On top of these challenges, it is often necessary to learn generalizable values and policies across a parametric space of games, e.g., for game parameter inference when information is incomplete. To address these challenges, we propose in this paper a Pontryagin-mode neural operator that outperforms the current state-of-the-art hybrid PINN model on safety performance across games with parametric state constraints. Our key contribution is the introduction of a costate loss defined on the discrepancy between forward and backward costate rollouts, which are computationally cheap. We show that the costate dynamics, which can reflect state constraint violation, effectively enables the learning of differentiable values with large Lipschitz constants, without requiring manually supervised data as suggested by the hybrid PINN model. More importantly, we show that the close relationship between costates and policies makes the former critical in learning feedback control policies with generalizable safety performance.
- DeepReach: A deep learning approach to high-dimensional reachability. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1817–1824. IEEE, 2021.
- Optimizeddp: An efficient, user-friendly library for optimal control and dynamic programming. arXiv preprint arXiv:2204.05520, 2022.
- Pierre Cardaliaguet. Information issues in differential game theory. In ESAIM: Proceedings, volume 35, pages 1–13. EDP Sciences, 2012.
- Viscosity solutions of hamilton-jacobi equations. Transactions of the American mathematical society, 277(1):1–42, 1983.
- Mitigating propagation failures in pinns using evolutionary sampling. 2022.
- Reach-avoid problems with time-varying dynamics, targets and constraints. In Proceedings of the 18th international conference on hybrid systems: computation and control, pages 11–20, 2015.
- A differential game control problem with state constraints. Mathematical Control and Related Fields, 13(2):554–582, 2023.
- Convergence of the deep bsde method for coupled fbsdes. Probability, Uncertainty and Quantitative Risk, 5(1):1–33, 2020.
- Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):8505–8510, 2018.
- Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pages 12556–12569. PMLR, 2023.
- A neural network-based policy iteration algorithm with global h2-superlinear convergence for stochastic games on domains. Foundations of Computational Mathematics, 21(2):331–374, 2021.
- Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404:109136, 2020.
- Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24(89):1–97, 2023.
- Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548–26560, 2021.
- Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
- Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218–229, 2021.
- Olvi L Mangasarian. Sufficient conditions for the optimal control of nonlinear systems. SIAM Journal on control, 4(1):139–152, 1966.
- A toolbox of Hamilton-Jacobi solvers for analysis of nondeterministic continuous and hybrid systems. In Hybrid Systems: Computation and Control: 8th International Workshop, HSCC 2005, Zurich, Switzerland, March 9-11, 2005. Proceedings 8, pages 480–494. Springer, 2005.
- Kolmogorov n–width and lagrangian physics-informed neural networks: a causality-conforming manifold for convection-dominated pdes. Computer Methods in Applied Mechanics and Engineering, 404:115810, 2023.
- Adaptive deep learning for high-dimensional hamilton–jacobi–bellman equations. SIAM Journal on Scientific Computing, 43(2):A1221–A1247, 2021.
- High-order essentially nonoscillatory schemes for Hamilton–Jacobi equations. SIAM Journal on numerical analysis, 28(4):907–922, 1991.
- Level set methods and dynamic implicit surfaces. Appl. Mech. Rev., 57(3):B15–B15, 2004.
- Learning nash equilibrium for general-sum markov games from batch data. In Artificial Intelligence and Statistics, pages 232–241. PMLR, 2017.
- On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. arXiv preprint arXiv:2004.01806, 2020.
- Nonzero-sum differential games. Journal of optimization theory and applications, 3(3):184–206, 1969.
- Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021.
- Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning. Nonlinearity, 35(1):278, 2021.
- Gradient-enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:114823, 2022.
- Approximating discontinuous nash equilibrial values of two-player general-sum differential games. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3022–3028. IEEE, 2023a.
- Value approximation for two-player general-sum differential games with state constraints, 2023b.