2000 character limit reached
Policy Optimization for PDE Control with a Warm Start
Published 1 Mar 2024 in eess.SY, cs.AI, cs.SY, and math.OC | (2403.01005v1)
Abstract: Dimensionality reduction is crucial for controlling nonlinear partial differential equations (PDE) through a "reduce-then-design" strategy, which identifies a reduced-order model and then implements model-based control solutions. However, inaccuracies in the reduced-order modeling can substantially degrade controller performance, especially in PDEs with chaotic behavior. To address this issue, we augment the reduce-then-design procedure with a policy optimization (PO) step. The PO step fine-tunes the model-based controller to compensate for the modeling error from dimensionality reduction. This augmentation shifts the overall strategy into reduce-then-design-then-adapt, where the model-based controller serves as a warm start for PO. Specifically, we study the state-feedback tracking control of PDEs that aims to align the PDE state with a specific constant target subject to a linear-quadratic cost. Through extensive experiments, we show that a few iterations of PO can significantly improve the model-based controller performance. Our approach offers a cost-effective alternative to PDE control using end-to-end reinforcement learning.
- Closed-loop turbulence control: Progress and challenges. Applied Mechanics Reviews, 67(5):050801, 2015.
- Reduced order controllers for Burgers’ equation with a nonlinear observer. International Journal of Applied Mathematics and Computer Science, 11(6):1311–1330, 2001.
- Proper orthogonal decomposition surrogate models for nonlinear dynamical systems: Error estimates and suboptimal control. In Dimension Reduction of Large-Scale Systems: Proceedings of a Workshop held in Oberwolfach, Germany, October 19–25, 2003, pages 261–306. Springer, 2005.
- Numerical feedback controller design for PDE systems using model reduction: Techniques and case studies. In Real-Time PDE-Constrained Optimization, pages 53–72. SIAM, 2007.
- Explicit model predictive control for large-scale systems via model reduction. Journal of guidance, control, and dynamics, 31(4):918–926, 2008.
- Closed-loop control of an open cavity flow using reduced-order models. Journal of Fluid Mechanics, 641:1–50, 2009.
- Koopman operator-based model reduction for switched-system control of PDEs. Automatica, 106:184–191, 2019.
- Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders. Journal of Computational Physics, 404:108973, 2020.
- Model reduction and neural networks for parametric PDEs. The SMAI Journal of Computational Mathematics, 7:121–157, 2021.
- A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs. Journal of Scientific Computing, 87:1–36, 2021.
- On closures for reduced order models—a spectrum of first-principle to machine-learned avenues. Physics of Fluids, 33(9), 2021.
- Data-driven discovery of Koopman eigenfunctions for control. Machine Learning: Science and Technology, 2(3):035023, 2021.
- Jacques Louis Lions. Optimal Control of Systems Governed by Partial Differential Equations, volume 170. Springer, 1971.
- Optimal control: linear quadratic methods, 1990.
- H-Infinity Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach. Birkhäuser, Boston, 1995.
- Boundary Control of PDEs: A Course on Backstepping Designs. SIAM, 2008.
- Feedback stabilization of a thermal fluid system with mixed boundary control. Computers & Mathematics with Applications, 71(11):2170–2191, 2016.
- Dynamic mode decomposition with control. SIAM Journal on Applied Dynamical Systems, 15(1):142–161, 2016.
- Approximation of high-dimensional parametric PDEs. Acta Numerica, 24:1–159, 2015.
- A survey of projection-based model reduction methods for parametric dynamical systems. SIAM Review, 57(4):483–531, 2015.
- Model reduction for flow analysis and control. Annual Review of Fluid Mechanics, 49:387–417, 2017.
- Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge University Press, 2012.
- Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Cambridge University Press, 2022.
- Learning nonlinear reduced models from data with operator inference. Annual Review of Fluid Mechanics, 56:521–548, 2024.
- Linear closed-loop control of fluid instabilities and noise-induced perturbations: A review of approaches and tools. Applied Mechanics Reviews, 68(2):020801, 2016.
- Active control of transonic buffet flow. Journal of Fluid Mechanics, 824:312–351, 2017.
- Estimation and control of fluid flows using sparsity-promoting dynamic mode decomposition. IEEE Control Systems Letters, 5(4):1145–1150, 2020.
- Machine Learning Control-Taming Nonlinear Dynamics and Turbulence, volume 116. Springer, 2017.
- Bayesian optimization for active flow control. Acta Mechanica Sinica, pages 1–13, 2021.
- Reinforcement learning for bluff body active flow control in experiments and simulations. Proceedings of the National Academy of Sciences, 117(42):26091–26098, 2020.
- A review on deep reinforcement learning for fluid mechanics. Computers & Fluids, 225:104973, 2021.
- Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica, 45(2):477–484, 2009.
- High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
- Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
- Benjamin Recht. A tour of reinforcement learning: The view from continuous control. Annual Review of Control, Robotics, and Autonomous Systems, 2:253–279, 2019.
- Controlgym: Large-scale safety-critical control environments for benchmarking reinforcement learning algorithms. arXiv preprint arXiv:2311.18736, 2023.
- Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning, pages 1467–1476, 2018.
- Policy optimization for ℋ2subscriptℋ2\mathcal{H}_{2}caligraphic_H start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT linear control with ℋ∞subscriptℋ\mathcal{H}_{\infty}caligraphic_H start_POSTSUBSCRIPT ∞ end_POSTSUBSCRIPT robustness guarantee: Implicit regularization and global convergence. SIAM Journal on Control and Optimization, 59(6):4081–4109, 2021.
- Combining model-based and model-free methods for nonlinear control: A provably convergent policy gradient approach. arXiv preprint arXiv:2006.07476, 2020.
- Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity. Advances in Neural Information Processing Systems, 34:2949–2964, 2021.
- Revisiting LQR control from the perspective of receding-horizon policy gradient. IEEE Control Systems Letters, 7:1664–1669, 2023.
- Learning the Kalman filter with fine-grained sample complexity. arXiv preprint arXiv:2301.12624, 2023.
- Toward a theoretical foundation of policy optimization for learning control policies. Annual Review of Control, Robotics, and Autonomous Systems, 6:123–158, 2023.
- Learning minimax-optimal terminal state estimators and smoothers. In 22nd IFAC World Congress, pages 12391–12396, 2023.
- Global convergence of receding-horizon policy search in learning estimator designs. arXiv preprint arXiv:2309.04831, 2023.
- Modal analysis of fluid flows: An overview. AIAA Journal, 55(12):4013–4041, 2017.
- Online convex optimization in the bandit setting: Gradient descent without a gradient. In ACM-SIAM Symposium on Discrete Algorithms, pages 385–394, 2005.
- Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015.
- Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527–566, 2017.
- Reinforcement learning-based estimation for partial differential equations. In SIAM Conference on Applications of Dynamical Systems, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
