Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Safe Reinforcement Learning driven Weights-varying Model Predictive Control for Autonomous Vehicle Motion Control (2402.02624v1)

Published 4 Feb 2024 in cs.RO, cs.AI, cs.LG, cs.SY, and eess.SY

Abstract: Determining the optimal cost function parameters of Model Predictive Control (MPC) to optimize multiple control objectives is a challenging and time-consuming task. Multiobjective Bayesian Optimization (BO) techniques solve this problem by determining a Pareto optimal parameter set for an MPC with static weights. However, a single parameter set may not deliver the most optimal closed-loop control performance when the context of the MPC operating conditions changes during its operation, urging the need to adapt the cost function weights at runtime. Deep Reinforcement Learning (RL) algorithms can automatically learn context-dependent optimal parameter sets and dynamically adapt for a Weightsvarying MPC (WMPC). However, learning cost function weights from scratch in a continuous action space may lead to unsafe operating states. To solve this, we propose a novel approach limiting the RL actions within a safe learning space representing a catalog of pre-optimized BO Pareto-optimal weight sets. We conceive a RL agent not to learn in a continuous space but to proactively anticipate upcoming control tasks and to choose the most optimal discrete actions, each corresponding to a single set of Pareto optimal weights, context-dependent. Hence, even an untrained RL agent guarantees a safe and optimal performance. Experimental results demonstrate that an untrained RL-WMPC shows Pareto-optimal closed-loop behavior and training the RL-WMPC helps exhibit a performance beyond the Pareto-front.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (22)
  1. V. Ramasamy, R. K. Sidharthan, R. Kannan, and G. Muralidharan, “Optimal tuning of model predictive controller weights using genetic algorithm with interactive decision tree for industrial cement kiln process,” Processes, vol. 7, no. 12, p. 938, 2019.
  2. L. L. Rodrigues, A. S. Potts, O. A. Vilcanqui, and A. J. Sguarezi Filho, “Tuning a model predictive controller for doubly fed induction generator employing a constrained genetic algorithm,” IET Electric Power Applications, vol. 13, no. 6, pp. 812–819, 2019.
  3. A. Kapnopoulos and A. Alexandridis, “A cooperative particle swarm optimization approach for tuning an mpc-based quadrotor trajectory tracking scheme,” Aerospace Science and Technology, vol. 127, p. 107725, 2022.
  4. D. Piga, M. Forgione, S. Formentin, and A. Bemporad, “Performance-oriented model learning for data-driven mpc design,” IEEE control systems letters, vol. 3, no. 3, pp. 577–582, 2019.
  5. A. Gharib, D. Stenger, R. Ritschel, and R. Voßwinkel, “Multi-objective optimization of a path-following mpc for vehicle guidance: A bayesian optimization approach,” in 2021 European Control Conference (ECC).   IEEE, 2021, pp. 2197–2204.
  6. F. Sorourifar, G. Makrygirgos, A. Mesbah, and J. A. Paulson, “A data-driven automatic tuning method for mpc under uncertainty using constrained bayesian optimization,” IFAC-PapersOnLine, vol. 54, no. 3, pp. 243–250, 2021.
  7. J. A. Paulson, K. Shao, and A. Mesbah, “Probabilistically robust bayesian optimization for data-driven design of arbitrary controllers with gaussian process emulators,” in 2021 60th IEEE Conference on Decision and Control (CDC).   IEEE, 2021, pp. 3633–3639.
  8. A. Rupenyan, M. Khosravi, and J. Lygeros, “Performance-based trajectory optimization for path following control using bayesian optimization,” in 2021 60th IEEE Conference on Decision and Control (CDC).   IEEE, 2021, pp. 2116–2121.
  9. G. Makrygiorgos, A. D. Bonzanini, V. Miller, and A. Mesbah, “Performance-oriented model learning for control via multi-objective bayesian optimization,” Computers & Chemical Engineering, vol. 162, p. 107770, 2022.
  10. B. Zarrouki, C. Wang, and J. Betz, “Adaptive stochastic nonlinear model predictive control with look-ahead deep reinforcement learning for autonomous vehicle motion control,” 2023.
  11. E. Bøhn, S. Moe, S. Gros, and T. A. Johansen, “Reinforcement learning of the prediction horizon in model predictive control,” IFAC-PapersOnLine, vol. 54, no. 6, pp. 314–320, 2021.
  12. E. Bøhn, S. Gros, S. Moe, and T. A. Johansen, “Optimization of the model predictive control meta-parameters through reinforcement learning,” Engineering Applications of Artificial Intelligence, vol. 123, p. 106211, 2023.
  13. M. Mehndiratta, E. Camci, and E. Kayacan, “Automated tuning of nonlinear model predictive controller by reinforcement learning,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2018, pp. 3016–3021.
  14. B. Zarrouki, V. Klös, N. Heppner, S. Schwan, R. Ritschel, and R. Voßwinkel, “Weights-varying mpc for autonomous vehicle guidance: a deep reinforcement learning approach,” in 2021 European Control Conference (ECC).   IEEE, 2021, pp. 119–125.
  15. B. Zarrouki, “Reinforcement learning of model predictive control parameters for autonomous vehicle guidance,” Master’s thesis, 2020.
  16. K. P. Wabersich and M. N. Zeilinger, “Performance and safety of bayesian model predictive control: Scalable model-based rl with guarantees,” arXiv preprint arXiv:2006.03483, 2020.
  17. F. Berkenkamp, A. P. Schoellig, and A. Krause, “Safe controller optimization for quadrotors with gaussian processes,” in 2016 IEEE international conference on robotics and automation (ICRA).   IEEE, 2016, pp. 491–496.
  18. F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause, “Safe model-based reinforcement learning with stability guarantees,” Advances in neural information processing systems, vol. 30, 2017.
  19. B. Zarrouki, C. Wang, and J. Betz, “A stochastic nonlinear model predictive control with an uncertainty propagation horizon for autonomous vehicle motion control,” arXiv preprint arXiv:2310.18753, 2023.
  20. M. Emmerich, K. C. Giannakoglou, and B. Naujoks, “Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels,” IEEE Transactions on Evolutionary Computation, vol. 10, no. 4, pp. 421–439, 2006.
  21. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” 2017.
  22. A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” Journal of Machine Learning Research, vol. 22, no. 268, pp. 1–8, 2021. [Online]. Available: http://jmlr.org/papers/v22/20-1364.html
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now