Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A hierarchical control framework for autonomous decision-making systems: Integrating HMDP and MPC (2401.06833v1)

Published 12 Jan 2024 in cs.SY, cs.AI, cs.RO, and eess.SY

Abstract: This paper proposes a comprehensive hierarchical control framework for autonomous decision-making arising in robotics and autonomous systems. In a typical hierarchical control architecture, high-level decision making is often characterised by discrete state and decision/control sets. However, a rational decision is usually affected by not only the discrete states of the autonomous system, but also the underlying continuous dynamics even the evolution of its operational environment. This paper proposes a holistic and comprehensive design process and framework for this type of challenging problems, from new modelling and design problem formulation to control design and stability analysis. It addresses the intricate interplay between traditional continuous systems dynamics utilized at the low levels for control design and discrete Markov decision processes (MDP) for facilitating high-level decision making. We model the decision making system in complex environments as a hybrid system consisting of a controlled MDP and autonomous (i.e. uncontrolled) continuous dynamics. Consequently, the new formulation is called as hybrid Markov decision process (HMDP). The design problem is formulated with a focus on ensuring both safety and optimality while taking into account the influence of both the discrete and continuous state variables of different levels. With the help of the model predictive control (MPC) concept, a decision maker design scheme is proposed for the proposed hybrid decision making model. By carefully designing key ingredients involved in this scheme, it is shown that the recursive feasibility and stability of the proposed autonomous decision making scheme are guaranteed. The proposed framework is applied to develop an autonomous lane changing system for intelligent vehicles.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. P. Antsaklis. Autonomy and metrics of autonomy. Annual Reviews in Control, 49:15–26, 2020.
  2. Sos: safe, optimal and small strategies for hybrid markov decision processes. In Quantitative Evaluation of Systems: 16th International Conference, QEST 2019, Glasgow, UK, September 10–12, 2019, Proceedings 16, pages 147–164. Springer, 2019.
  3. J. J Beard and A. Baheri. Safety verification of autonomous systems: A multi-fidelity reinforcement learning approach. arXiv preprint arXiv, 2203, 2022.
  4. R. Bellman. A Markovian decision process. Journal of Mathematics and Mechanics, pages 679–684, 1957.
  5. A fast Markov decision process-based algorithm for collision avoidance in urban air mobility. IEEE Transactions on Intelligent Transportation Systems, 23(9):15420–15433, 2022.
  6. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control, 46:8–28, 2018.
  7. V. A. Butakov and P. Ioannou. Personalized driver/vehicle lane change models for ADAS. IEEE Transactions on Vehicular Technology, 64(10):4422–4431, 2014.
  8. The assistant project: AI for high level decisions in manufacturing. International Journal of Production Research, 61(7):2288–2306, 2023.
  9. H. Chen and F. Allgöwer. A quasi-infinite horizon nonlinear model predictive control scheme with guaranteed stability. Automatica, 34(10):1205–1217, 1998.
  10. Optimal control of nonlinear systems: A predictive control approach. Automatica, 39(4):633–641, 2003.
  11. Wen-Hua Chen. Perspective view of autonomous control in unknown environment: Dual control for exploitation and exploration vs reinforcement learning. Neurocomputing, 497:50–63, 2022.
  12. Markov decision process in the problem of dynamic pricing policy. Automatic Control and Computer Sciences, 45:361–371, 2011.
  13. Observer-based distributed model predictive control of complex systems with state coupling constraints. IET Control Theory & Applications, 16(13):1364–1372, 2022.
  14. Synthesis of human-in-the-loop control protocols for autonomous systems. IEEE Transactions on Automation Science and Engineering, 13(2):450–462, 2016.
  15. Guaranteed safe controller synthesis for switched systems using analytical solutions. In 2023 IEEE Conference on Control Technology and Applications (CCTA), pages 784–790. IEEE, 2023.
  16. Situation assessment and decision making for lane change assistance using ensemble learning methods. Expert Systems with Applications, 42(8):3875–3882, 2015.
  17. Autonomous vehicle cut-in algorithm for lane-merging scenarios via policy-based reinforcement learning nested within finite-state machine. IEEE Transactions on Intelligent Transportation Systems, 23(10):17594–17606, 2022.
  18. Autonomous car controller using behaviour planning based on finite state machine. In 2022 6th International Conference on Trends in Electronics and Informatics (ICOEI), pages 296–302. IEEE, 2022.
  19. Path planning and tracking for vehicle collision avoidance based on model predictive control with multiconstraints. IEEE Transactions on Vehicular Technology, 66(2):952–964, 2016.
  20. Tripfield: A 3D potential field model and its applications to local path planning of autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 24(3):3541–3554, 2023.
  21. Collision avoidance analysis for lane changing and merging. IEEE Transactions on Vehicular Technology, 49(6):2295–2308, 2000.
  22. Online and compositional learning of controllers with application to floor heating. In Tools and Algorithms for the Construction and Analysis of Systems: 22nd International Conference, TACAS 2016, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2016, Eindhoven, The Netherlands, April 2-8, 2016, Proceedings 22, pages 244–259. Springer, 2016.
  23. AID-RL: Active information-directed reinforcement learning for autonomous source seeking and estimation. Neurocomputing, 544:126281, 2023.
  24. Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving. Robotics and Autonomous Systems, 131:103568, 2020.
  25. A dynamic automated lane change maneuver based on vehicle-to-vehicle communication. Transportation Research Part C: Emerging Technologies, 62:87–102, 2016.
  26. H. Michalska and D. Q. Mayne. Robust receding horizon control of constrained nonlinear systems. IEEE Transactions on Automatic Control, 38(11):1623–1633, 1993.
  27. A multiple-goal reinforcement learning method for complex vehicle overtaking maneuvers. IEEE Transactions on Intelligent Transportation Systems, 12(2):509–522, 2011.
  28. Planning for safe abortable overtaking maneuvers in autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 508–514. IEEE, 2021.
  29. The mixed-observable constrained linear quadratic regulator problem: the exact solution and practical algorithms. IEEE Transactions on Automatic Control, 2022.
  30. Planning and decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 1:187–210, 2018.
  31. Augmented reality (ar) for surgical robotic and autonomous systems: State of the art, challenges, and solutions. Sensors, 23(13):6202, 2023.
  32. Why do retail customers adopt artificial intelligence (AI) based autonomous decision-making systems? IEEE Transactions on Engineering Management, 2022.
  33. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
  34. Decision making for autonomous vehicles: Combining safety and optimality. IFAC-PapersOnLine, 53(2):15380–15387, 2020.
  35. Stability analysis of switched linear systems defined by regular languages. IEEE Transactions on Automatic Control, 62(5):2568–2575, 2016.
  36. Policy synthesis for switched linear systems with markov decision process switching. IEEE Transactions on Automatic Control, 68(1):532–539, 2022.
  37. A hazard analysis approach based on stpa and finite state machine for autonomous vehicles. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 150–156. IEEE, 2021.
  38. A reinforcement learning approach to autonomous decision making of intelligent vehicles on highways. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 50(10):3884–3897, 2018.
  39. Surviving disturbances: A predictive control framework with guaranteed safety. Automatica, 158, 2023.
  40. Advanced planning for autonomous vehicles using reinforcement learning and deep inverse reinforcement learning. Robotics and Autonomous Systems, 114:1–18, 2019.
  41. Cooperative multiagent deep reinforcement learning for reliable surveillance via autonomous multi-UAV control. IEEE Transactions on Industrial Informatics, 18(10):7086–7096, 2022.
  42. Integrated decision making and motion control for autonomous emergency avoidance based on driving primitives transition. IEEE Transactions on Vehicular Technology, 72(4):4207–4221, 2023.
  43. A hierarchical planning and control framework for structured highway driving. IFAC-PapersOnLine, 50(1):9101–9107, 2017.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now