Addressing Myopic Constrained POMDP Planning with Recursive Dual Ascent
Abstract: Lagrangian-guided Monte Carlo tree search with global dual ascent has been applied to solve large constrained partially observable Markov decision processes (CPOMDPs) online. In this work, we demonstrate that these global dual parameters can lead to myopic action selection during exploration, ultimately leading to suboptimal decision making. To address this, we introduce history-dependent dual variables that guide local action selection and are optimized with recursive dual ascent. We empirically compare the performance of our approach on a motivating toy example and two large CPOMDPs, demonstrating improved exploration, and ultimately, safer outcomes.
- Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47: 235–256.
- A POMDP model for safe geological carbon sequestration. In Workshop on Tackling Climate Change with Machine Learning, Advances in Neural Information Processing Systems (NeurIPS).
- POMDPs.jl: a framework for sequential decision making under uncertainty. 18: 1–5.
- Recursively-Constrained Partially Observable Markov Decision Processes. arXiv:2310.09688.
- Online planning for constrained POMDPs with continuous spaces through dual ascent. In International Conference on Automated Planning and Scheduling (ICAPS).
- Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101: 99–134.
- Point-based value iteration for constrained POMDPs. In International Joint Conference on Artificial Intelligence (IJCAI).
- Algorithms for Decision Making. MIT Press.
- Kurniawati, H. 2022. Partially observable Markov decision processes and robotics. Annual Review of Control, Robotics, and Autonomous Systems, 5: 253–277.
- Monte-Carlo tree search for constrained POMDPs. In Advances in Neural Information Processing Systems (NeurIPS), volume 31.
- C-MCTS: Safe Planning with Monte Carlo Tree Search. arXiv:2305.16209.
- Approximate linear programming for constrained partially observable Markov decision processes. In AAAI Conference on Artificial Intelligence (AAAI), volume 29.
- A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48: 67–113.
- Monte-Carlo planning in large POMDPs. In Advances in Neural Information Processing Systems (NeurIPS), 2164–2172.
- An online algorithm for constrained POMDPs. In IEEE International Conference on Robotics and Automation (ICRA), 3966–3973.
- Column generation algorithms for constrained POMDPs. Journal of Artificial Intelligence Research, 62: 489–533.
- Scalable gradient ascent for controllers in constrained POMDPs. In IEEE International Conference on Robotics and Automation (ICRA), 9085–9091.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.