2000 character limit reached
Safe Exploration Using Bayesian World Models and Log-Barrier Optimization
Published 9 May 2024 in cs.LG and cs.AI | (2405.05890v1)
Abstract: A major challenge in deploying reinforcement learning in online tasks is ensuring that safety is maintained throughout the learning process. In this work, we propose CERL, a new method for solving constrained Markov decision processes while keeping the policy safe during learning. Our method leverages Bayesian world models and suggests policies that are pessimistic w.r.t. the model's epistemic uncertainty. This makes CERL robust towards model inaccuracies and leads to safe exploration during learning. In our experiments, we demonstrate that CERL outperforms the current state-of-the-art in terms of safety and optimality in solving CMDPs from image observations.
- Constrained policy optimization, 2017.
- E. Altman. Constrained Markov Decision Processes. Chapman and Hall, 1999.
- Linear stochastic bandits under safety constraints, 2019.
- Constrained policy optimization via bayesian world models, 2022.
- Logarithmic online regret bounds for undiscounted reinforcement learning. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2007. URL https://proceedings.neurips.cc/paper/2006/file/c1b70d965ca504aa751ddb62ad69c63f-Paper.pdf.
- Gosafe: Globally optimal safe robot learning. In ICRA, 2021.
- Safe model-based reinforcement learning with stability guarantees, 2017.
- Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, pp. 1–35, 2021.
- Risk-constrained reinforcement learning with percentile risk criteria. CoRR, abs/1512.01629, 2015. URL http://arxiv.org/abs/1512.01629.
- Lyapunov-based safe policy optimization for continuous control, 2019.
- Deep reinforcement learning in a handful of trials using probabilistic dynamics models. CoRR, abs/1805.12114, 2018. URL http://arxiv.org/abs/1805.12114.
- Efficient model-based reinforcement learning through optimistic policy search and planning, 2020.
- Safe exploration in continuous action spaces, 2018.
- Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, pp. 465–472, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
- Exploration-exploitation in constrained mdps, 2020.
- Bayesian reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 8(5–6):359–483, 2015. ISSN 1935-8245. doi: 10.1561/2200000049. URL http://dx.doi.org/10.1561/2200000049.
- Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555–2565, 2019a.
- Dream to control: Learning behaviors by latent imagination. CoRR, abs/1912.01603, 2019b. URL http://arxiv.org/abs/1912.01603.
- Mastering atari with discrete world models, 2021.
- Information-based transductive active learning, 2024.
- When to trust your model: Model-based policy optimization, 2019.
- Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
- Constrained variational policy optimization for safe reinforcement learning, 2022.
- Benchmarking Safe Exploration in Deep Reinforcement Learning. 2019.
- Proximal policy optimization algorithms, 2017.
- Safe exploration for optimization with gaussian processes. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 997–1005, Lille, France, 07–09 Jul 2015. PMLR. URL http://proceedings.mlr.press/v37/sui15.html.
- Gosafeopt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence, 2023.
- Optimistic active exploration of dynamical systems. NeurIPS, 2024.
- Action robust reinforcement learning and applications in continuous control, 2019.
- Safe exploration for interactive machine learning. NeurIPS, 32, 2019.
- Log barriers for safe black-box optimization with application to safe reinforcement learning. arXiv preprint arXiv:2207.10415, 2022.
- Tuning legged locomotion controllers via safe bayesian optimization. In Conference on Robot Learning, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.