Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (2312.14470v1)

Published 22 Dec 2023 in cs.LG

Abstract: This paper studies safe Reinforcement Learning (safe RL) with linear function approximation and under hard instantaneous constraints where unsafe actions must be avoided at each step. Existing studies have considered safe RL with hard instantaneous constraints, but their approaches rely on several key assumptions: $(i)$ the RL agent knows a safe action set for {\it every} state or knows a {\it safe graph} in which all the state-action-state triples are safe, and $(ii)$ the constraint/cost functions are {\it linear}. In this paper, we consider safe RL with instantaneous hard constraints without assumption $(i)$ and generalize $(ii)$ to Reproducing Kernel Hilbert Space (RKHS). Our proposed algorithm, LSVI-AE, achieves $\tilde{\cO}(\sqrt{d3H4K})$ regret and $\tilde{\cO}(H \sqrt{dK})$ hard constraint violation when the cost function is linear and $\cO(H\gamma_K \sqrt{K})$ hard constraint violation when the cost function belongs to RKHS. Here $K$ is the learning horizon, $H$ is the length of each episode, and $\gamma_K$ is the information gain w.r.t the kernel used to approximate cost functions. Our results achieve the optimal dependency on the learning horizon $K$, matching the lower bound we provide in this paper and demonstrating the efficiency of LSVI-AE. Notably, the design of our approach encourages aggressive policy exploration, providing a unique perspective on safe RL with general cost functions and no prior knowledge of safe actions, which may be of independent interest.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Improved Algorithms for Linear Stochastic Bandits. In Advances in Neural Information Processing Systems 24.
  2. Linear stochastic bandits under safety constraints. In Advances Neural Information Processing Systems (NeurIPS), 9256–9266.
  3. Safe reinforcement learning with linear function approximation. In International Conference on Machine Learning, 243–253. PMLR.
  4. Learning dexterous in-hand manipulation. The Int. Journal of Robotics Research, 39(1): 3–20.
  5. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach. In AAAI Conf. Artificial Intelligence, 3682–3689.
  6. Safe exploration for constrained reinforcement learning with provable guarantees. arXiv preprint arXiv:2112.00885.
  7. Efficient algorithms for budget-constrained markov decision processes. IEEE Transactions on Automatic Control, 59(10): 2813–2817.
  8. Prediction, Learning, and Games. Cambridge University Press.
  9. Learning Infinite-horizon Average-reward Markov Decision Process with Constraints. In Int. Conf. Machine Learning (ICML), 3246–3270. PMLR.
  10. On kernelized multi-armed bandits. In Int. Conf. Machine Learning (ICML), 844–853. PMLR.
  11. Provably Efficient Safe Exploration via Primal-Dual Policy Optimization. In Int. Conf. Artificial Intelligence and Statistics (AISTATS), volume 130, 3304–3312. PMLR.
  12. Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes. In Advances Neural Information Processing Systems (NeurIPS), volume 33, 8378–8390. Curran Associates, Inc.
  13. Convergence and optimality of policy gradient primal-dual method for constrained Markov decision processes. In acc, 2851–2856. IEEE.
  14. Exploration-exploitation in constrained MDPs. arXiv preprint arXiv:2003.02189.
  15. Provably Efficient Model-Free Constrained RL with Linear Function Approximation. In NeurIPS.
  16. Online Convex Optimization with Hard Constraints: Towards the Best of Two Worlds and Beyond. In Advances Neural Information Processing Systems (NeurIPS).
  17. Rectified Pessimistic-Optimistic Learning for Stochastic Continuum-armed Bandit with Constraints. arXiv preprint arXiv:2211.14720.
  18. Nearly minimax optimal reinforcement learning for linear markov decision processes. In Int. Conf. Machine Learning (ICML), 12790–12822. PMLR.
  19. Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning, 8971–9019. PMLR.
  20. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems, 34: 13406–13418.
  21. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, 2137–2143. PMLR.
  22. Bandit Algorithms. Cambridge University Press.
  23. Non-stationary bandits with knapsacks. In Advances Neural Information Processing Systems (NeurIPS), volume 35, 16522–16532.
  24. Learning policies with zero or bounded constraint violation for constrained MDPs. In Advances Neural Information Processing Systems (NeurIPS), volume 34.
  25. An Efficient Pessimistic-Optimistic Algorithm for Stochastic Linear Bandits with General Constraints. In Advances Neural Information Processing Systems (NeurIPS).
  26. Stochastic Bandits with Linear Constraints. In Int. Conf. Artificial Intelligence and Statistics (AISTATS).
  27. A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraints. iclm.
  28. Mastering the game of go without human knowledge. nature, 550(7676): 354–359.
  29. Learning in Markov decision processes under constraints. arXiv preprint arXiv:2002.12435.
  30. Safe exploration in finite markov decision processes with gaussian processes. Advances in neural information processing systems, 29.
  31. Vershynin, R. 2010. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027.
  32. Safe exploration and optimization of constrained MDPs using Gaussian processes. In AAAI Conf. Artificial Intelligence, volume 32, 6548–6555. ISBN 978-1-57735-800-8.
  33. Provably Efficient Model-Free Algorithms for Non-stationary CMDPs. In Int. Conf. Artificial Intelligence and Statistics (AISTATS), 6527–6570. PMLR.
  34. A Provably-Efficient Model-Free Algorithm for Infinite-Horizon Average-Reward Constrained Markov Decision Processes. In AAAI Conf. Artificial Intelligence.
  35. Triple-Q: A Model-Free Algorithm for Constrained Reinforcement Learning with Sublinear Regret and Zero Constraint Violation. In Int. Conf. Artificial Intelligence and Statistics (AISTATS).
  36. Budget constrained bidding by model-free reinforcement learning in display advertising. In Proc. ACM Int. Conf. Information and Knowledge Management (CIKM), 1443–1451.
  37. On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622.
  38. Regret and cumulative constraint violation analysis for online convex optimization with long term constraints. In Int. Conf. Machine Learning (ICML), 11998–12008. PMLR.
  39. Regret and cumulative constraint violation analysis for distributed online constrained convex optimization. IEEE Transactions on Automatic Control.
  40. A Low Complexity Algorithm with O⁢(T)𝑂𝑇O(\sqrt{T})italic_O ( square-root start_ARG italic_T end_ARG ) Regret and O⁢(1)𝑂1O(1)italic_O ( 1 ) Constraint Violations for Online Convex Optimization with Long Term Constraints. Journal of Machine Learning Research, 21(1): 1–24.
  41. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In colt, 4532–4576. PMLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Honghao Wei (16 papers)
  2. Xin Liu (820 papers)
  3. Lei Ying (89 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now