Papers
Topics
Authors
Recent
Search
2000 character limit reached

Efficient Duple Perturbation Robustness in Low-rank MDPs

Published 11 Apr 2024 in cs.LG and math.OC | (2404.08089v1)

Abstract: The pursuit of robustness has recently been a popular topic in reinforcement learning (RL) research, yet the existing methods generally suffer from efficiency issues that obstruct their real-world implementation. In this paper, we introduce duple perturbation robustness, i.e. perturbation on both the feature and factor vectors for low-rank Markov decision processes (MDPs), via a novel characterization of $(\xi,\eta)$-ambiguity sets. The novel robust MDP formulation is compatible with the function representation view, and therefore, is naturally applicable to practical RL problems with large or even continuous state-action spaces. Meanwhile, it also gives rise to a provably efficient and practical algorithm with theoretical convergence rate guarantee. Examples are designed to justify the new robustness concept, and algorithmic efficiency is supported by both theoretical bounds and numerical simulations.

Authors (4)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  2. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017.
  3. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  4. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334–1373, 2016.
  5. Accelerating robotic reinforcement learning via parameterized action primitives. Advances in Neural Information Processing Systems, 34:21847–21859, 2021.
  6. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
  7. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  8. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803–3810. IEEE, 2018.
  9. Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access, 9:153171–153187, 2021.
  10. Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280, 2005.
  11. Robust solutions to Markov decision problems with uncertain transition matrices. Operations Research, 53(5):780–798, 2005.
  12. Robust Markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 48(1):203–226, 2023.
  13. Fast Bellman updates for robust MDPs. In International Conference on Machine Learning, pages 1979–1988. PMLR, 2018.
  14. Partial policy iteration for ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-robust Markov decision processes. The Journal of Machine Learning Research, 22(1):12612–12657, 2021.
  15. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331–3339. PMLR, 2021.
  16. Toward theoretical understandings of robust Markov decision processes: Sample complexity and asymptotics. The Annals of Statistics, 50(6):3223–3248, 2022.
  17. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. arXiv preprint arXiv:2305.09659, 2023.
  18. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137–2143. PMLR, 2020.
  19. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pages 10746–10756. PMLR, 2020.
  20. A free lunch from the noise: Provable and practical exploration for representation learning. In Uncertainty in Artificial Intelligence, pages 1686–1696. PMLR, 2022a.
  21. Stochastic nonlinear control via finite-dimensional spectral dynamic embedding. arXiv preprint arXiv:2304.03907, 2023.
  22. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 33:20095–20107, 2020.
  23. Representation learning for online and offline RL in low-rank MDPs. arXiv preprint arXiv:2110.04652, 2021.
  24. Scaling up robust MDPs using function approximation. In International conference on machine learning, pages 181–189. PMLR, 2014.
  25. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pages 511–520. PMLR, 2021.
  26. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv: 2209.06620, 2023.
  27. Latent variable representation for reinforcement learning. arXiv preprint arXiv:2212.08765, 2022b.
  28. Edward Allan Silver. Markovian decision processes with uncertain transition probabilities or rewards. PhD thesis, Massachusetts Institute of Technology, Dept. of Civil Engineering, 1963.
  29. Markovian decision processes with uncertain transition probabilities. Operations Research, 21(3):728–740, 1973.
  30. Robust Markov decision processes. Mathematics of Operations Research, 38(1):153–183, 2013.
  31. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2016.
  32. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632, 2017.
  33. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817–2826. PMLR, 2017.
  34. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024–21037, 2020.
  35. Spectral decomposition representation for reinforcement learning. arXiv preprint arXiv:2208.09515, 2022c.
  36. Pseudo-MDPs and factored linear action models. In 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 1–9. IEEE, 2014.
  37. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431–4506, 2021.
  38. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 70(4):2563–2578, 2022.
  39. Leveraging non-uniformity in first-order non-convex optimization. In International Conference on Machine Learning, pages 7555–7564. PMLR, 2021.
  40. Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019.
  41. Strong duality for the CDT subproblem: a necessary and sufficient condition. SIAM Journal on Optimization, 19(4):1735–1756, 2009.
  42. An optimality gap test for a semidefinite relaxation of a quadratic program with two quadratic constraints. SIAM Journal on Optimization, 31(1):866–886, January 2021.
  43. Elad Hazan. Sparse approximate solutions to semidefinite programs. In Latin American symposium on theoretical informatics, pages 306–316. Springer, 2008.
  44. Approximation algorithms and semidefinite programming. Springer Science & Business Media, 2012.
  45. Linear least-squares algorithms for temporal difference learning. Machine learning, 22:33–57, 1996.
  46. Optimistic natural policy gradient: A simple efficient policy optimization framework for online RL. arXiv preprint arXiv: 2305.11032, 2023.
  47. Optimistic policy optimization with bandit feedback. arXiv preprint arXiv: 2002.08243, 2020.
  48. Strong duality in nonconvex quadratic optimization with two quadratic constraints. SIAM Journal on optimization, 17(3):844–860, 2006.
  49. A Devinatz. Integral representations of positive definite functions. Transactions of the American Mathematical Society, 74(1):56–77, 1953.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free