Papers
Topics
Authors
Recent
Search
2000 character limit reached

On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond

Published 6 Jan 2024 in cs.LG, cs.AI, and stat.ML | (2401.03301v2)

Abstract: We seek to understand what facilitates sample-efficient learning from historical datasets for sequential decision-making, a problem that is popularly known as offline reinforcement learning (RL). Further, we are interested in algorithms that enjoy sample efficiency while leveraging (value) function approximation. In this paper, we address these fundamental questions by (i) proposing a notion of data diversity that subsumes the previous notions of coverage measures in offline RL and (ii) using this notion to {unify} three distinct classes of offline RL algorithms based on version spaces (VS), regularized optimization (RO), and posterior sampling (PS). We establish that VS-based, RO-based, and PS-based algorithms, under standard assumptions, achieve \emph{comparable} sample efficiency, which recovers the state-of-the-art sub-optimality bounds for finite and linear model classes with the standard assumptions. This result is surprising, given that the prior work suggested an unfavorable sample complexity of the RO-based algorithm compared to the VS-based algorithm, whereas posterior sampling is rarely considered in offline RL due to its explorative nature. Notably, our proposed model-free PS-based algorithm for offline RL is {novel}, with sub-optimality bounds that are {frequentist} (i.e., worst-case) in nature.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Non-linear reinforcement learning in large action spaces: Structural conditions and sample-efficiency of posterior sampling. In Conference on Learning Theory, pages 2776–2814. PMLR, 2022.
  2. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res., 22(98):1–76, 2021.
  3. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. In Gábor Lugosi and Hans Ulrich Simon, editors, Learning Theory, 19th Annual Conference on Learning Theory, COLT 2006, Pittsburgh, PA, USA, June 22-25, 2006, Proceedings, volume 4005 of Lecture Notes in Computer Science, pages 574–588. Springer, 2006.
  4. The multiplicative weights update method: a meta-algorithm and applications. Theory of Computing, 8(6):121–164, 2012. doi: 10.4086/toc.2012.v008a006.
  5. Local rademacher complexities. The Annals of Statistics, 33(4):1497–1537, 2005.
  6. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042–1051. PMLR, 2019.
  7. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence, pages 378–388. PMLR, 2022.
  8. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852–3878. PMLR, 2022.
  9. A provably efficient model-free posterior sampling method for episodic reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
  10. Pessimistic nonlinear least-squares value iteration for offline reinforcement learning. arXiv preprint arXiv:2310.01380, 2023.
  11. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pages 2701–2709. PMLR, 2020.
  12. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
  13. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487, 2021.
  14. David A. Freedman. On tail probabilities for martingales. The Annals of Probability, 3(1):100–118, 1975. ISSN 00911798.
  15. Offline primal-dual reinforcement learning for linear mdps. arXiv preprint arXiv:2305.12944, 2023.
  16. Sample complexity of nonparametric off-policy evaluation on low-dimensional manifolds using deep networks. arXiv preprint arXiv:2206.02887, 2022.
  17. Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning, pages 1704–1713. PMLR, 2017.
  18. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137–2143. PMLR, 2020.
  19. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021a.
  20. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pages 5084–5096. PMLR, 2021b.
  21. Policy learning" without”overlap: Pessimism and generalized empirical bernstein’s inequality. arXiv preprint arXiv:2212.09900, 2022.
  22. Batch reinforcement learning. In Reinforcement learning, pages 45–73. Springer, 2012.
  23. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
  24. Off-policy policy gradient with stationary distribution correction. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, volume 115 of Proceedings of Machine Learning Research, pages 1180–1190. AUAI Press, 2019.
  25. Pascal Massart. Some applications of concentration inequalities to statistics. Annales de la Faculté des Sciences de Toulouse, 9:245–303, 2000.
  26. Finite-time bounds for fitted value iteration. J. Mach. Learn. Res., 9:815–857, 2008.
  27. VIPer: Provably efficient algorithm for offline RL with neural function approximation. In The Eleventh International Conference on Learning Representations, 2023.
  28. Offline neural contextual bandits: Pessimism, optimization and generalization. In International Conference on Learning Representations, 2022a.
  29. On sample complexity of offline reinforcement learning with deep reLU networks in besov spaces. Transactions of Machine Learning Research, 2022b.
  30. On instance-dependent bounds for offline reinforcement learning with linear function approximation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9310–9318, 2023.
  31. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
  32. Revisiting the linear-programming framework for offline rl with general function approximation. In International Conference on Machine Learning, pages 26769–26791. PMLR, 2023.
  33. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34, 2021.
  34. Optimal conservative offline RL with general function approximation via augmented lagrangian. In The Eleventh International Conference on Learning Representations, 2023.
  35. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4):1221–1243, 2014.
  36. William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25:285–294, 1933.
  37. On the theory of transfer learning: The importance of task diversity. Advances in neural information processing systems, 33:7852–7862, 2020.
  38. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations, 2022.
  39. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355, 2022a.
  40. Representation learning for online and offline RL in low-rank MDPs. In International Conference on Learning Representations, 2022b.
  41. What are the statistical limits of offline RL with linear function approximation? In International Conference on Learning Representations, 2021.
  42. Optimistic rates for multi-task representation learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  43. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681–688, 2011.
  44. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 2021.
  45. A self-play posterior sampling algorithm for zero-sum markov games. In International Conference on Machine Learning, pages 24496–24523. PMLR, 2022.
  46. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent MDP and markov game. In The Eleventh International Conference on Learning Representations, 2023.
  47. Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995–7004. PMLR, 2019.
  48. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems, 34, 2021.
  49. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning, pages 10978–10989. PMLR, 2020.
  50. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626–13640, 2021.
  51. Policy learning with adaptively collected data. Management Science, 2023.
  52. Offline reinforcement learning with realizability and single-policy concentrability. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 of Proceedings of Machine Learning Research, pages 2730–2775. PMLR, 02–05 Jul 2022.
  53. Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834–857, 2022.
  54. Tong Zhang. Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press, 2023.
  55. Local optimization achieves global optimality in multi-agent reinforcement learning. arXiv preprint arXiv:2305.04819, 2023.
  56. A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962, 2022.
  57. Importance weighted actor-critic for optimal conservative offline reinforcement learning. arXiv preprint arXiv:2301.12714, 2023.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free