Papers
Topics
Authors
Recent
Search
2000 character limit reached

Conservative DDPG -- Pessimistic RL without Ensemble

Published 8 Mar 2024 in cs.AI and cs.LG | (2403.05732v2)

Abstract: DDPG is hindered by the overestimation bias problem, wherein its QQ-estimates tend to overstate the actual QQ-values. Traditional solutions to this bias involve ensemble-based methods, which require significant computational resources, or complex log-policy-based approaches, which are difficult to understand and implement. In contrast, we propose a straightforward solution using a QQ-target and incorporating a behavioral cloning (BC) loss penalty. This solution, acting as an uncertainty measure, can be easily implemented with minimal code and without the need for an ensemble. Our empirical findings strongly support the superiority of Conservative DDPG over DDPG across various MuJoCo and Bullet tasks. We consistently observe better performance in all evaluated tasks and even competitive or superior performance compared to TD3 and TD7, all achieved with significantly reduced computational requirements.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (25)
  1. Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp.  679–684, 1957.
  2. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
  3. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org, 2016–2021.
  4. An actor/critic algorithm that is equivalent to q-learning. In G. Tesauro, D. Touretzky, and T. Leen (eds.), Advances in Neural Information Processing Systems, volume 7. MIT Press, 1994. URL https://proceedings.neurips.cc/paper_files/paper/1994/file/23ce1851341ec1fa9e0c259de10bf87c-Paper.pdf.
  5. Meta-q-learning. arXiv preprint arXiv:1910.00125, 2019.
  6. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.  1587–1596. PMLR, 2018a.
  7. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp.  1587–1596. PMLR, 2018b.
  8. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp.  2052–2062. PMLR, 2019.
  9. For sale: State-action representation learning for deep reinforcement learning. arXiv preprint arXiv:2306.02451, 2023.
  10. Continuous deep q-learning with model-based acceleration. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp.  2829–2838, New York, New York, USA, 20–22 Jun 2016. PMLR. URL https://proceedings.mlr.press/v48/gu16.html.
  11. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.  1861–1870. PMLR, 2018.
  12. Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
  13. Matthias Heger. Consideration of risk in reinforcement learning. In Machine Learning Proceedings 1994, pp.  105–111. Elsevier, 1994.
  14. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp.  267–274, 2002.
  15. Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487, 2020.
  16. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  17. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  18. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pp.  5331–5340. PMLR, 2019.
  19. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp.  1889–1897, Lille, France, 07–09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/schulman15.html.
  20. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  21. Deterministic policy gradient algorithms. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp.  387–395, Bejing, China, 22–24 Jun 2014. PMLR. URL https://proceedings.mlr.press/v32/silver14.html.
  22. Reinforcement learning: An introduction. MIT press, 2018.
  23. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.  5026–5033. IEEE, 2012.
  24. Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), Mar. 2016. doi: 10.1609/aaai.v30i1.10295. URL https://ojs.aaai.org/index.php/AAAI/article/view/10295.
  25. Q-learning. Machine learning, 8:279–292, 1992.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free