Papers
Topics
Authors
Recent
Search
2000 character limit reached

LESSON: Learning to Integrate Exploration Strategies for Reinforcement Learning via an Option Framework

Published 5 Oct 2023 in cs.LG | (2310.03342v2)

Abstract: In this paper, a unified framework for exploration in reinforcement learning (RL) is proposed based on an option-critic model. The proposed framework learns to integrate a set of diverse exploration strategies so that the agent can adaptively select the most effective exploration strategy over time to realize a relevant exploration-exploitation trade-off for each given task. The effectiveness of the proposed exploration framework is demonstrated by various experiments in the MiniGrid and Atari environments.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (29)
  1. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732, 2017.
  2. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  3. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507–517. PMLR, 2020.
  4. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016.
  5. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, Vol. 47:253–279, 2012. doi: 10.1613/jair.3912. URL http://arxiv.org/abs/1207.4708. cite arxiv:1207.4708.
  6. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.
  7. Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018.
  8. Temporally-extended {{\{{\\\backslash\epsilon}}\}}-greedy exploration. arXiv preprint arXiv:2006.01782, 2020.
  9. {{\{{\\\backslash\epsilon}}\}}-bmc: A bayesian ensemble approach to epsilon-greedy exploration in model-free reinforcement learning. arXiv preprint arXiv:2007.00869, 2020.
  10. Reinforcement learning with deep energy-based policies. arXiv preprint arXiv:1702.08165, 2017, 2017.
  11. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine learning, pp. 1861–1870. PMLR, 2018.
  12. Diversity actor-critic: Sample-aware entropy regularization for sample-efficient exploration. In International Conference on Machine Learning, 2021a.
  13. A max-min entropy framework for reinforcement learning. In Advances in Neural Information Processing Systems, 2021b.
  14. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017.
  15. Population-guided parallel policy search for reinforcement learning. In International Conference on Learning Representations, 2020.
  16. An adaptive entropy regularization framework for multi-agent reinforcement learning. In International Conference on Machine Learning, 2023.
  17. A variational approach to mutual information-based coordination for multi-agent reinforcement learning. In Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023.
  18. Flexible option learning. Advances in Neural Information Processing Systems, 34:4632–4646, 2021.
  19. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.
  20. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  21. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, 29, 2016.
  22. Count-based exploration with neural density models. In International Conference on Machine Learning, pp. 2721–2730. PMLR, 2017.
  23. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL http://jmlr.org/papers/v22/20-1364.html.
  24. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  25. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814, 2015.
  26. Reinforcement Learning: An Introduction. MIT Press, 2018.
  27. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181–211, 1999.
  28. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  29. Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 34:29021–29033, 2021.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up