Papers
Topics
Authors
Recent
Search
2000 character limit reached

EduGym: An Environment and Notebook Suite for Reinforcement Learning Education

Published 17 Nov 2023 in cs.LG, cs.AI, cs.CY, and stat.ML | (2311.10590v2)

Abstract: Due to the empirical success of reinforcement learning, an increasing number of students study the subject. However, from our practical teaching experience, we see students entering the field (bachelor, master and early PhD) often struggle. On the one hand, textbooks and (online) lectures provide the fundamentals, but students find it hard to translate between equations and code. On the other hand, public codebases do provide practical examples, but the implemented algorithms tend to be complex, and the underlying test environments contain multiple reinforcement learning challenges at once. Although this is realistic from a research perspective, it often hinders educational conceptual understanding. To solve this issue we introduce EduGym, a set of educational reinforcement learning environments and associated interactive notebooks tailored for education. Each EduGym environment is specifically designed to illustrate a certain aspect/challenge of reinforcement learning (e.g., exploration, partial observability, stochasticity, etc.), while the associated interactive notebook explains the challenge and its possible solution approaches, connecting equations and code in a single document. An evaluation among RL students and researchers shows 86% of them think EduGym is a useful tool for reinforcement learning education. All notebooks are available from https://www.edugym.org/, while the full software package can be installed from https://github.com/RLG-Leiden/edugym.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Students’ perceptions of lecturing approaches: traditional versus interactive teaching. Advances in medical education and practice, pp.  229–241, 2017.
  2. Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018. https://github.com/openai/spinningup.
  3. MO-Gym: A library of multi-objective reinforcement learning environments. In Proceedings of the 34th Benelux Conference on Artificial Intelligence BNAIC/Benelearn 2022, 2022.
  4. A survey of exploration methods in reinforcement learning. arXiv preprint arXiv:2109.00157, 2021.
  5. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013.
  6. Richard Bellman. The theory of dynamic programming. Bulletin of the American Mathematical Society, 60(6):503–515, 1954.
  7. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  8. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  9. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.
  10. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004.
  11. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv:1912.01588, 2019.
  12. Intrinsically motivated goal-conditioned reinforcement learning: a short survey. 2021.
  13. Gymnasium robotics, 2023. http://github.com/Farama-Foundation/Gymnasium-Robotics.
  14. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414–419, 2022.
  15. A survey on policy search for robotics. Foundations and Trends® in Robotics, 2(1–2):1–142, 2013.
  16. Model-free risk-sensitive reinforcement learning. arXiv preprint arXiv:2111.02907, 2021.
  17. Mushroomrl: Simplifying reinforcement learning research. The Journal of Machine Learning Research, 22(1):5867–5871, 2021.
  18. First return, then explore. Nature, 590(7847):580–586, 2021.
  19. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47–53, 2022.
  20. Beobench: a toolkit for unified access to building simulations for reinforcement learning. In Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, pp.  374–382, 2022.
  21. Chainerrl: A deep reinforcement learning library. The Journal of Machine Learning Research, 22(1):3557–3570, 2021.
  22. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015.
  23. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
  24. Reinforcement learning algorithm for partially observable markov decision problems. Advances in neural information processing systems, 7, 1994.
  25. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998.
  26. ViZDoom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pp.  341–348, Santorini, Greece, Sep 2016. IEEE. doi: 10.1109/CIG.2016.7860433. The Best Paper Award.
  27. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013.
  28. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pp.  3053–3062. PMLR, 2018.
  29. Memory approaches to reinforcement learning in non-Markovian domains. Citeseer, 1992.
  30. RA McCallum. Efficient exploration in reinforcement learning with hidden state. In AAAI Fall Symposium on Model-directed Autonomous Systems. Citeseer, 1997.
  31. Playing atari with deep reinforcement learning, 2013.
  32. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning, 16(1):1–118, 2023.
  33. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13:103–130, 1993.
  34. Discovering the structure of a reactive environment by exploration. Advances in neural information processing systems, 2, 1989.
  35. Risk sensitive reinforcement learning. Advances in Neural Information Processing Systems, 11, 1998.
  36. OpenAI. Robogym. https://github.com/openai/robogym, 2020.
  37. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  38. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
  39. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygf-kSYwH.
  40. Memory gym: Partially observable challenges to memory-based agents. In The Eleventh International Conference on Learning Representations, 2023.
  41. Deep reinforcement learning for de novo drug design. Science advances, 4(7):eaap7885, 2018.
  42. Stable baselines3, 2019. https://github.com/DLR-RM/stable-baselines3.
  43. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019.
  44. On-line Q-learning using connectionist systems, volume 37. University of Cambridge, Department of Engineering Cambridge, UK, 1994.
  45. Universal value function approximators. In International conference on machine learning, pp.  1312–1320. PMLR, 2015.
  46. Proximal policy optimization algorithms, 2017.
  47. David Silver. Introduction to reinforcement learning. 2015. www.deepmind.com/learning-resources/introduction-to-reinforcement-learning-with-david-silver.
  48. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140–1144, 2018.
  49. Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, volume 2000, pp.  943–950, 2000.
  50. Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pp.  216–224. Elsevier, 1990.
  51. Reinforcement learning: An introduction. MIT press, 2018.
  52. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  53. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:15032–15043, 2021.
  54. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.  5026–5033. IEEE, 2012.
  55. Gymnasium, March 2023. URL https://zenodo.org/record/8127025.
  56. Dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.org/10.1016/j.simpa.2020.100022. URL https://www.sciencedirect.com/science/article/pii/S2665963820300099.
  57. Reinforcement learning lecture series. 2021. www.deepmind.com/learning-resources/reinforcement-learning-lecture-series-2021.
  58. Q-learning. Machine learning, 8:279–292, 1992.
  59. Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
  60. Spriteworld: A flexible, configurable reinforcement learning environment, 2019. https://github.com/deepmind/spriteworld/.
  61. Reinforcement learning specialization. 2023. https://www.coursera.org/specializations/reinforcement-learning.
  62. Reinforcement learning. Adaptation, learning, and optimization, 12(3):729, 2012.
  63. Flow: Architecture and benchmarking for reinforcement learning in traffic control. arXiv preprint arXiv:1710.05465, 10, 2017.
  64. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https://arxiv.org/abs/1910.10897.
  65. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pp.  321–384, 2021.
  66. Optimizing chemical reactions with deep reinforcement learning. ACS central science, 3(12):1337–1344, 2017.
  67. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1–10, 2019.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up