Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtasks (2401.14226v1)

Published 25 Jan 2024 in cs.LG

Abstract: Improving sample efficiency is central to Reinforcement Learning (RL), especially in environments where the rewards are sparse. Some recent approaches have proposed to specify reward functions as manually designed or learned reward structures whose integrations in the RL algorithms are claimed to significantly improve the learning efficiency. Manually designed reward structures can suffer from inaccuracy and existing automatically learning methods are often computationally intractable for complex tasks. The integration of inaccurate or partial reward structures in RL algorithms fail to learn optimal policies. In this work, we propose an RL algorithm that can automatically structure the reward function for sample efficiency, given a set of labels that signify subtasks. Given such minimal knowledge about the task, we train a high-level policy that selects optimal sub-tasks in each state together with a low-level policy that efficiently learns to complete each sub-task. We evaluate our algorithm in a variety of sparse-reward environments. The experiment results show that our approach significantly outperforms the state-of-art baselines as the difficulty of the task increases.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Hindsight Experience Replay. 5048–5058.
  2. Spatio-temporal abstractions in reinforcement learning through neural encoding. (2016).
  3. libalf: The automata learning framework. In Proceedings of the 22nd International Conference on Computer Aided Verification. 360–364.
  4. Exploration in Reward Machines with Low Regret. In International Conference on Artificial Intelligence and Statistics, Vol. 206. 4114–4146.
  5. Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning. In 2020 IEEE International Conference on Robotics and Automation. 10349–10355.
  6. LTLf/LDLf Non-Markovian Rewards. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. 1771–1778.
  7. LTL and Beyond: Formal Languages for Reward Function Specification in Reinforcement Learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. 6065–6073.
  8. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning. PMLR, 1430–1440.
  9. Toward A Thousand Lights: Decentralized Deep Reinforcement Learning for Large-Scale Traffic Signal Control. In The Thirty-Fourth AAAI Conference on Artificial Intelligence. 3414–3421.
  10. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research 74 (2022), 1159–1199.
  11. Reinforcement learning with stochastic reward machines. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 6429–6436.
  12. Multi-Agent Intention Progression with Reward Machines. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. 215–222.
  13. Reinforcement-Learning-Based Robust Resource Management for Multi-Radio Systems. Sensors 23, 10 (2023), 4821.
  14. Thomas G Dietterich. 2000. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of artificial intelligence research 13 (2000), 227–303.
  15. Generalizing goal-conditioned reinforcement learning with variational causal reasoning. Advances in Neural Information Processing Systems 35 (2022), 26532–26548.
  16. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 35603–35620.
  17. Curriculum-guided hindsight experience replay. Advances in neural information processing systems 32 (2019).
  18. Induction of subgoal automata for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 3890–3897.
  19. A survey on interpretable reinforcement learning. arXiv preprint arXiv:2112.13112 (2021).
  20. E. Mark Gold. 1978. Complexity of Automaton Identification from Given Data. Inf. Control. 37, 3 (1978), 302–320.
  21. Survey of Reinforcement Learning based on Human Prior Knowledge. Journal of Uncertain Systems 15, 01 (2022), 2230001.
  22. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems 35 (2022), 15281–15295.
  23. DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence. 7647–7656.
  24. Reinforcement Learning for Temporal Logic Control Synthesis with Probabilistic Satisfaction Guarantees. In 58th IEEE Conference on Decision and Control. 5338–5343.
  25. Explainability in deep reinforcement learning. Knowledge-Based Systems 214 (2021), 106685.
  26. Rodrigo Toro Icarte. 2022. Reward Machines. Ph.D. Dissertation. University of Toronto, Canada.
  27. Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning, Vol. 80. 2112–2121.
  28. Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning. J. Artif. Intell. Res. 73 (2022), 173–208.
  29. Learning concise models from long execution traces. In 57th ACM/IEEE Design Automation Conference (DAC). 1–6.
  30. Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. 3854–3860.
  31. Temporal-Logic-Based Reward Shaping for Continuing Reinforcement Learning Tasks. In Thirty-Fifth AAAI Conference on Artificial Intelligence1. 7995–8003.
  32. Creativity of ai: Automatic symbolic option discovery for facilitating deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 7042–7050.
  33. Predicting chemical structure using reinforcement learning with a stack-augmented conditional variational autoencoder. J. Cheminformatics 14, 1 (2022), 83.
  34. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems 29 (2016).
  35. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299 (2022).
  36. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2970–2977.
  37. George H Mealy. 1955. A method for synthesizing sequential circuits. The Bell System Technical Journal 34, 5 (1955), 1045–1079.
  38. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529–533.
  39. Model-based reinforcement learning: A survey. Foundations and Trends® in Machine Learning 16, 1 (2023), 1–118.
  40. Reward Machines for Cooperative Multi-Agent Reinforcement Learning. In Proceedings of the 2021 International Conference on Autonomous Agents and Multiagent Systems. 934–942.
  41. Daniel Neider and Nils Jansen. 2013. Regular model checking using solver technologies and automata learning. In NASA Formal Methods Symposium. Springer, 16–31.
  42. José Oncina and Pedro Garcia. 1992. Inferring regular languages in polynomial updated time. In Pattern recognition and image analysis: selected papers from the IVth Spanish Symposium. World Scientific, 49–61.
  43. Ronald Parr and Stuart Russell. 1997. Reinforcement learning with hierarchies of machines. Advances in neural information processing systems 10 (1997).
  44. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR) 54, 5 (2021), 1–35.
  45. Amir Pnueli. 1977. The Temporal Logic of Programs. In 18th Annual Symposium on Foundations of Computer Science, Providence. 46–57.
  46. Erika Puiutta and Eric MSP Veith. 2020. Explainable reinforcement learning: A survey. In International cross-domain conference for machine learning and knowledge extraction. 77–95.
  47. Haoyuan Sun and Feng Wu. 2023. Less Is More: Refining Datasets for Offline Reinforcement Learning with Reward Machines. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems. 1239–1247.
  48. Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
  49. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 112, 1-2 (1999), 181–211.
  50. Curiosity in exploring chemical spaces: intrinsic rewards for molecular reinforcement learning. Mach. Learn. Sci. Technol. 3, 3 (2022), 35008.
  51. Learning reward machines for partially observable reinforcement learning. Advances in neural information processing systems 32 (2019).
  52. Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning 8 (1992), 279–292.
  53. Joint Inference of Reward Machines and Policies for Reinforcement Learning. In Proceedings of the Thirtieth International Conference on Automated Planning and Scheduling. 590–598.
  54. Reinforcement Learning with Knowledge Representation and Reasoning: A Brief Survey. CoRR abs/2304.12090 (2023).
  55. Goal space abstraction in hierarchical reinforcement learning via set-based reachability analysis. In 2023 IEEE International Conference on Development and Learning (ICDL). IEEE, 423–428.
  56. Reinforcement Learning for Radio Resource Management in RAN Slicing: A Survey. IEEE Commun. Mag. 61, 2 (2023), 118–124.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Shuai Han (26 papers)
  2. Mehdi Dastani (27 papers)
  3. Shihan Wang (15 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets