Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Curricular Subgoals for Inverse Reinforcement Learning (2306.08232v1)

Published 14 Jun 2023 in cs.LG and cs.AI

Abstract: Inverse Reinforcement Learning (IRL) aims to reconstruct the reward function from expert demonstrations to facilitate policy learning, and has demonstrated its remarkable success in imitation learning. To promote expert-like behavior, existing IRL methods mainly focus on learning global reward functions to minimize the trajectory difference between the imitator and the expert. However, these global designs are still limited by the redundant noise and error propagation problems, leading to the unsuitable reward assignment and thus downgrading the agent capability in complex multi-stage tasks. In this paper, we propose a novel Curricular Subgoal-based Inverse Reinforcement Learning (CSIRL) framework, that explicitly disentangles one task with several local subgoals to guide agent imitation. Specifically, CSIRL firstly introduces decision uncertainty of the trained agent over expert trajectories to dynamically select subgoals, which directly determines the exploration boundary of different task stages. To further acquire local reward functions for each stage, we customize a meta-imitation objective based on these curricular subgoals to train an intrinsic reward generator. Experiments on the D4RL and autonomous driving benchmarks demonstrate that the proposed methods yields results superior to the state-of-the-art counterparts, as well as better interpretability. Our code is available at https://github.com/Plankson/CSIRL.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Hindsight experience replay. In Annual Conference on Neural Information Processing Systems, 2017.
  2. Szilárd Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 23(2):740–759, 2020.
  3. Curriculum learning. In International Conference on Machine Learning, 2009.
  4. Modeling human driving behavior through generative adversarial imitation learning. IEEE Transactions on Intelligent Transportation Systems, 2022.
  5. Alex J Chan and Mihaela van der Schaar. Scalable bayesian inverse reinforcement learning. arXiv preprint arXiv:2102.06483, 2021.
  6. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, 2021.
  7. Causal confusion in imitation learning. In Annual Conference on Neural Information Processing Systems, 2019.
  8. Goal-conditioned imitation learning. In Annual Conference on Neural Information Processing Systems, 2019.
  9. One-shot imitation learning. In Annual Conference on Neural Information Processing Systems, 2017.
  10. Dynamic conditional imitation learning for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23(12):22988–23001, 2022.
  11. Curriculum-guided hindsight experience replay. In Annual Conference on Neural Information Processing Systems, 2019.
  12. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, 2018.
  13. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2017.
  14. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
  15. Iq-learn: Inverse soft-q learning for imitation. In Annual Conference on Neural Information Processing Systems, 2021.
  16. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
  17. Generative adversarial imitation learning. In Annual Conference on Neural Information Processing Systems, 2016.
  18. Optimistic planning of deterministic systems. In European Workshop on Reinforcement Learning, 2008.
  19. Mohamed Khalil Jabri. Robot manipulation learning using generative adversarial imitation learning. In International Joint Conference on Artificial Intelligence, 2021.
  20. Strictly batch imitation learning by energy-based distribution matching. In Annual Conference on Neural Information Processing Systems, 2020.
  21. Regularized inverse reinforcement learning. In International Conference on Learning Representations, 2020.
  22. Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference on Artificial Intelligence, 1993.
  23. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2018.
  24. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2019.
  25. Edouard Leurent. An environment for autonomous driving decision-making. https://github.com/eleurent/highway-env, 2018.
  26. Nonlinear inverse reinforcement learning with gaussian processes. Annual Conference on Neural Information Processing Systems, 24, 2011.
  27. Anomaly detection and correction of optimizing autonomous systems with inverse reinforcement learning. IEEE Transactions on Cybernetics, 2022.
  28. Active exploration for inverse reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2022.
  29. Learn goal-conditioned policy with intrinsic motivation for deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2022.
  30. Zero-shot reward specification via grounded natural language. In International Conference on Learning Representations, 2022.
  31. The driving school system: Learning basic driving skills from a teacher in a real car. IEEE Transactions on Intelligent Transportation Systems, 12(4):1135–1146, 2011.
  32. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.
  33. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
  34. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999.
  35. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
  36. Learning to select goals in automated planning with deep-q learning. Expert Systems with Applications, 202:117265, 2022.
  37. What matters for adversarial imitation learning? In Advances in Neural Information Processing Systems, 2021.
  38. Learning from trajectories via subgoal discovery. In Annual Conference on Neural Information Processing Systems, 2019.
  39. Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88–97, 1991.
  40. Temporal difference models: Model-free deep rl for model-based control. arXiv preprint arXiv:1802.09081, 2018.
  41. Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
  42. Automated curricula through setter-solver interactions. arXiv preprint arXiv:1909.12892, 2019.
  43. Toward the fundamental limits of imitation learning. In Annual Conference on Neural Information Processing Systems, 2020.
  44. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems, 2018.
  45. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11108, 2019.
  46. Efficient reductions for imitation learning. In Artificial Intelligence and Statistics, 2010.
  47. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics, 2011.
  48. Universal value function approximators. In International Conference on Machine Learning, 2015.
  49. Inverse reinforcement learning under noisy observations. arXiv preprint arXiv:1710.10116, 2017.
  50. Inverse reinforcement learning via nonparametric spatio-temporal subgoal modeling. Journal of Machine Learning Research, 19(69):1–45, 2018.
  51. Learning rewards from linguistic feedback. In AAAI Conference on Artificial Intelligence, 2021.
  52. Reinforcement learning with a disentangled universal value function for item recommendation. In AAAI Conference on Artificial Intelligence, 2021.
  53. Establishing style-oriented driver models by imitating human driving behaviors. IEEE Transactions on Intelligent Transportation Systems, 16(5):2522–2530, 2015.
  54. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, 2019.
  55. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, 2022.
  56. Objective-aware traffic simulation via inverse reinforcement learning. arXiv preprint arXiv:2105.09560, 2021.
  57. Li Zhou and Kevin Small. Inverse reinforcement learning with natural language goals. In AAAI Conference on Artificial Intelligence, 2021.
  58. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008.
  59. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning, 2010.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Shunyu Liu (48 papers)
  2. Yunpeng Qing (7 papers)
  3. Shuqi Xu (11 papers)
  4. Hongyan Wu (24 papers)
  5. Jiangtao Zhang (8 papers)
  6. Jingyuan Cong (2 papers)
  7. Tianhao Chen (14 papers)
  8. Yunfu Liu (1 paper)
  9. Mingli Song (163 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Plankson/CSIRL: Curricular Subgoal for Inverse Reinforcement Learning (12 stars)