Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Diffusion-Reward Adversarial Imitation Learning (2405.16194v4)

Published 25 May 2024 in cs.LG, cs.AI, and cs.RO

Abstract: Imitation learning aims to learn a policy from observing expert demonstrations without access to reward signals from environments. Generative adversarial imitation learning (GAIL) formulates imitation learning as adversarial learning, employing a generator policy learning to imitate expert behaviors and discriminator learning to distinguish the expert demonstrations from agent trajectories. Despite its encouraging results, GAIL training is often brittle and unstable. Inspired by the recent dominance of diffusion models in generative modeling, we propose Diffusion-Reward Adversarial Imitation Learning (DRAIL), which integrates a diffusion model into GAIL, aiming to yield more robust and smoother rewards for policy learning. Specifically, we propose a diffusion discriminative classifier to construct an enhanced discriminator, and design diffusion rewards based on the classifier's output for policy learning. Extensive experiments are conducted in navigation, manipulation, and locomotion, verifying DRAIL's effectiveness compared to prior imitation learning methods. Moreover, additional experimental results demonstrate the generalizability and data efficiency of DRAIL. Visualized learned reward functions of GAIL and DRAIL suggest that DRAIL can produce more robust and smoother rewards. Project page: https://nturobotlearninglab.github.io/DRAIL/

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
  2. Wasserstein generative adversarial networks. In International Conference on Machine Learning, 2017.
  3. Playing hard exploration games by watching youtube. In Neural Information Processing Systems, 2018.
  4. Variational inference: A review for statisticians. Journal of the American statistical Association, 2017.
  5. Diffusion model-augmented behavioral cloning. In International Conference on Machine Learning, 2024.
  6. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
  7. Deep reinforcement learning from human preferences. In Neural Information Processing Systems, 2017.
  8. Primal wasserstein imitation learning. arXiv preprint arXiv:2006.04678, 2020.
  9. Diffusion models beat gans on image synthesis. In Neural Information Processing Systems, 2021.
  10. Implicit behavioral cloning. In Conference on Robotic Learning, 2022.
  11. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
  12. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
  13. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 2015.
  14. Generative adversarial nets. In Neural Information Processing Systems, 2014.
  15. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330, 2022.
  16. Imitation learning with concurrent actions in 3d games. In IEEE Conference on Computational Intelligence and Games, 2018.
  17. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  18. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2018.
  19. Generative adversarial imitation learning. In Neural Information Processing Systems, 2016.
  20. Denoising diffusion probabilistic models. In Neural Information Processing Systems, 2020.
  21. Imitation learning: A survey of learning methods. ACM Computing Surveys, 2017.
  22. Augmenting gail with bc for sample efficient imitation learning. In Conference on Robot Learning, 2021.
  23. Variational diffusion models. In Neural Information Processing Systems, 2021.
  24. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
  25. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
  26. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations, 2024.
  27. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Machine Learning, 2019a.
  28. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019b.
  29. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2019.
  30. Generalizable imitation learning from observation via inferring goal proximity. In Neural Information Processing Systems, 2021.
  31. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871, 2018.
  32. Your diffusion model is secretly a zero-shot classifier. In IEEE International Conference on Computer Vision, 2023.
  33. Learning to drive by imitation: An overview of deep behavior cloning methods. IEEE Transactions on Intelligent Vehicles, 2020.
  34. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
  35. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021.
  36. What matters for adversarial imitation learning? In Neural Information Processing Systems, 2021.
  37. An algorithmic perspective on imitation learning. Foundations and Trends® in Robotics, 2018.
  38. Imitating human behaviour with diffusion models. In International Conference on Learning Representations, 2023.
  39. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464, 2018.
  40. Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Neural Information Processing Systems, 1989.
  41. Recent advances in robot learning from demonstration. Annual review of control, robotics, and autonomous systems, 2020.
  42. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
  43. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 2015.
  44. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2011.
  45. Stefan Schaal. Learning from demonstration. In Neural Information Processing Systems, 1997.
  46. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
  47. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
  48. Denoising diffusion implicit models. In International Conference on Machine Learning, 2021.
  49. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning, 2023.
  50. Apprenticeship learning using linear programming. In Proceedings of the 25th international conference on Machine learning, 2008.
  51. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, 2018.
  52. Generative adversarial imitation from observation. In International Conference on Machine Learning, 2019.
  53. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022.
  54. Diffail: Diffusion adversarial imitation learning. arXiv preprint arXiv:2312.06348, 2023.
  55. Diffail: Diffusion adversarial imitation learning. In AAAI Conference on Artificial Intelligence, 2024.
  56. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems, 2023.
  57. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008.
  58. Task-relevant adversarial imitation learning. In Conference on Robotic Learning, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Chun-Mao Lai (5 papers)
  2. Hsiang-Chun Wang (2 papers)
  3. Ping-Chun Hsieh (31 papers)
  4. Yu-Chiang Frank Wang (88 papers)
  5. Min-Hung Chen (41 papers)
  6. Shao-Hua Sun (22 papers)
Citations (3)
View on Semantic Scholar