Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Imitation Bootstrapped Reinforcement Learning (2311.02198v6)

Published 3 Nov 2023 in cs.LG and cs.AI

Abstract: Despite the considerable potential of reinforcement learning (RL), robotic control tasks predominantly rely on imitation learning (IL) due to its better sample efficiency. However, it is costly to collect comprehensive expert demonstrations that enable IL to generalize to all possible scenarios, and any distribution shift would require recollecting data for finetuning. Therefore, RL is appealing if it can build upon IL as an efficient autonomous self-improvement procedure. We propose imitation bootstrapped reinforcement learning (IBRL), a novel framework for sample-efficient RL with demonstrations that first trains an IL policy on the provided demonstrations and then uses it to propose alternative actions for both online exploration and bootstrapping target values. Compared to prior works that oversample the demonstrations or regularize RL with an additional imitation loss, IBRL is able to utilize high quality actions from IL policies since the beginning of training, which greatly accelerates exploration and training efficiency. We evaluate IBRL on 6 simulation and 3 real-world tasks spanning various difficulty levels. IBRL significantly outperforms prior methods and the improvement is particularly more prominent in harder tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023.
  2. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning (ICML), 2023.
  3. Hydra: Hybrid robot actions for imitation learning. In Conference on Robot Learning (CoRL), 2023.
  4. RT-1: Robotics Transformer for Real-World Control at Scale. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  5. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations (ICLR), 2021.
  6. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  7. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
  8. Human-level play in the game of “diplomacy” by combining language models with strategic reasoning. Science, 378(6624):1067–1074, 2022.
  9. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), 2018.
  10. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018.
  11. Mastering atari with discrete world models. In International Conference on Learning Representations (ICLR), 2021.
  12. Watch and match: Supercharging imitation with regularized optimal transport. In Conference on Robot Learning (CoRL), 2022.
  13. Temporal difference learning for model predictive control. In International Conference on Machine Learning (ICML), 2022.
  14. MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations. In International Conference on Learning Representations (ICLR), 2023a.
  15. On pre-training for visuo-motor control: Revisiting a learning-from-scratch baseline. In International Conference on Machine Learning (ICML), 2023b.
  16. Deep residual learning for image recognition. In Conference on Computer vision and Pattern Recognition (CVPR), 2016.
  17. Deep Q-learning from demonstrations. In AAAI Conference on Artificial Intelligence, 2018.
  18. Dropout Q-functions for doubly efficient reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
  19. Vision-based manipulators need to also see from their hands. In International Conference on Learning Representations (ICLR), 2022.
  20. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
  21. Pre-Training for Robots: Offline RL Enables Learning New Tasks in a Handful of Trials. Robotics: Science and Systems (RSS), 2022.
  22. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), 2016.
  23. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
  24. A graph placement methodology for fast chip design. Nature, 594(7862):207–212, 2021.
  25. Overcoming exploration in reinforcement learning with demonstrations. In International Conference on Robotics and Automation (ICRA), 2018.
  26. Accelerating online reinforcement learning with offline datasets. arXiv:2006.09359, 2020.
  27. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
  28. {SQIL}: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=S1xKd24twB.
  29. Goal conditioned imitation learning using score-based diffusion policies. In Robotics: Science and Systems, 2023.
  30. On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
  31. Masked world models for visual control. In Conference on Robot Learning (CoRL), 2022.
  32. RRL: resnet as representation for reinforcement learning. In Marina Meila and Tong Zhang (eds.), International Conference on Machine Learning (ICML), 2021.
  33. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, 2017.
  34. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research (JMLR), 15(1):1929–1958, 2014.
  35. Sebastian Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In M. Mozer, P. Smolensky, D. Touretzky, J. Elman, and A. Weigend (eds.), Proceedings of 4th Connectionist Models Summer School. Erlbaum Associates, June 1993.
  36. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv:1707.08817, 2017.
  37. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019.
  38. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  39. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
  40. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations (ICLR), 2022.
  41. Visual imitation made easy. In Conference on Robot Learning (CoRL), 2021.
  42. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
  43. Learning visual robotic control efficiently with contrastive pre-training and data augmentation. In International Conference on Intelligent Robots and Systems (IROS), 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Hengyuan Hu (22 papers)
  2. Suvir Mirchandani (17 papers)
  3. Dorsa Sadigh (162 papers)
Citations (17)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now