Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play (2312.04549v1)

Published 7 Dec 2023 in cs.RO, cs.AI, cs.CV, cs.LG, cs.SY, and eess.SY

Abstract: Learning from unstructured and uncurated data has become the dominant paradigm for generative approaches in language and vision. Such unstructured and unguided behavior data, commonly known as play, is also easier to collect in robotics but much more difficult to learn from due to its inherently multimodal, noisy, and suboptimal nature. In this paper, we study this problem of learning goal-directed skill policies from unstructured play data which is labeled with language in hindsight. Specifically, we leverage advances in diffusion models to learn a multi-task diffusion model to extract robotic skills from play data. Using a conditional denoising diffusion process in the space of states and actions, we can gracefully handle the complexity and multimodality of play data and generate diverse and interesting robot behaviors. To make diffusion models more useful for skill learning, we encourage robotic agents to acquire a vocabulary of skills by introducing discrete bottlenecks into the conditional behavior generation process. In our experiments, we demonstrate the effectiveness of our approach across a wide variety of environments in both simulation and the real world. Results visualizations and videos at https://play-fusion.github.io

Definition Search Book Streamline Icon: https://streamlinehq.com
References (78)
  1. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999.
  2. Automated learning and discovery state-of-the-art and research topics in a rapidly growing field. Ai Magazine, 20(3):78–78, 1999.
  3. M. Pickett and A. G. Barto. Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In ICML, volume 19, pages 506–513, 2002.
  4. The option-critic architecture. In AAAI, 2017.
  5. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1999.
  6. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 2016.
  7. Reinforcement learning with sequences of motion primitives for robust manipulation. Transactions on Robotics, 2012.
  8. J. Kober and J. Peters. Learning motor primitives for robotics. In ICRA, 2009.
  9. Skill learning and task outcome prediction for manipulation. In ICRA, 2011.
  10. Accelerating robotic reinforcement learning via parameterized action primitives. NeurIPS, 2021.
  11. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In ICRA, 2022.
  12. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188–204. PMLR, 2021.
  13. Hierarchical neural dynamic policies. RSS, 2021.
  14. Cross-domain transfer via semantic skill imitation. arXiv preprint arXiv:2212.07407, 2022.
  15. Diversity is all you need: learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
  16. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019.
  17. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG), 39(4):39–1, 2020.
  18. T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pages 8624–8633. PMLR, 2020.
  19. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418–3428. PMLR, 2019.
  20. Dynamics-aware embeddings. arXiv preprint arXiv:1908.09357, 2019.
  21. Learning latent plans from play. arXiv preprint arXiv:1903.01973, 2019.
  22. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019.
  23. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047, 2022.
  24. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  25. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  26. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  27. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  28. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  29. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022.
  30. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022.
  31. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
  32. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573, 2023.
  33. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
  34. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022.
  35. Transporter networks: Rearranging the visual world for robotic manipulation. CoRL, 2020.
  36. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages 894–906. PMLR, 2022.
  37. L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094–8. Citeseer, 1993.
  38. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
  39. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019.
  40. Retrieval-augmented reinforcement learning. In International Conference on Machine Learning, pages 7740–7765. PMLR, 2022.
  41. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021.
  42. C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 40:105, 2020.
  43. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022.
  44. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  45. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303–1315. PMLR, 2022.
  46. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4):11205–11212, 2022.
  47. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023.
  48. C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020.
  49. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  50. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.
  51. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint arXiv:2210.15629, 2022.
  52. Behavior transformers: Cloning k𝑘kitalic_k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022.
  53. Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551, 2018.
  54. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
  55. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111, 2023.
  56. Structdiffusion: Language-guided 304 creation of physically-valid structures using unseen objects. arXiv preprint arXiv:2211.04604, 305:2, 2022.
  57. Mastering the game of go without human knowledge. Nature, 2017.
  58. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
  59. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, Feb. 2015.
  60. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, jun 2013.
  61. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273–1286, 2021.
  62. Masked trajectory models for prediction, representation, and control. arXiv preprint arXiv:2305.02968, 2023.
  63. Vector quantized models for planning. In International Conference on Machine Learning, pages 8302–8313. PMLR, 2021.
  64. Neural discrete representation learning. In NeurIPS, pages 6309–6318, 2017.
  65. N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
  66. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385.
  67. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
  68. Goal-conditioned imitation learning. arXiv preprint arXiv:1906.05838, 2019.
  69. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  70. Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857–16867, 2020.
  71. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018.
  72. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  73. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022.
  74. Calvin. https://github.com/mees/calvin/.
  75. Diffusion policy. https://github.com/real-stanford/diffusion_policy.
  76. Relay policy learning environments. https://github.com/google-research/relay-policy-learning/.
  77. Cliport. https://github.com/cliport/cliport.
  78. From play to policy: Conditional behavior generation from uncurated robot data. https://github.com/jeffacce/play-to-policy.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Lili Chen (34 papers)
  2. Shikhar Bahl (18 papers)
  3. Deepak Pathak (91 papers)
Citations (34)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play