Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Offline Goal-Conditioned Reinforcement Learning for Shape Control of Deformable Linear Objects (2403.10290v1)

Published 15 Mar 2024 in cs.RO, cs.SY, and eess.SY

Abstract: Deformable objects present several challenges to the field of robotic manipulation. One of the tasks that best encapsulates the difficulties arising due to non-rigid behavior is shape control, which requires driving an object to a desired shape. While shape-servoing methods have been shown successful in contexts with approximately linear behavior, they can fail in tasks with more complex dynamics. We investigate an alternative approach, using offline RL to solve a planar shape control problem of a Deformable Linear Object (DLO). To evaluate the effect of material properties, two DLOs are tested namely a soft rope and an elastic cord. We frame this task as a goal-conditioned offline RL problem, and aim to learn to generalize to unseen goal shapes. Data collection and augmentation procedures are proposed to limit the amount of experimental data which needs to be collected with the real robot. We evaluate the amount of augmentation needed to achieve the best results, and test the effect of regularization through behavior cloning on the TD3+BC algorithm. Finally, we show that the proposed approach is able to outperform a shape-servoing baseline in a curvature inversion experiment.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li, et al., “Challenges and outlook in robotic manipulation of deformable objects,” IEEE Robotics & Automation Magazine, vol. 29, no. 3, pp. 67–77, 2022.
  2. D. Berenson, “Manipulation of deformable objects without modeling and simulating deformation,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4525–4532, IEEE, 2013.
  3. M. Aranda, J. A. Corrales Ramon, Y. Mezouar, A. Bartoli, and E. Özgür, “Monocular visual shape tracking and servoing for isometrically deforming objects,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7542–7549, 2020.
  4. M. Shetab-Bushehri, M. Aranda, Y. Mezouar, and E. Özgür, “As-rigid-as-possible shape servoing,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3898–3905, 2022.
  5. M. Shetab-Bushehri, M. Aranda, Y. Mezouar, and E. Özgür, “Lattice-based shape tracking and servoing of elastic objects,” IEEE Transactions on Robotics, 2023.
  6. J. Matas, S. James, and A. J. Davison, “Sim-to-real reinforcement learning for deformable object manipulation,” in Conference on Robot Learning, pp. 734–743, 2018.
  7. R. Jangir, G. Alenya, and C. Torras, “Dynamic cloth manipulation with deep reinforcement learning,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4630–4636, IEEE, 2020.
  8. M. Daniel, A. Magassouba, M. Aranda, L. Lequièvre, J. A. Corrales Ramón, R. Iglesias Rodriguez, and Y. Mezouar, “Multi actor-critic ddpg for robot action space decomposition: A framework to control large 3d deformation of soft linear objects,” IEEE Robotics and Automation Letters, vol. 9, no. 2, pp. 1318–1325, 2024.
  9. J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey,” The International Journal of Robotics Research, vol. 37, no. 7, pp. 688–716, 2018.
  10. S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020.
  11. M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, et al., “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” arXiv preprint arXiv:1802.09464, 2018.
  12. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” Advances in neural information processing systems, vol. 30, 2017.
  13. M. Liu, M. Zhu, and W. Zhang, “Goal-conditioned reinforcement learning: Problems and solutions,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 (L. D. Raedt, ed.), pp. 5502–5511, International Joint Conferences on Artificial Intelligence Organization, 7 2022. Survey Track.
  14. S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforcement learning,” Advances in neural information processing systems, vol. 34, pp. 20132–20145, 2021.
  15. K. Almaghout, A. Cherubini, and A. Klimchik, “Robotic co-manipulation of deformable linear objects for large deformation tasks,” Robotics and Autonomous Systems, p. 104652, 2024.
  16. J. Zhu, B. Navarro, P. Fraisse, A. Crosnier, and A. Cherubini, “Dual-arm robotic manipulation of flexible cables,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 479–484, 2018.
  17. S. Jin, C. Wang, and M. Tomizuka, “Robust deformation model approximation for robotic cable manipulation,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6586–6593, IEEE, 2019.
  18. R. Lagneau, A. Krupa, and M. Marchal, “Automatic shape control of deformable wires based on model-free visual servoing,” IEEE Robotics and Automation Letters, vol. 5, no. 4, pp. 5252–5259, 2020.
  19. B. Balaguer and S. Carpin, “Combining imitation and reinforcement learning to fold deformable planar objects,” in 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1405–1412, IEEE, 2011.
  20. Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel, “Learning to Manipulate Deformable Objects without Demonstrations,” in Proceedings of Robotics: Science and Systems, (Corvalis, Oregon, USA), July 2020.
  21. H. Han, G. Paul, and T. Matsubara, “Model-based reinforcement learning approach for deformable linear object manipulation,” in 2017 13th IEEE Conference on Automation Science and Engineering (CASE), pp. 750–755, IEEE, 2017.
  22. R. Laezza, M. Shetab-Bushehri, E. Ozgür, Y. Mezouar, and Y. Karayiannidis, “Offline reinforcement learning for shape control of deformable linear objects from limited real data,” 3rd ICRA Workshop on Representing and Manipulating Deformable Objects, 2023.
  23. Y. Niu, S. Jin, Z. Zhang, J. Zhu, D. Zhao, and L. Zhang, “Goats: Goal sampling adaptation for scooping with curriculum reinforcement learning,” arXiv preprint arXiv:2303.05193, 2023.
  24. S. Zhaole, J. Zhu, and R. B. Fisher, “Dexdlo: Learning goal-conditioned dexterous policy for dynamic manipulation of deformable linear objects,” arXiv preprint arXiv:2312.15204, 2023.
  25. P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg, “Learning rope manipulation policies using dense object descriptors trained on synthetic depth data,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9411–9418, IEEE, 2020.
  26. M. Yan, Y. Zhu, N. Jin, and J. Bohg, “Self-supervised learning of state estimation for manipulating deformable linear objects,” IEEE robotics and automation letters, vol. 5, no. 2, pp. 2372–2379, 2020.
  27. S. Huo, A. Duan, C. Li, P. Zhou, W. Ma, H. Wang, and D. Navarro-Alarcon, “Keypoint-based planar bimanual shaping of deformable linear objects under environmental constraints with hierarchical action framework,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 5222–5229, 2022.
  28. A. Caporali, P. Kicki, K. Galassi, R. Zanella, K. Walas, and G. Palli, “Deformable linear objects manipulation with online model parameters estimation,” IEEE Robotics and Automation Letters, 2024.
  29. R. Laezza and Y. Karayiannidis, “Learning shape control of elastoplastic deformable linear objects,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4438–4444, 2021.
  30. Y. Yang, J. A. Stork, and T. Stoyanov, “Learning differentiable dynamics models for shape control of deformable linear objects,” Robotics and Autonomous Systems, vol. 158, 2022.
  31. M. Yu, K. Lv, H. Zhong, S. Song, and X. Li, “Global model learning for large deformation control of elastic deformable linear objects: An efficient and adaptive approach,” IEEE Transactions on Robotics, vol. 39, no. 1, pp. 417–436, 2022.
  32. T. Tang and M. Tomizuka, “Track deformable objects from point clouds with structure preserved registration,” The International Journal of Robotics Research, vol. 41, no. 6, pp. 599–614, 2022.
  33. A. Escande, N. Mansard, and P.-B. Wieber, “Hierarchical quadratic programming: Fast online humanoid-robot motion generation,” The International Journal of Robotics Research, vol. 33, no. 7, pp. 1006–1028, 2014.
  34. M. Alexa, D. Cohen-Or, and D. Levin, “As-rigid-as-possible shape interpolation,” in Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 157–164, 2000.
  35. R. Yang, L. Yong, X. Ma, H. Hu, C. Zhang, and T. Zhang, “What is essential for unseen goal generalization of offline goal-conditioned rl?,” in International Conference on Machine Learning, pp. 39543–39571, PMLR, 2023.
  36. S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in International conference on machine learning, pp. 1587–1596, PMLR, 2018.
  37. S. Sinha, H. Bharadhwaj, A. Srinivas, and A. Garg, “D2rl: Deep dense architectures in reinforcement learning,” arXiv preprint arXiv:2010.09163, 2020.
  38. T. Seno and M. Imai, “d3rlpy: An offline deep reinforcement learning library,” Journal of Machine Learning Research, vol. 23, no. 315, pp. 1–20, 2022.
  39. A. Kumar, A. Singh, S. Tian, C. Finn, and S. Levine, “A workflow for offline model-free robotic reinforcement learning,” in Conference on Robot Learning, pp. 417–428, PMLR, 2022.
  40. P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep reinforcement learning that matters,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now