Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks (2012.03385v4)

Published 6 Dec 2020 in cs.RO and cs.LG

Abstract: Rearranging and manipulating deformable objects such as cables, fabrics, and bags is a long-standing challenge in robotic manipulation. The complex dynamics and high-dimensional configuration spaces of deformables, compared to rigid objects, make manipulation difficult not only for multi-step planning, but even for goal specification. Goals cannot be as easily specified as rigid object poses, and may involve complex relative spatial relations such as "place the item inside the bag". In this work, we develop a suite of simulated benchmarks with 1D, 2D, and 3D deformable structures, including tasks that involve image-based goal-conditioning and multi-step deformable manipulation. We propose embedding goal-conditioning into Transporter Networks, a recently proposed model architecture for learning robotic manipulation that rearranges deep features to infer displacements that can represent pick and place actions. In simulation and in physical experiments, we demonstrate that goal-conditioned Transporter Networks enable agents to manipulate deformable structures into flexibly specified configurations without test-time visual anchors for target locations. We also significantly extend prior results using Transporter Networks for manipulating deformable objects by testing on tasks with 2D and 3D deformables. Supplementary material is available at https://berkeleyautomation.github.io/bags/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (69)
  1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  2. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba, “Hindsight Experience Replay,” in Neural Information Processing Systems (NeurIPS), 2017.
  3. B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A Survey of Robot Learning From Demonstration,” Robotics and Autonomous Systems, vol. 57, 2009.
  4. Y. Bai, W. Yu, and C. K. Liu, “Dexterous Manipulation of Cloth,” in European Association for Computer Graphics, 2016.
  5. B. Balaguer and S. Carpin, “Combining Imitation and Reinforcement Learning to Fold Deformable Planar Objects,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2011.
  6. D. Baraff and A. Witkin, “Large Steps in Cloth Simulation,” in ACM SIGGRAPH, 1998.
  7. D. Batra, A. X. Chang, S. Chernova, A. J. Davidson, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, M. Savva, and H. Su, “Rearrangement: A Challenge for Embodied AI,” arXiv preprint arXiv:2011.01975, 2020.
  8. C. M. Bishop, “Mixture Density Networks,” Aston University, 1994.
  9. J. Borras, G. Alenya, and C. Torras, “A Grasping-centered Analysis for Cloth Manipulation,” arXiv preprint arXiv:1906.08202, 2019.
  10. G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” 2016.
  11. E. Corona, G. Alenya, A. Gabas, and C. Torras, “Active Garment Recognition and Target Grasping Point Detection Using Deep Learning,” in Pattern Recognition, 2018.
  12. E. Coumans and Y. Bai, “PyBullet, a Python Module for Physics Simulation for Games, Robotics and Machine Learning,” http://pybullet.org, 2016–2020.
  13. M. Cusumano-Towner, A. Singh, S. Miller, J. F. O’Brien, and P. Abbeel, “Bringing Clothing Into Desired Configurations with Limited Perception,” in IEEE International Conference on Robotics and Automation (ICRA), 2011.
  14. A. Doumanoglou, A. Kargakos, T.-K. Kim, and S. Malassiotis, “Autonomous Active Recognition and Unfolding of Clothes Using Random Decision Forests and Probabilistic Planning,” in IEEE International Conference on Robotics and Automation (ICRA), 2014.
  15. F. Ebert, C. Finn, S. Dasari, A. Xie, A. Lee, and S. Levine, “Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control,” arXiv preprint arXiv:1812.00568, 2018.
  16. Z. Erickson, H. Clever, G. Turk, C. K. Liu, and C. Kemp, “Deep Haptic Model Predictive Control for Robot-Assisted Dressing,” in IEEE International Conference on Robotics and Automation (ICRA), 2018.
  17. Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp, “Assistive Gym: A Physics Simulation Framework for Assistive Robotics,” in IEEE International Conference on Robotics and Automation (ICRA), 2020.
  18. P. Florence, L. Manuelli, and R. Tedrake, “Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation,” in Conference on Robot Learning (CoRL), 2018.
  19. E. Frazzoli, M. A. Dahleh, and E. Feron, “Maneuver-based Motion Planning for Nonlinear Systems with Symmetries,” IEEE Transactions on Robotics (T-RO), 2005.
  20. A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. Gonzalez, N. Jamali, K. Yamane, S. Iba, and K. Goldberg, “Learning Dense Visual Correspondences in Simulation to Smooth and Fold Real Fabrics,” in IEEE International Conference on Robotics and Automation (ICRA), 2021.
  21. N. Hayashi, T. Suehiro, and S. Kudoh, “Planning Method for a Wrapping-With-Fabric Task Using Regrasping,” in IEEE International Conference on Robotics and Automation (ICRA), 2017.
  22. N. Hayashi, T. Tomizawa, T. Suehiro, and S. Kudoh, “Dual Arm Robot Fabric Wrapping Operation Using Target Lines,” in IEEE International Conference on Robotics and Biomimetics (ROBIO), 2014.
  23. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  24. R. B. Hellman, C. Tekin, M. van der Schaar, and V. J. Santos, “Functional Contour-following via Haptic Perception and Reinforcement Learning,” in IEEE Transactions on Haptics, 2018.
  25. J. Hopcroft, J. Kearney, and D. Krafft, “A Case Study of Flexible Object Manipulation,” in International Journal of Robotics Research (IJRR), 1991.
  26. R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg, “VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation,” in Robotics: Science and Systems (RSS), 2020.
  27. R. Jangir, G. Alenya, and C. Torras, “Dynamic Cloth Manipulation with Deep Reinforcement Learning,” in IEEE International Conference on Robotics and Automation (ICRA), 2020.
  28. D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine, “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation,” in Conference on Robot Learning (CoRL), 2018.
  29. H. Kazerooni and C. Foley, “A Robot Mechanism for Grapsing Sacks,” in IEEE Transactions on Automation Science and Engineering, 2005.
  30. D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” in International Conference on Learning Representations (ICLR), 2015.
  31. A. Kirchheim, M. Burwinkel, and W. Echelmeyer, “Automatic Unloading of Heavy Sacks From Containers,” in IEEE International Conference on Automation and Logistics (ICAL), 2008.
  32. Y. Kita, T. Ueshiba, E. S. Neo, and N. Kita, “A Method For Handling a Specific Part of Clothing by Dual Arms,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2009.
  33. ——, “Clothes State Recognition Using 3D Observed Data,” in IEEE International Conference on Robotics and Automation (ICRA), 2009.
  34. E. Klingbeil, D. Rao, B. Carpenter, V. Ganapathi, A. Y. Ng, and O. Khatib, “Grasping With Application to an Autonomous Checkout Robot,” in IEEE International Conference on Robotics and Automation (ICRA), 2011.
  35. R. Lee, D. Ward, A. Cosgun, V. Dasagi, P. Corke, and J. Leitner, “Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience,” in Conference on Robot Learning (CoRL), 2020.
  36. S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end Training of Deep Visuomotor Policies,” in Journal of Machine Learning Research (JMLR), 2016.
  37. X. Lin, Y. Wang, J. Olkin, and D. Held, “SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation,” in Conference on Robot Learning (CoRL), 2020.
  38. M. Lippi, P. Poklukar, M. C. Welle, A. Varava, H. Yin, A. Marino, and D. Kragic, “Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
  39. J. Long, E. Shelhamer, and T. Darrell, “Fully Convolutional Networks for Semantic Segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  40. J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg, “Learning Ambidextrous Robot Grasping Policies,” Science Robotics, vol. 4, no. 26, 2019.
  41. J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel, “Cloth Grasp Point Detection Based on Multiple-View Geometric Cues with Application to Robotic Towel Folding,” in IEEE International Conference on Robotics and Automation (ICRA), 2010.
  42. J. Matas, S. James, and A. J. Davison, “Sim-to-Real Reinforcement Learning for Deformable Object Manipulation,” Conference on Robot Learning (CoRL), 2018.
  43. T. Morita, J. Takamatsu, K. Ogawara, H. Kimura, and K. Ikeuchi, “Knot Planning from Observation,” in IEEE International Conference on Robotics and Automation (ICRA), 2003.
  44. A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine, “Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulation,” in IEEE International Conference on Robotics and Automation (ICRA), 2017.
  45. OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning Dexterous In-Hand Manipulation,” in International Journal of Robotics Research (IJRR), 2019.
  46. F. Osawa, H. Seki, and Y. Kamiya, “Unfolding of Massive Laundry and Classification Types by Dual Manipulator,” Journal of Advanced Computational Intelligence and Intelligent Informatics, 2007.
  47. D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, “Zero-Shot Visual Imitation,” in International Conference on Learning Representations (ICLR), 2018.
  48. D. A. Pomerleau, “Efficient Training of Artificial Neural Networks for Autonomous Navigation,” Neural Comput., vol. 3, 1991.
  49. J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Robotic Manipulation and Sensing of Deformable Objects in Domestic and Industrial Applications: a Survey,” in International Journal of Robotics Research (IJRR), 2018.
  50. J. Schulman, J. Ho, C. Lee, and P. Abbeel, “Learning from Demonstrations Through the Use of Non-Rigid Registration,” in International Symposium on Robotics Research (ISRR), 2013.
  51. D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, K. Yamane, S. Iba, J. Canny, and K. Goldberg, “Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
  52. D. Seita, N. Jamali, M. Laskey, R. Berenstein, A. K. Tanwani, P. Baskaran, S. Iba, J. Canny, and K. Goldberg, “Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making,” in International Symposium on Robotics Research (ISRR), 2019.
  53. L. Sun, G. Aragon-Camarasa, P. Cockshott, S. Rogers, and J. P. Siebert, “A Heuristic-Based Approach for Flattening Wrinkled Clothes,” Towards Autonomous Robotic Systems, 2014.
  54. L. Sun, G. Aragon-Camarasa, S. Rogers, and J. P. Siebert, “Accurate Garment Surface Analysis using an Active Stereo Robot Head with Application to Dual-Arm Flattening,” in IEEE International Conference on Robotics and Automation (ICRA), 2015.
  55. P. Sundaresan, J. Grannen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and K. Goldberg, “Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data,” in IEEE International Conference on Robotics and Automation (ICRA), 2020.
  56. J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, “Sim-to-Real: Learning Agile Locomotion For Quadruped Robots,” in Robotics: Science and Systems (RSS), 2018.
  57. M. Verschoor and A. Jalba, “Efficient and accurate collision response for elastically deformable models,” ACM Transactions on Graphics, vol. 38, pp. 1–20, 03 2019.
  58. A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar, “Learning Robotic Manipulation through Visual Planning and Acting,” in Robotics: Science and Systems (RSS), 2019.
  59. J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser, “Spatial Action Maps for Mobile Manipulation,” in Robotics: Science and Systems (RSS), 2020.
  60. Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel, “Learning to Manipulate Deformable Objects without Demonstrations,” in Robotics: Science and Systems (RSS), 2020.
  61. M. Yan, Y. Zhu, N. Jin, and J. Bohg, “Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects,” in IEEE Robotics and Automation Letters (RA-L), 2020.
  62. W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto, “Learning Predictive Representations for Deformable Objects Using Contrastive Estimation,” in Conference on Robot Learning (CoRL), 2020.
  63. K. Zakka, A. Zeng, J. Lee, and S. Song, “Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly,” in IEEE International Conference on Robotics and Automation (ICRA), 2020.
  64. A. Zeng, “Learning Visual Affordances for Robotic Manipulation,” Ph.D. dissertation, Princeton University, 2019.
  65. A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee, “Transporter Networks: Rearranging the Visual World for Robotic Manipulation,” in Conference on Robot Learning (CoRL), 2020.
  66. A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, “TossingBot: Learning to Throw Arbitrary Objects with Residual Physics,” in Robotics: Science and Systems (RSS), 2019.
  67. A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser, “Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
  68. A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, N. Fazeli, F. Alet, N. C. Dafle, R. Holladay, I. Morona, P. Q. Nair, D. Green, I. Taylor, W. Liu, T. Funkhouser, and A. Rodriguez, “Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching,” in IEEE International Conference on Robotics and Automation (ICRA), 2018.
  69. K. Zhang, M. Sharma, J. Liang, and O. Kroemer, “A Modular Robotic Arm Control Stack for Research: franka-interface and frankapy,” arXiv preprint arXiv:2011.02398, 2020.
Citations (148)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now