Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Differentiable Particles for General-Purpose Deformable Object Manipulation (2405.01044v1)

Published 2 May 2024 in cs.RO

Abstract: Deformable object manipulation is a long-standing challenge in robotics. While existing approaches often focus narrowly on a specific type of object, we seek a general-purpose algorithm, capable of manipulating many different types of objects: beans, rope, cloth, liquid, . . . . One key difficulty is a suitable representation, rich enough to capture object shape, dynamics for manipulation and yet simple enough to be acquired effectively from sensor data. Specifically, we propose Differentiable Particles (DiPac), a new algorithm for deformable object manipulation. DiPac represents a deformable object as a set of particles and uses a differentiable particle dynamics simulator to reason about robot manipulation. To find the best manipulation action, DiPac combines learning, planning, and trajectory optimization through differentiable trajectory tree optimization. Differentiable dynamics provides significant benefits and enable DiPac to (i) estimate the dynamics parameters efficiently, thereby narrowing the sim-to-real gap, and (ii) choose the best action by backpropagating the gradient along sampled trajectories. Both simulation and real-robot experiments show promising results. DiPac handles a variety of object types. By combining planning and learning, DiPac outperforms both pure model-based planning methods and pure data-driven learning methods. In addition, DiPac is robust and adapts to changes in dynamics, thereby enabling the transfer of an expert policy from one object to another with different physical properties, e.g., from a rigid rod to a deformable rope.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973–8979. IEEE, 2019.
  2. Ab Initio Particle-based Object Manipulation. In Proceedings of Robotics: Science and Systems, July 2021. doi: 10.15607/RSS.2021.XVII.071.
  3. Daxbench: Benchmarking deformable object manipulation with differentiable physics. In The Eleventh International Conference on Learning Representations, 2022.
  4. Imitation learning as state matching via differentiable physics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7846–7855, 2023.
  5. Iterative residual policy for goal-conditioned dynamic manipulation of deformable objects. In Proceedings of Robotics: Science and Systems (RSS), 2022.
  6. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
  7. Bringing clothing into desired configurations with limited perception. In 2011 IEEE international conference on robotics and automation, pages 3893–3900. IEEE, 2011.
  8. End-to-end differentiable physics for learning and control. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  9. Brax–a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281, 2021.
  10. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515–11522. IEEE, 2021.
  11. Disect: A differentiable simulation engine for autonomous robotic cutting. arXiv preprint arXiv:2105.12244, 2021.
  12. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4):1–14, 2018a.
  13. Three-dimensional deformable object manipulation using fast online gaussian process regression. IEEE Robotics and Automation Letters, 3(2):979–986, 2018b.
  14. 3-d deformable object manipulation using deep neural networks. IEEE Robotics and Automation Letters, 4(4):4255–4261, 2019.
  15. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.03311, 2021.
  16. Differentiable algorithm networks for composable robot learning. In Proceedings of Robotics: Science and Systems (RSS), 2019.
  17. Segment anything. arXiv:2304.02643, 2023.
  18. Learning force-based manipulation of deformable objects from multiple demonstrations. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 177–184. IEEE, 2015.
  19. Aquarium: A fully differentiable fluid-structure interaction solver for robotics applications. arXiv preprint arXiv:2301.07028, 2023.
  20. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018.
  21. Learning visible connectivity dynamics for cloth smoothing. arXiv preprint arXiv:2105.10389, 2021.
  22. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256–266. PMLR, 2022.
  23. Tangled: Learning to untangle ropes with rgb-d perception. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 837–844. IEEE, 2013.
  24. Focused adaptation of dynamics models for deformable object manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5931–5937. IEEE, 2023.
  25. Pomdp approach to robotized clothes separation. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1324–1329. IEEE, 2012.
  26. Pods: Policy optimization via differentiable simulation. In International Conference on Machine Learning, pages 7805–7817. PMLR, 2021.
  27. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2146–2153. IEEE, 2017.
  28. Scalable differentiable physics for learning and control. arXiv preprint arXiv:2007.02168, 2020.
  29. U2-net: Going deeper with nested u-structure for salient object detection. In Pattern Recognition, volume 106, page 107404, 2020.
  30. Arthur George Richards. Robust constrained model predictive control. PhD thesis, Massachusetts Institute of Technology, 2005.
  31. Learning deformable object manipulation from expert demonstrations. IEEE Robotics and Automation Letters, 7(4):8775–8782, 2022.
  32. C. Schenck and D. Fox. Spnets: Differentiable fluid dynamics for deep neural networks. In Proceedings of the Second Conference on Robot Learning (CoRL), Zurich, Switzerland, 2018a.
  33. Spnets: Differentiable fluid dynamics for deep neural networks. In Conference on Robot Learning, pages 317–335. PMLR, 2018b.
  34. Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4568–4575. IEEE, 2021.
  35. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016.
  36. A particle method for history-dependent materials. Computer methods in applied mechanics and engineering, 118(1-2):179–196, 1994.
  37. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9411–9418. IEEE, 2020.
  38. Differentiable physics and stable modes for tool-use and manipulation planning. In Robotics: Science and Systems, 2018.
  39. Learning robotic manipulation through visual planning and acting. arXiv preprint arXiv:1905.04411, 2019.
  40. Dynamic-Resolution Model Learning for Object Pile Manipulation. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. doi: 10.15607/RSS.2023.XIX.047.
  41. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. arXiv preprint arXiv:2303.02346, 2023.
  42. Accelerated policy learning with parallel differentiable simulation. arXiv arXiv:2204.07137, 2022.
  43. Learning predictive representations for deformable objects using contrastive estimation. In Conference on Robot Learning, pages 564–574. PMLR, 2021.
  44. Model-based strategy for grasping 3d deformable objects using a multi-fingered robotic hand. Robotics and Autonomous Systems, 95:196–206, 2017.
  45. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010.14406, 2020.
  46. Learning fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets