Learning Generalizable Tool-use Skills through Trajectory Generation (2310.00156v5)
Abstract: Autonomous systems that efficiently utilize tools can assist humans in completing many common tasks such as cooking and cleaning. However, current systems fall short of matching human-level of intelligence in terms of adapting to novel tools. Prior works based on affordance often make strong assumptions about the environments and cannot scale to more complex, contact-rich tasks. In this work, we tackle this challenge and explore how agents can learn to use previously unseen tools to manipulate deformable objects. We propose to learn a generative model of the tool-use trajectories as a sequence of tool point clouds, which generalizes to different tool shapes. Given any novel tool, we first generate a tool-use trajectory and then optimize the sequence of tool poses to align with the generated trajectory. We train a single model on four different challenging deformable object manipulation tasks, using demonstration data from only one tool per task. The model generalizes to various novel tools, significantly outperforming baselines. We further test our trained policy in the real world with unseen tools, where it achieves the performance comparable to human. Additional materials can be found on our project website: https://sites.google.com/view/toolgen.
- Improvisation through physical understanding: Using novel objects as tools with visual foresight. Robotics: Science and Systems (RSS), 2019.
- Learning task-oriented grasping for tool manipulation from simulated self-supervision. The International Journal of Robotics Research (IJRR), 2020.
- Keto: Learning keypoint representations for tool manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2020.
- Gift: Generalizable interaction-aware functional tool affordances without labels. In Robotics: Science and Systems (RSS), 2021.
- Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. International Conference on Learning Representations (ICLR), 2022.
- Learning closed-loop dough manipulation using a differentiable reset module. IEEE Robotics and Automation Letters, 7(4):9857–9864, 2022.
- Planning with spatial-temporal abstraction from point clouds for deformable object manipulation. In Conference on Robot Learning (CoRL), 2022.
- Understanding tools: Task-oriented object modeling, learning and recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
- Understanding physical effects for effective tool-use. IEEE Robotics and Automation Letters (R-AL), 2022.
- Virdo++: Real-world, visuo-tactile dynamics and perception of deformable objects. 2022.
- Disect: A differentiable simulation engine for autonomous robotic cutting. arXiv preprint arXiv:2105.12244, 2021.
- Roboninja: Learning an adaptive cutting policy for multi-material objects. Robotics: Science and Systems (RSS), 2023.
- Plasticinelab: A soft-body manipulation benchmark with differentiable physics. In International Conference on Learning Representations, 2021.
- Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
- A smooth representation of belief over so (3) for deep rotation learning with uncertainty. arXiv preprint arXiv:2006.01031, 2020.
- On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2019.
- Projective manifold gradient layer for deep rotation regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6646–6655, 2022.
- Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541–4550, 2019.
- Toolflownet: Robotic manipulation with tools via predicting tool flow from point clouds. In Conference on Robot Learning (CoRL), 2022.
- Sinkhorn divergences for unbalanced optimal transport. arXiv preprint arXiv:1910.12958, 2019.
- O. Sorkine-Hornung and M. Rabinovich. Least-squares rigid motion using svd. Computing, 1(1):1–5, 2017.
- An analysis of svd for deep rotation estimation. Advances in Neural Information Processing Systems, 33:22554–22565, 2020.