ClothPPO: A Proximal Policy Optimization Enhancing Framework for Robotic Cloth Manipulation with Observation-Aligned Action Spaces
Abstract: Vision-based robotic cloth unfolding has made great progress recently. However, prior works predominantly rely on value learning and have not fully explored policy-based techniques. Recently, the success of reinforcement learning on the LLM has shown that the policy gradient algorithm can enhance policy with huge action space. In this paper, we introduce ClothPPO, a framework that employs a policy gradient algorithm based on actor-critic architecture to enhance a pre-trained model with huge 106 action spaces aligned with observation in the task of unfolding clothes. To this end, we redefine the cloth manipulation problem as a partially observable Markov decision process. A supervised pre-training stage is employed to train a baseline model of our policy. In the second stage, the Proximal Policy Optimization (PPO) is utilized to guide the supervised model within the observation-aligned action space. By optimizing and updating the strategy, our proposed method increases the garment's surface area for cloth unfolding under the soft-body manipulation task. Experimental results show that our proposed framework can further improve the unfolding performance of other state-of-the-art methods.
- Speedfolding: Learning efficient bimanual folding of garments. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2022.
- Cloth3d: clothed 3d humans. In European Conference on Computer Vision, pages 344–359. Springer, 2020.
- Openai gym. arXiv preprint arXiv:1606.01540, 2016.
- Cloth funnels: Canonicalized-alignment for multi-purpose garment manipulation. In International Conference of Robotics and Automation (ICRA), 2022.
- Daxbench: Benchmarking deformable object manipulation with differentiable physics. In ICLR, 2023.
- Life regression based patch slimming for vision transformers. Neural Networks, page 106340, 2024.
- Folding clothes autonomously: A complete pipeline. IEEE Transactions on Robotics, 32(6):1461–1478, 2016.
- An introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning, 11(3-4):219–354, 2018.
- Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515–11522. IEEE, 2021.
- Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.
- Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robotic Learning (CoRL), 2021.
- Disassembling convolutional segmentation network. International Journal of Computer Vision, 131(7):1741–1760, 2023.
- Self-supervised cloth reconstruction via action-conditioned cloth tracking. arXiv preprint arXiv:2302.09502, 2023.
- Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018.
- Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. In Conference on Robot Learning, 2020.
- Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256–266. PMLR, 2022.
- Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734–743. PMLR, 2018.
- Foldsformer: Learning sequential multi-step cloth manipulation with space-time attention. IEEE Robotics and Automation Letters, 8(2):760–767, 2022.
- Qdp: Learning to sequentially optimise quasi-static and dynamic manipulation primitives for robotic cloth manipulation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2023.
- Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2146–2153. IEEE, 2017.
- Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
- Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. 2022.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4568–4575. IEEE, 2021.
- Edward Jay Sondik. The optimal control of partially observable Markov processes. Stanford University, 1971.
- A geometric approach to robotic unfolding of garments. Robotics and Autonomous Systems, 75:233–243, 2016.
- Gerhard Tutz. Sequential models in categorical regression. Computational Statistics & Data Analysis, 11(3):275–295, 1991.
- Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1–6, 2022.
- Spatial action maps for mobile manipulation. arXiv preprint arXiv:2004.09141, 2020.
- Learning to manipulate deformable objects without demonstrations, 2020.
- Robotube: Learning household manipulation from human videos with simulated twin environments. In Conference on Robot Learning, pages 1–10. PMLR, 2023.
- Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.