DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy (2505.11032v2)

Abstract: Garment manipulation is a critical challenge due to the diversity in garment categories, geometries, and deformations. Despite this, humans can effortlessly handle garments, thanks to the dexterity of our hands. However, existing research in the field has struggled to replicate this level of dexterity, primarily hindered by the lack of realistic simulations of dexterous garment manipulation. Therefore, we propose DexGarmentLab, the first environment specifically designed for dexterous (especially bimanual) garment manipulation, which features large-scale high-quality 3D assets for 15 task scenarios, and refines simulation techniques tailored for garment modeling to reduce the sim-to-real gap. Previous data collection typically relies on teleoperation or training expert reinforcement learning (RL) policies, which are labor-intensive and inefficient. In this paper, we leverage garment structural correspondence to automatically generate a dataset with diverse trajectories using only a single expert demonstration, significantly reducing manual intervention. However, even extensive demonstrations cannot cover the infinite states of garments, which necessitates the exploration of new algorithms. To improve generalization across diverse garment shapes and deformations, we propose a Hierarchical gArment-manipuLation pOlicy (HALO). It first identifies transferable affordance points to accurately locate the manipulation area, then generates generalizable trajectories to complete the task. Through extensive experiments and detailed analysis of our method and baseline, we demonstrate that HALO consistently outperforms existing methods, successfully generalizing to previously unseen instances even with significant variations in shape and deformation where others fail. Our project page is available at: https://wayrise.github.io/DexGarmentLab/.

An Overview of DexGarmentLab: Dexterous Garment Manipulation Environment with Generalizable Policy

DexGarmentLab introduces the first simulation environment specifically designed for dexterous, especially bimanual, garment manipulation, significantly advancing the field of deformable object manipulation. This paper addresses critical challenges in garment manipulation such as diversity in garment categories, complex geometries, and deformations. The proposed environment features high-quality 3D assets, a novel data collection pipeline, and an innovative manipulation policy, referred to as HALO (Hierarchical gArment manipuLation pOlicy), which enhances generalization capabilities across varied garment shapes and deformations.

Key Components

1. DexGarmentLab Environment: Built upon NVIDIA IsaacSim, DexGarmentLab offers a realistic simulation platform with scenarios encompassing interactions with both garments and other objects, such as hangers and human avatars. It uses advanced simulation techniques like Position-Based Dynamics (PBD) and Finite Element Method (FEM) to model garment behaviors accurately. The environment comprises more than 2,500 garments in 8 categories and 15 task scenarios.
2. Automated Data Collection Pipeline: The pipeline leverages garment structural correspondence, allowing the generation of diverse trajectories through a single expert demonstration, thereby reducing manual intervention. GAM (Garment Affordance Model) is key to identifying suitable manipulation points across garments, facilitating efficient data collection without extensive human effort.
3. Generalizable Policy (HALO): HALO combines GAM with a novel Structure-Aware Diffusion Policy (SADP) to enhance manipulation across unseen garment instances. Using transfer learning techniques and a hierarchical framework, HALO excels in accurately locating manipulation points and generating adaptive trajectories based on garment structures, thereby outperforming existing baseline methods such as Diffusion Policy (DP) and 3D Diffusion Policy (DP3).

Experimental Analysis

Extensive experiments demonstrate the superior data efficiency and generalization ability of HALO. In 14 garment manipulation tasks, HALO consistently achieves higher success rates than DP and DP3. The ablation paper further highlights the importance of GAM and SADP components in handling diverse shapes and interaction scenarios. In real-world experiments simulating tasks such as Fold Tops and Hang Dress, HALO maintains high success rates, underscoring its practical applicability.

Implications and Future Directions

DexGarmentLab bridges the sim-to-real gap in garment manipulation, offering a platform for developing robust robotic manipulation capabilities applicable to both domestic and industrial settings. This research opens avenues for exploring more intricate tasks involving multiple garments and mobile dexterous robots. Moreover, the simulation methodologies and generalizable policy framework can be extended to other deformable object manipulation challenges, broadening the scope for practical deployments in household robotics and beyond.

The paper sets a foundation for future research in dexterous robotic manipulation, challenging researchers to enhance algorithms for better real-world applicability and explore broader manipulative tasks that go beyond single-arm or single-garment interactions.