Benchmarking the Sim-to-Real Gap in Cloth Manipulation (2310.09543v2)

Abstract: Realistic physics engines play a crucial role for learning to manipulate deformable objects such as garments in simulation. By doing so, researchers can circumvent challenges such as sensing the deformation of the object in the realworld. In spite of the extensive use of simulations for this task, few works have evaluated the reality gap between deformable object simulators and real-world data. We present a benchmark dataset to evaluate the sim-to-real gap in cloth manipulation. The dataset is collected by performing a dynamic as well as a quasi-static cloth manipulation task involving contact with a rigid table. We use the dataset to evaluate the reality gap, computational time, and simulation stability of four popular deformable object simulators: MuJoCo, Bullet, Flex, and SOFA. Additionally, we discuss the benefits and drawbacks of each simulator. The benchmark dataset is open-source. Supplementary material, videos, and code, can be found at https://sites.google.com/view/cloth-sim2real-benchmark.

• The paper introduces an open benchmark dataset to quantify the sim-to-real gap using Chamfer and Hausdorff distances.
• It evaluates dynamic and quasi-static cloth tasks, revealing that MuJoCo and SOFA outperform in dynamic simulations.
• Findings emphasize the need for optimized simulator tuning for achieving real-time, high-fidelity cloth manipulation.

Benchmarking the Sim-to-Real Gap in Cloth Manipulation

The paper "Benchmarking the Sim-to-Real Gap in Cloth Manipulation" presents a systematic evaluation of the fidelity of physics simulators in replicating real-world dynamics of cloth manipulation. With simulated environments serving a pivotal role in developing robotic manipulation skills, understanding the sim-to-real gap is critical for effective real-world application. The authors address the scarce quantitative evaluation in current literature regarding this gap, particularly focusing on dynamic and quasi-static manipulation tasks of cloths.

Contributions and Methods

This paper introduces an open-source benchmark dataset designed to evaluate the sim-to-real gap in cloth manipulation using popular simulators: MuJoCo, Bullet, Flex, and SOFA. The dataset comprises point clouds and RGB-D images from real-world data capturing cloth manipulation tasks, which involve dynamic motion and quasi-static contact with a surface.

The authors assess the performance of the simulators in two key tasks:

1. Dynamic Task: Evaluates cloth dynamics without surface contact, emphasizing the effect of acceleration on the fabric.
2. Quasi-static Task: Involves the dragging of the cloth over a rigid surface, assessing the ability to simulate contact dynamics.

To quantify the sim-to-real gap, the metric of choice is the Chamfer Distance (CD) and the Hausdorff Distance (HD) between the real-world cloth data and simulated meshes. Furthermore, the authors explore the stability and computational efficiency of these simulators to determine their potential for real-time applications.

Key Findings

The results indicate significant variability in the fidelity of cloth dynamics simulation across the different engines. MuJoCo and SOFA exhibit better performance in dynamic tasks, while all simulators show relatively low distances in quasi-static scenarios. However, no simulator perfectly captures the nuances of dynamic cloth behavior, especially under high acceleration conditions.

The paper further highlights computational considerations. While MuJoCo and Bullet potentially offer real-time simulation capabilities, stability issues arise unless appropriately tuned. Flex and SOFA present robust stability across frequencies but may lack the immediacy or nuanced parameterization required for high-fidelity dynamic tasks.

Implications and Future Directions

The research underscores the necessity of fine-tuning and potentially hybrid approaches, such as domain randomization or sim-to-real-to-sim techniques, to mitigate the sim-to-real discrepancy further. The provided dataset and benchmark pave the way for future research aimed at refining simulator parameters or developing new simulation methodologies to more accurately mirror real-world dynamics.

As robotic applications increasingly involve interaction with deformable materials, enhancing simulator fidelity is paramount. This work lays a foundational benchmark for ongoing developments in simulation technology, encouraging innovations that could drive more effective sim-to-real transfers.

In conclusion, the paper offers a valuable framework and dataset for researchers striving to bridge the sim-to-real gap in cloth manipulation, facilitating advancements in the field towards more reliable and applicable real-world solutions. Future work could extend this benchmark to include a broader range of materials and more complex manipulation scenarios.