SoftMimicGen: A Data Generation System for Scalable Robot Learning in Deformable Object Manipulation

Abstract: Large-scale robot datasets have facilitated the learning of a wide range of robot manipulation skills, but these datasets remain difficult to collect and scale further, owing to the intractable amount of human time, effort, and cost required. Simulation and synthetic data generation have proven to be an effective alternative to fuel this need for data, especially with the advent of recent work showing that such synthetic datasets can dramatically reduce real-world data requirements and facilitate generalization to novel scenarios unseen in real-world demonstrations. However, this paradigm has been limited to rigid-body tasks, which are easy to simulate. Deformable object manipulation encompasses a large portion of real-world manipulation and remains a crucial gap to address towards increasing adoption of the synthetic simulation data paradigm. In this paper, we introduce SoftMimicGen, an automated data generation pipeline for deformable object manipulation tasks. We introduce a suite of high-fidelity simulation environments that encompasses a wide range of deformable objects (stuffed animal, rope, tissue, towel) and manipulation behaviors (high-precision threading, dynamic whipping, folding, pick-and-place), across four robot embodiments: a single-arm manipulator, bimanual arms, a humanoid, and a surgical robot. We apply SoftMimicGen to generate datasets across the task suite, train high-performing policies from the data, and systematically analyze the data generation system. Project website: \href{https://softmimicgen.github.io}{softmimicgen.github.io}.

• The paper introduces a novel non-rigid registration mechanism to adapt sparse human demonstrations for scalable synthesis of rich deformable object manipulation datasets.
• Empirical results show success rates of 70%-100% across diverse tasks and robot embodiments, significantly outperforming previous rigid-object approaches.
• The system’s robust pipeline enables efficient sim-to-real transfer, reducing data collection costs while advancing imitation-based robotic learning.

SoftMimicGen: Scalable Data Generation for Deformable Object Manipulation

Motivation and Background

The rapid progress of robot foundation models and generalist manipulation policies is fundamentally linked to the availability of large, high-quality demonstration datasets. Traditional data collection via teleoperation is expensive and labor-intensive, especially for complex manipulations such as those involving deformable objects. Recent advances in synthetic simulation-based data generation—including frameworks like MimicGen—have facilitated scalable dataset creation for rigid object manipulation; however, simulation and trajectory adaptation for deformable objects remain unresolved challenges due to their high dimensionality and non-linear material properties.

SoftMimicGen (2603.25725) addresses these limitations by introducing a data generation pipeline that enables manipulation dataset synthesis for a wide range of deformable objects and robot embodiments with minimal human supervision. The key innovation is leveraging non-rigid registration to adapt human demonstration trajectories to arbitrary deformable object configurations, thereby generalizing the object-centric invariance principle introduced in MimicGen to the deformable scenario.

System Pipeline and Deformable Object Representation

SoftMimicGen begins from a small set of teleoperated demonstrations, which are parsed into object-centric subtasks. Unlike rigid objects, where invariance can be exploited via a static SE(3) transformation, deformable objects require a more expressive representation. Each object is modeled as a collection of 3D nodes—a point cloud, effectively capturing its physical configuration. This flexible representation eliminates the requirement for canonical coordinate frames and enables the manipulation of objects with complex deformations.

Figure 1: SoftMimicGen pipeline: human-teleoperated demonstration segmentation, non-rigid registration, trajectory adaptation via warp fields, and execution in novel environments.

Non-rigid registration is employed to map the configuration of the object in the demonstration to its state in the target scene, generating a smooth deformation field. This field is used for source demonstration segment selection (via registration cost minimization) and for trajectory warping, ensuring accurate and robust transfer of manipulation instructions.

Non-Rigid Registration and Trajectory Warping

Given two configurations of a deformable object, SoftMimicGen solves an optimization to obtain a deformation field $\mathbf{f}: \mathbb{R}^3 \rightarrow \mathbb{R}^3$ minimizing the point-wise distance between mapped nodes and regularizing for smoothness. This operation produces correspondence and warp fields allowing pose adaptation of end-effectors and maintaining local spatial relationships through the manipulation. Each pose $p_t$ is transformed as $p_t \rightarrow \mathbf{f}(p_t)$, and rotations are adapted via the Jacobian of $\mathbf{f}$.

This approach enables the replay of demonstrations in configurations far from those seen in the original data, overcoming the brittleness and domain limitations of rigid-body transfer mechanisms.

Task Suite and Multi-Embodiment Applicability

SoftMimicGen is evaluated on a diverse task suite comprising ten challenging deformable manipulation scenarios and four distinct robot embodiments: GR1 humanoid, Franka Panda arm, dVRK surgical robot, and bimanual YAM arms. Tasks include high-precision threading, towel unfolding/folding, rope shaping, dynamic block whipping, tissue manipulation, and pick-and-place of plush toys.

Figure 2: Task suite spans humanoid, single-arm, bimanual, and surgical robots operating on ropes, towels, tissues, and rigid blocks.

The pipeline's generality is demonstrated by successful application to both deformable and rigid object tasks, confirming that SoftMimicGen strictly generalizes prior approaches such as MimicGen, DexMimicGen, and SkillMimicGen.

Empirical Results and Policy Performance

SoftMimicGen-generated datasets achieve high success rates (70%–100%) across the task suite for both BC-RNN-GMM and Diffusion Policy models. Policies trained exclusively on generated data consistently outperform those trained solely on scarce human demonstrations—performance improvements range from 25% to 97%, confirming the value of synthetic simulation data even in the absence of extensive real-world collection.

Figure 3: SoftMimicGen enables high-fidelity simulation and diverse manipulation behaviors for task-specific dataset generation.

Real-world deployment experiments further validate the pipeline. Policies trained with 1,000 simulation-generated demonstrations achieve substantial zero-shot sim-to-real transfer; sim–real co-training with limited real-world supervision (30 demos) further boosts performance, notably achieving 93.3% success rate on YAM Bag Loading task. Notably, the pipeline enables end-to-end learning using point-based representations in conjunction with domain-agnostic frameworks (e.g., Point Bridge), bypassing explicit registration during policy execution.

(Figure 4)

Figure 4: Real-world rollouts of deformable tasks using policies trained on SoftMimicGen-generated datasets, demonstrating sim-to-real transfer.

Comparative Analysis and Ablations

In direct comparison, SoftMimicGen vastly outperforms MimicGen on deformable tasks: for rope manipulation, MimicGen achieves only 8% successful demonstration synthesis, whereas SoftMimicGen achieves 98%. Larger generated datasets correlate robustly with improved policy success rates, emphasizing the critical role of scale in imitation-based learning for dexterous manipulation.

Replay- or registration-based controllers are susceptible to sensor noise and require explicit correspondence at inference; SoftMimicGen circumvents these limitations by training policies to directly operate on raw image or structured point representations, providing robustness and inference-time flexibility.

Implications and Future Directions

SoftMimicGen's synthesis mechanism, grounded in non-rigid registration, enables scalable, automated demonstration generation for tasks previously considered impractical for synthetic augmentation. This has direct implications for robot foundation model training, reducing human labor, facilitating generalization, and expanding the scope of robotic skill acquisition in simulation. From a theoretical standpoint, the pipeline elevates object-centric invariance from SE(3) transformations to non-linear, configuration-space-adaptive mappings, setting a precedent for further generalizations across manipulation paradigms.

Future developments could focus on relaxing the fixed subtask sequencing assumption, supporting more flexible and conditional task structures, and integrating sim-to-real adaptation under more severe distribution shifts. Advanced registration algorithms or learned correspondence models may further boost robustness for manipulation in noisy real-world environments.

Conclusion

SoftMimicGen introduces a scalable, general-purpose synthetic data generation system for deformable object manipulation, leveraging non-rigid registration and trajectory warping to adapt sparse human demonstrations into large, representative datasets. Its applicability to multiple embodiments, robustness across diverse tasks, and support for efficient policy learning—both in simulation and real-world deployment—address longstanding barriers in robotic skill acquisition and data-driven research. The formal extension of object-centric invariance through non-rigid transformations constitutes an important advancement, with practical and theoretical ramifications for the study and development of manipulation-focused foundation models in robotics.