SoftMimicGen for Deformable Manipulation

• SoftMimicGen is a unified system that generates diverse datasets for scalable robot learning in deformable object manipulation.
• It leverages non-rigid registration and high-fidelity simulation to transform limited human demonstrations into synthetic trajectories.
• The system significantly improves manipulation policies over rigid-body methods, achieving high success in tasks such as rope whipping and towel folding.

SoftMimicGen is a unified data generation system designed to address the challenges of scalable robot learning in the domain of deformable object manipulation. By leveraging a small set of human teleoperated demonstrations and high-fidelity simulation, SoftMimicGen enables automated synthesis of large-scale, diverse datasets necessary for training robust manipulation policies. The system integrates non-rigid registration, a suite of task environments involving various robot embodiments and manipulation behaviors, and advanced imitation learning methods. The approach demonstrates marked improvements over prior rigid-body methods, particularly for complex tasks involving cloth, rope, tissue, and other soft objects (Moghani et al., 26 Mar 2026).

1. Pipeline Architecture

SoftMimicGen executes in three principal stages:

• Environment setup: Each data-generation episode instantiates a new configuration of a deformable-object task within Isaac Lab, sampling object states, backgrounds, and illumination.
• Data generation: Beginning with 1–10 segmented human teleoperated demonstrations per task, the system applies non-rigid registration between novel scene configurations and source task segments to produce correspondence warp fields. These fields transform demonstrated end-effector trajectories to fit the new scene geometry, with successful executions logged as new demonstrations.
• Policy training and evaluation: The synthetic dataset—consisting of approximately 1,000 demonstrations per task—is used to train visuomotor policies via behavioral cloning (BC–RNN–GMM) or diffusion-policy objectives. Policies are evaluated in both simulation and on real robotic hardware.

The operational pipeline includes the following components:

• Human data collection → subtask segmentation → randomized scene sampler → non-rigid registration → trajectory warping → simulated execution → dataset aggregation → policy learning → simulated and physical evaluation.

2. Simulation Environments and Object Modeling

SoftMimicGen spans ten manipulative environments, each built on Isaac Lab’s GPU-accelerated soft-body solver. Key tasks include:

• Humanoid–Teddy: A GR1 humanoid places a plush toy into a basket.
• Franka–Rope/Jenga: Franka Panda manipulates a rope in both static and dynamic (whipping) contexts.
• Surgical–Threading/Tissue: dVRK robot passes surgical thread through a ring or manipulates soft tissue.
• YAM–Towel/BagLoading: Bimanual towel folding or dual-arm bag opening and object placing.

Objects are discretized into $N_o$ nodes $O=\{n_i\in\mathbb{R}^3\}$, with internal forces modeled via mass–spring systems ($$m_i\ddot{x}_i = \sum_{j} k_s(\|x_j-x_i\|-L_{ij}^0)\frac{x_j-x_i}{\|x_j-x_i\|} - c_d\dot{x}_i + f_{ext,i}$$) or corotational finite-element methods (FEM) for tasks requiring large deformations. Collision and friction models incorporate penalty forces and Coulomb friction.

3. Robot Embodiments and Manipulation Behaviors

Four distinct robot platforms are included:

• Single-Arm Manipulator: Franka Panda (7+1 DoF), acting via end-effector SE(3) deltas and gripper commands.
• Bimanual System: YAM arms (each 7+gripper DoF) for coordinated bi-arm tasks.
• Humanoid: GR1 robot (6-DoF torso, 7-DoF arms, 5-DoF dexterous hand).
• Surgical Robot: dVRK with 6-DoF plus forceps gripper.

Supported task categories include high-precision threading (sub-millimeter accuracy), dynamic rope whipping, large-deformation folding, and pick-and-place involving deformable support structures.

4. Data Generation Methodology

4.1 Sampling and Domain Randomization

Scene parameters (e.g., object placement, pose, cloth friction $\mu_s\sim U[0.3, 0.7]$, mass scaling $m_{scale}\sim U[0.9, 1.1]$) are randomized per episode. Camera intrinsics and lighting are also varied to induce policy robustness.

4.2 Demonstration Synthesis via Non-Rigid Registration

Non-rigid registration aligns scene node configurations $O'$ with source segments $\{\tau_i\}$ by solving

$$f_i^\ast = \arg\min_f \left[\sum_{k=1}^{N_{src}}\|f(a_k) - b_k\|^2 + \lambda_{smooth}\int \|\nabla f(x)\|^2 dx\right].$$

The optimal warp $f_i^\ast$ is applied to both the position and orientation of the original demonstration trajectory. Successful trials—by task-specific predicates—are included in the dataset.

4.3 Data Scale and Modalities

Each task achieves approximately 1,000 demonstration trajectories, with each trajectory comprising 200–500 timesteps. Logged data includes RGB images (640×480), depth (480×360), robot state, actions, and ground-truth node positions.

5. Policy Learning

Two families of policies are trained:

• Behavioral Cloning (BC–RNN–GMM): An LSTM with 256 hidden units models the policy as a 5-component GMM over actions, minimizing

$$L_{BC} = -\frac{1}{N}\sum_{i=1}^N\sum_{t=0}^{H_i}\log\pi_\theta(a_t^i|o_t^i).$$

Encoders include ResNet18 (visual), PointNet (nodes), and an LSTM, outputting GMM parameters for end-effector motions and gripper.

• Diffusion Policy: Actions are generated by a conditional diffusion model, with U-Net backbone featuring cross-attention to observations.

Hyperparameters include AdamW optimizer, learning rates ($O=\{n_i\in\mathbb{R}^3\}$0 for BC, $O=\{n_i\in\mathbb{R}^3\}$1 for diffusion), batch size 64–200, and up to 200 training epochs.

6. Experimental Analysis

6.1 Simulation Performance

Policies trained with SoftMimicGen-generated datasets substantially outperform those trained only on the sparse human source set. For instance, on Franka–Rope, success improves from $O=\{n_i\in\mathbb{R}^3\}$2 (source only) to $O=\{n_i\in\mathbb{R}^3\}$3 (generated) with BC–RNN–GMM, and $O=\{n_i\in\mathbb{R}^3\}$4 for the diffusion variant.

Task	Source BC–RNN–GMM	Generated BC–RNN–GMM	Generated Diffusion
Humanoid–Teddy	0.0 ± 0.0	32.0 ± 3.3	42.0 ± 2.0
Franka–Rope	2.0 ± 2.0	99.3 ± 0.9	100.0 ± 0.0
Franka–Jenga	4.0 ± 3.3	89.3 ± 15.1	80.0 ± 3.3

6.2 Dataset Size Ablations

Increasing generated data improves policy reliability, with success rates plateauing beyond ≈500 demonstrations per task.

# Demonstrations	Franka–Rope Success (%)	Franka–Jenga Success (%)
50	82.7 ± 5.0	53.3 ± 8.2
250	100.0 ± 0.0	77.3 ± 6.8
500	98.7 ± 1.9	93.3 ± 5.0
750	93.3 ± 6.8	84.0 ± 15.0

6.3 Comparison with Rigid MimicGen

Rigid-body MimicGen achieves only 4/50 successful rope trajectory generations (SE(3) alignment), whereas SoftMimicGen succeeds on 49/50 via non-rigid registration, with significant gains in generalization to unseen object shapes.

6.4 Real-World Transfer

Using Point-Bridge for sim-to-real transfer, SoftMimicGen-trained policies achieve nontrivial zero-shot success on hardware and benefit significantly from combined sim+real co-training.

Task	Real-Only 30 demos	Zero-Shot 1,000 sims	Sim–Real Co-Train
Franka–Towel	76.6	70.0	76.6
Franka–Rope	46.7	33.3	76.6
YAM–BagLoading	33.3	63.3	93.3

A plausible implication is that synthetic data generation pipelines—when informed by high-fidelity non-rigid simulation and robust registration—can significantly reduce the need for real-world data in learning robust manipulation skills.

7. Limitations and Future Directions

Current limitations include:

• Manual subtask sequencing: Task pipelines require fixed sequences and heuristic segmentation; complex tasks with non-linear structure are not yet addressed automatically.
• Simulation–real gap: Registration presumes access to ground-truth nodal states; real-world sensing (noisy depth, point-clouds) poses challenges for applying these methods robustly.
• Computation: Non-rigid registration incurs a runtime cost (~50 ms/segment), creating a bottleneck for data throughput.
• Object diversity: Presently limited to four deformable classes; extending to fluids, granular media, and elasto-plastic solids is identified as an open direction.
• Generalization: While initial-state generalization is strong, transfer across tasks and robot embodiments warrants further study.

SoftMimicGen demonstrates that scalable, automated generation of high-diversity deformable manipulation data is feasible and effective for both simulation and real-robot learning. The system and all supporting resources have been made available to support continued research and application in the domain of robotic manipulation of deformable objects (Moghani et al., 26 Mar 2026).