SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation

Abstract: RGB sim-to-real for deformable manipulation has remained largely unsolved without real-world fine-tuning. We present SimWeaver, which trains zero-shot RGB VLA policies on 200 simulated demonstrations per task, reaching above 80% per-task and 91% average real-world success across 5 diverse deformable tasks including plastic-bag manipulation, without teleoperation or per-task calibration. SimWeaver combines a reliable measurement-backed simulator (SimWeaver-Sim) with an extensible asset framework supporting single-image generation(SimWeaver-Asset), a deterministic topology-aware trajectory synthesizer (SimWeaver-Syn), and a sim-to-real protocol with ISP-aware photometric augmentation (SimWeaver-Real). On silk grasping, the sim-trained policy reaches 100% under visual distribution shifts where real-data baselines drop to 9-70%, at two orders of magnitude lower per-trajectory cost. We will release SimWeaver and a representative asset subset. Project page: https://simweaver.github.io/

• The paper introduces SimWeaver, a pipeline that achieves zero-shot sim-to-real transfer for deformable manipulation without real-world demonstrations.
• It employs deterministic, topology-aware trajectory synthesis and physics-driven, sensor-specific augmentations to enhance reliability.
• Experimental results show an average success rate of 91.3% across five tasks, outperforming traditional real-data-trained policies under domain shifts.

SimWeaver: Zero-Shot RGB Sim-to-Real Transfer for Deformable Manipulation

Motivation and Problem Setting

Robotic manipulation of deformable objects from RGB observations, without access to real-world demonstrations or task-specific manual calibration, remains an unresolved challenge due to simulator limitations, high-dimensional configuration spaces, and the particular difficulty of predicting physical fabric behavior and visual appearance. Prior works primarily achieve reliable sim-to-real transfer in rigid or near-rigid regimes, and even state-of-the-art deformable frameworks depend heavily on point cloud input, teleoperation for trajectory generation, or tedious, asset-specific calibration, which are not scalable for general deformable categories such as garments and plastic bags. "SimWeaver: Zero-Shot RGB Sim-to-Real for Deformable Manipulation" (2606.15338) presents an end-to-end, pixel-based pipeline producing Vision-Language-Action (VLA) policies trained exclusively on simulation data, achieving compelling zero-shot transfer in five challenging deformable manipulation tasks.

System Overview

The SimWeaver system architecture consists of four major components: a high-fidelity, measurement-anchored simulator (SimWeaver-Sim), an extensible asset framework (SimWeaver-Asset), a deterministic, topology-aware demonstration synthesizer (SimWeaver-Syn), and an ISP-informed sim-to-real protocol (SimWeaver-Real).

Figure 1: Overview of SimWeaver showing the asset pipeline, demonstration synthesis, and policy transfer pathway from simulated to real-world execution.

This modular structure enables automatic generation of physically plausible deformable assets, robust synthesis of diverse demonstrations per task, and pixel-based learning with carefully calibrated augmentations accounting for both physical and sensor stochasticities.

Deformable Simulation Reliability and Physical Anchoring

A central contribution of SimWeaver is the deployment of a contact-reliable deformable simulator designed to eliminate determinism-violating artifacts (penetration, grasp jitter, replays failing after reset) that plague widely used Position-Based Dynamics or VBD solvers in the context of thin, multi-layer fabrics.

Figure 2: Simulator failure modes of commercial PBD-based and VBD-based cloth solvers, contrasted with the penetration-free, stable behavior and contact-zone partitioning of SimWeaver-Sim.

The simulator's stability arises from an active-collision-region scheme, in which self-collision resolution is spatially modulated around robot contact areas, suppressing oscillation and preventing physical anomalies during gripper interaction. All simulated fabric physical properties (bend, stretch, mass, friction) are parameterized using direct measurements from industrial textile protocols (RGBench), obviating heuristic per-asset tuning and ensuring consistency between synthetic and real-world deployments.

Deterministic, Topology-Aware Trajectory Synthesis

SimWeaver-Syn generates high-quality, task-diverse demonstration trajectories without teleoperation or learned generative models. For each asset, a semantic topological graph is constructed over mesh landmarks (e.g., garment sleeves, bag handles), encoding feasible bimanual grasp pairs and manipulation primitives. Feasibility predicates (occlusion checks, layer separation, workspace reachability, safety, anti-crossing) are all analytically defined in terms of instantaneous simulation state, yielding a discrete selection process for grasping and manipulation that is robust under large deformations and occlusions.

Closed-loop synthesis evaluates each stage, retries upon grasp failure (dominant in deformable manipulation), and guarantees determinism—synthesized trajectories are always executable with replay success rates of 100% for hundreds of resets. This sharply contrasts with stochastic, diffusion-based learned synthesizers that require teleoperated seeds and extensive post-hoc rejection filtering.

Sim-to-Real Protocol: Physics- and Sensor-Aware Domain Randomization

Standard rendering realism and domain randomization are augmented by two key deformable-specific modifications in SimWeaver-Real:

• Physics-driven state randomization: Initial states of fabrics are sampled through physical simulation primitives (drop, fold, settle), ensuring that the demonstration distribution covers the high-dimensional space of wrinkles, folds, and self-contact, instead of naïve pose randomization.
• Sensor-specific photometric augmentation: Photometric transformations are directly matched to measured ISP noise modes of Intel D435i cameras (per-unit color shifts, gain-loop drift), substantially exceeding the coverage of standard color jitter. Disabling these augmentations collapses sim-to-real transfer success to 0% across all tasks, highlighting their necessity in closing the visual domain gap.

  Figure 3: Empirical characterization of RGB sensor color bias and variation, grounding the choices of simulation-to-real photometric augmentation ranges.

Extensible Asset Generation and Physical Grounding

The asset pipeline includes both large-scale mesh imports (e.g., ~2,000 CLOTH3D assets) and single-image, physically-parametrized asset generation (e.g., EmbodiedGen). All assets are annotated with measured physical parameters from RGBench's fabric library; this enables seamless addition of new object types with photorealistic and physically realistic behaviors without retuning.

Main Results: Robustness, Generalization, and Efficiency

SimWeaver achieves consistent, high-performing sim-to-real transfer on five diverse deformable-manipulation tasks: snack packaging (plastic bag insertion), garment folding, garment unfolding, silk flattening, and silk grasping. Policies receive only 200 synthetic demonstrations per task and are deployed zero-shot on a dual-arm real robot.

Success rates (n=23 trials per task):

Task	Success (%)
Snack packaging	86.96
Garment folding	91.30
Garment unfolding	82.61
Silk unfolding	95.65
Silk grasping	100.00
Average	91.3

The minimum per-task success is $>80\%$ and the pooled mean is $91.3\%$, without any teleoperation, per-task, or per-asset calibration.

Figure 4: Qualitative sim-to-real rollout comparison: simulated trajectories (left) and zero-shot real-robot execution (right) for five deformable manipulation tasks.

Figure 5: Silk grasping—policy trained on sim data (with augmentation) matches or outperforms real-data-trained policy under both in-distribution and OOD shifts (texture, lighting, rotation). In contrast, real-trained policies degrade sharply under OOD.

Strong/Contradictory Claims

• The sim-trained policy is more sample efficient and robust to distribution shift than real-data-trained policies: SimWeaver-trained policies retain $100\%$ success under severe distributional shift (e.g., lighting, rotation, and texture), where real-data-trained policies’ success rates drop precipitously (as low as 9–70%).
• SimWeaver’s cost per usable demonstration is two orders of magnitude lower than real-robot collection, with high throughput and negligible unusable data rates. The pipeline achieves approximately $2824$ usable trajectories/day at $\$0.03$ each.

Implications and Future Directions

SimWeaver represents a significant shift toward scalable, robust, and sample-efficient sim-to-real learning for deformable object manipulation using only RGB input. The deterministic, topology-aware generator and the physically annotated asset framework could serve as foundational infrastructure for broader deformable manipulation benchmarks and rapid prototyping in both academic and industrial contexts.

Points of potential extension include expanding real-world task taxonomies, incorporating multi-modal sensors for hybrid policies, and scaling the framework with even larger automatically generated asset libraries. Opportunities also exist in benchmarking policy architectures with respect to their reliance on physically realistic synthesis versus real-world demonstration data, and in integrating reinforcement or active learning for settings requiring online adaptation.

Conclusion

SimWeaver demonstrates that zero-shot sim-to-real transfer for complex, pixel-based deformable manipulation is feasible and practical with only a modest number of well-constructed synthetic demonstrations, provided that physical and sensor realism are systematically addressed at all stages of the pipeline. The strong sim-to-real correspondence, combined with sample and cost efficiency, suggests that systematic, measurement-backed simulators and deterministic task synthesis are sufficient for high-stakes LfD deployment scenarios in deformable manipulation, paving the way for broader adoption and benchmarking within the embodied AI community.