SimWeaver: Zero-Shot Sim-to-Real Framework

• SimWeaver is a zero-shot sim-to-real framework enabling efficient vision-language-action policy learning for deformable object manipulation in robotics.
• It integrates four subsystems—SimWeaver-Sim, -Asset, -Syn, and -Real—to synthesize high-diversity simulated demonstrations with deterministic replay and lower data collection costs.
• The framework achieves over 80% success across five deformable manipulation tasks, outperforming real-data baselines while eliminating fine-tuning, teleoperation, and depth sensing.

SimWeaver is a modular, zero-shot sim-to-real framework designed for end-to-end vision-language-action (VLA) policy learning for deformable object manipulation tasks in robotics. It achieves robust zero-shot transfer of RGB-based policies to real-world scenarios without the need for fine-tuning, teleoperation, per-task calibration, or depth input. SimWeaver integrates four tightly coupled subsystems—SimWeaver-Sim, SimWeaver-Asset, SimWeaver-Syn, and SimWeaver-Real—enabling the synthesis of a small number (200) of physically faithful, high-diversity simulated demonstrations per task, behavior cloning-based policy training, and direct deployment on robotic hardware. Across five deformable manipulation tasks, SimWeaver demonstrates above 80% task success rates with an average of 91.3% on real hardware, outperforming or matching real-data baselines, while imposing two orders of magnitude lower data-collection costs (Hu et al., 13 Jun 2026).

1. System Components and Architecture

SimWeaver’s architecture comprises four integrated modules, each addressing a critical aspect of sim-to-real transfer for deformable manipulation:

• SimWeaver-Sim: A measurement-backed deformable-object simulator based on the RGBench solver. It enforces physically meaningful parameters—mass per area, stretch and bending stiffness, static and dynamic friction—drawn from measured textile properties. Collision handling employs an “active collision region” approach: collision resolution is locally disabled around closing gripper jaws to prevent artifact-inducing jitter and penetration, then re-enabled for untangling, ensuring deterministic replay.
• SimWeaver-Asset: This subsystem provides a unified asset library of approximately 2,000 triangle-mesh garments and plastic bags, each associated with real-measured textile properties from the RGBench fabric database. It includes a single-image asset generator pipeline that infers the physical category of a novel fabric from a photograph using a vision-LLM and samples its physical parameters from empirical distributions for instantiation in simulation.
• SimWeaver-Syn: A deterministic, topology-aware demonstration synthesizer. Given a canonical mesh landmark graph $G=(V,E)$ denoting feasible bimanual grasp pairs, it algorithmically selects optimal candidate grasps, executes parameterized, constrained motion plans (lift, swing, return) using time-optimal or min-jerk planning and closed-form IK, and verifies both grasp and terminal state predicates, retrying on failure. No human-in-the-loop is involved.
• SimWeaver-Real: This sim-to-real protocol introduces domain and photometric randomization informed by empirical sensor ISP statistics from RealSense D435i cameras. Its procedures include physics-driven initial state randomization (e.g., pin-and-release) and per-image photometric augmentation (with specialized parameter ranges for brightness, contrast, saturation, and hue) matched to real-world camera drift/bias characteristics. Photometric augmentation is a necessary and sufficient condition for transfer: disabling it reduces zero-shot real success to 0% across tasks.

2. Physical Simulation Methodology

Deformable objects in SimWeaver are modeled as triangle meshes with per-vertex masses ($m_i$), subject to internal elastic energy $E(q)$ (summed stretch and bending), external forces (gravity, contact), and constraint forces from collision and friction. The forward simulation advances state using implicit Euler time integration:

$$M (q_{t+\Delta t} – q_t – \Delta t v_t) + \Delta t^2 \nabla E(q_{t+\Delta t}) + \Delta t C(q_{t+\Delta t},v_{t+\Delta t}) = 0$$

where $C(\cdot)$ enforces non-penetration and frictional contact using an active-collision-region window. This mechanism creates a shrinking region around the gripper jaw in which self-collision is suspended as the jaw closes; outside this region, collision handling is immediately reenabled. This selective resolution avoids grasp-stall and mesh interpenetration with no visible artifacts or measurable latency. Deterministic replay is guaranteed even under contact-rich manipulation, ensuring repeatable demonstration synthesis and downstream policy learning (Hu et al., 13 Jun 2026).

3. Asset Generation and Domain Randomization

Asset generation in SimWeaver-Asset links empirical measurement to mesh instantiation. When presented with a single RGB image $x$ of a fabric, a vision-LLM classifies the fabric into a category $c = \mathrm{VLM}(x)$. Physical parameters $p \sim P(p|c)$—including bending and stretch as measured by ASTM D1388 and D3107, per-area mass, and friction—are then sampled from the real-world distributions in the RGBench database. The mesh geometry is either copied from the existing asset pool or synthesized by EmbodiedGen conditioned on the classified type, then instantiated in the simulator.

Domain randomization (executed in SimWeaver-Real) includes:

• Cloth state: Physics-driven randomization, e.g., pin-and-release, random folding, or free-fall settling.
• Robot/randomization axes: Joint home pose noise (0.5×), cloth XY offset ($\pm$10 cm), yaw ($\pm$15°), fixed table variations (height $m_i$01.5 cm in silk experiments), and randomized textures and lighting.
• Material parameters: These are not perturbed and remain tied to empirical measurements.
• Photometric augmentation: Image transformations drawn from empirically fitted ranges—brightness $m_i$1, contrast $m_i$2, saturation $m_i$3, hue $m_i$4—plus minor affine perturbations, matched to color bias $m_i$5 and channel drift $m_i$6 (as measured on RealSense D435i units). Only this augmentation prepares simulation images for the VLA policy; it is a prerequisite for successful transference.

4. Trajectory Synthesis and Policy Learning

SimWeaver-Syn synthesizes demonstration trajectories using canonical mesh landmarks as semantic grasp candidates. The selection policy filters pairwise candidates via geometric constraints (reachability, no arm crossing, inter-grip distance, surface exposure, occlusion, and layer separation). The optimal pair $m_i$7 maximizes a score $m_i$8 over feasible pairs. Each synthesized policy executes multi-stage, parameterized bimanual manipulation, using either a time-optimal path parameterization (TOPP-RA) or min-jerk profile for smoothness. Terminal predicates are checked after each action; grasp retries are allowed up to $m_i$9. This protocol produces trajectories with 100% replay determinism and high pass rates (e.g., 89.7% for T-shirt flatten), in contrast to less reliable prior simulators (24% pass, 13% deterministic replay for SIM1).

Policy learning treats each demonstration as a trajectory $E(q)$0, with $E(q)$1 as multi-camera RGB images, $E(q)$2 as joint state, and $E(q)$3 as expert joint-velocity action. The VLA policy $E(q)$4 is trained by behavior cloning from a pretrained initialization, minimizing:

$E(q)$5

Optimization uses 30,000 AdamW steps (batch size 32) with cosine learning-rate decay (peak $E(q)$6) on four RTX 4090 GPUs in mixed-precision, without reinforcement learning or auxiliary losses.

5. Experimental Results and Cost Efficiency

Five real-world deformable object manipulation tasks were evaluated:

1. Snack packaging: Inserting an object into a plastic bag, then lifting its handles.
2. Garment folding: Folding a T-shirt to a ¼-fold.
3. Garment unfolding: Flattening a pinned draped T-shirt.
4. Silk unfolding: Flattening arbitrarily draped silk cloth.
5. Silk grasping: Bimanual lifting of thin silk without prior flattening.

Each task used 200 synthesized demonstrations. Real-world evaluation (23 trials/task) used two 6-DOF arms with parallel jaws and three RealSense D435i cameras (two wrist-mounted, one overhead). Success rates were as follows:

Task	Success Rate (%)
Snack packaging	86.96
Garment folding	91.30
Garment unfolding	82.61
Silk unfolding	95.65
Silk grasping	100.00
Average	91.30

On out-of-distribution shifts in silk grasping (texture, lighting, rotation), the SimWeaver-trained policy achieved 100% success (CI [85.7,100.0]), while a real-data-trained baseline (100 demonstrations) was significantly lower: 13.0% (texture), 69.6% (lighting), 8.7% (rotation).

SimWeaver yields substantial cost reductions per usable trajectory:

Method	Time/traj (min)	Traj/day	Cost/traj (\$)	Relative Cost
Real robot	4.6	104	2.71	1.0×
SIM1	16.2	710	0.10	0.037×
SimWeaver	4.1	2824	0.03	0.011×

6. Limitations and Future Directions

SimWeaver demonstrates that rigorous simulator fidelity, measurement-backed asset instantiation, deterministic topology-aware trajectory synthesis, and empirically matched photometric augmentation are jointly sufficient for training pixel-based VLA policies for deformable tasks with minimal demonstration count, removing the need for teleoperation, per-asset calibration, or depth sensing. This approach restores the competitiveness of pure-RGB solutions on challenging materials such as dark, reflective, or low-texture fabrics where conventional depth sensors underperform.

Identified limitations include:

• Full head-to-head real-data versus sim-data comparisons across all tasks are deferred to future work;
• Access to the comprehensive industrial fabric-measurement library (RGBench) requires a licensed release;
• The existing pipeline is limited to single-cloth and bag topologies; extension to more complex networks of deformables (e.g., ropes, fluids) would necessitate advanced simulation and representation.

Anticipated directions involve an open benchmark built on SimWeaver, investigation of multi-material and multi-object scenes, dynamic manipulation subtasks (e.g., shaking and pouring), and introduction of learned asset generation refinement pipelines.

7. Context and Significance

SimWeaver establishes a technical pathway for data-efficient, reproducible, and direct zero-shot sim-to-real transfer in pixel-based deformable manipulation. Notably, its methodology circumvents limitations of depth sensing and labor-intensive real-world calibration by exploiting physical parameterization, deterministic demonstration synthesis, and photometric alignment. The framework provides a scalable blueprint for future research in sim-to-real transfer across diverse material domains, as well as opportunities for standardized evaluation via anticipated open-source release and benchmarking (Hu et al., 13 Jun 2026).