SimWeaver-Sim: Deformable Object Simulator

• SimWeaver-Sim is a measurement-backed, high-fidelity deformable object simulator designed for zero-shot sim-to-real transfer in complex tasks like garment handling.
• It integrates a variational implicit dynamics engine, robust collision management, and ISP-aware photometric augmentation to ensure strong shape fidelity and contact stability.
• The simulator achieves efficient performance with <5 ms per simulation step and >89.7% real-world task success, eliminating the need for per-task calibration.

SimWeaver-Sim is a measurement-backed deformable object simulator central to the SimWeaver pipeline for zero-shot sim-to-real transfer in deformable manipulation, particularly through RGB-based visuomotor policy learning. Developed as an extensible, high-fidelity physics engine, SimWeaver-Sim is designed to bridge the realism gap between simulation and physical systems without the need for real-world data fine-tuning or per-task calibration. It achieves strong shape and contact fidelity with physically meaningful parameters bound to empirical fabric measurements, supporting the synthesis of reliable demonstrations for challenging tasks such as garment handling, plastic bag manipulation, and silk grasping (Hu et al., 13 Jun 2026).

1. Architecture and Core Components

SimWeaver-Sim builds upon the RGBench cloth solver, extending it with modules for robust collision handling, penetration prevention, and deterministic replay. It combines four principal subsystems:

• A deformable-dynamics engine based on a variational implicit time integrator (Vertex Block Descent) with per-vertex updates.
• A contact manager implementing active and forbidden collision regions specifically around gripper jaws, supporting penalty-based resolution of cloth–rigid, cloth–cloth, and cloth–table contacts.
• A configuration manager exposing parameters such as mass per area, bending/stretch stiffness, and friction, directly mapped to physical textile properties from the RGBench material library.
• An Omniverse Kit-based renderer that generates synchronized RGB data with realistic camera latency and photometric augmentation faithful to measured RealSense D435i parameters (Hu et al., 13 Jun 2026).

2. Measurement-Backed Continuous-Energy Modeling

SimWeaver-Sim eschews proxy parameter tuning for a continuous-energy (physics-based) approach, enforcing empirical consistency between simulation and measured textiles. The deformable object is modeled as a triangular mesh with vertex positions $\mathbf{x} \in \mathbb{R}^{3n}$. Temporal integration follows an implicit Euler scheme:

$$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$

where $\rho$ is the per-vertex mass density, $h$ is the timestep, and $E(\mathbf{x})$ aggregates the elastic energies:

• Stretch energy per edge $e=(i,j)$:

$$E_s^e = \frac{k_s}{2} (\| \mathbf{x}_i - \mathbf{x}_j \| - l^0_e)^2$$

• Bending energy per dihedral pair $(e, e')$:

$$E_b^{e,e'} = \frac{k_b}{2} (\theta(\mathbf{x};e,e') - \theta^0)^2$$

Contact and friction forces are penalty-based and physically grounded. Cloth–rigid proximity introduces a normal force proportional to penetration depth, while Coulomb friction is modeled at the velocity level based on blended static/dynamic coefficients directly measured from fabric–gripper interactions (ASTM D1388, D3107).

3. Contact, Collision, and Stability

A distinguishing feature of SimWeaver-Sim is its maintenance of active vs. forbidden contact regions around manipulation end-effectors. Cloth self-collisions and rigid contacts are managed via a combination of penalty-based forces and topological projectors. This scheme achieves zero penetration and zero catastrophic failure (explosive instability) rates across demanding thin-shell scenarios. In comparative evaluations, SimWeaver-Sim attains a physics-only simulation step cost below 5 ms, outperforming typical Position-Based Dynamics (PBD) and naïve Finite Element Analysis (FEA) engines (4.44 ms/step vs. 7.8 ms for Isaac/PBD and 10.4 ms for Newton/VBD) (Hu et al., 13 Jun 2026).

Multilayer contact stability is ensured even under extreme folding, preventing self-collision jitter and interpenetration. Gripper jaws can reliably close on stacked layers without penetration or oscillatory instabilities.

4. Rendering, Photometric Augmentation, and ISP Accuracy

The rendering subsystem is tightly integrated with an Omniverse-Kit pipeline, simulating pinhole camera intrinsics and full exposure/gain/white-balance stages. Each virtual camera (mirroring RealSense D435i characteristics) incorporates a calibrated rendering lag (40–80 ms) to replicate real-world sensor latency.

Photometric augmentation is strictly ISP-aware: renderings are stochastically transformed per camera using empirically bounded transformations—

• brightness $\alpha \sim \mathcal{U}(0.75,1.15)$
• contrast $$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$0
• saturation $$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$1
• hue shift $$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$2
• sharpness jitter, small affine warps, Gaussian noise $$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$3

These augmentations are crucial: disabling ISP-aware GPU-side augmentation in ablation causes task success rates to drop to 0% on all simulated deformable manipulation tasks.

5. Validation, Metrics, and Determinism

SimWeaver-Sim is validated using both simulation-specific and sim-to-real metrics:

Metric	Definition / Remarks	Reported Value / Context
Penetration Rate	Fraction of simulation frames with cloth–rigid interpenetration	Zero for thin garments
Explosion Rate	Fraction of simulation rollouts with unbounded energy growth	Zero under all test cases
Task/Grasp Success	Fraction of demo pick-and-lift motions completed without contact errors	>89.7% (e.g., T-shirt flattening pass rate)
Per-Step Time	Physics step run-time (modern GPU)	4.44 ms (SimWeaver-Sim), 7.8 ms (Isaac/PBD)
Real-World Success	Zero-shot trials passing closed-loop task predicates ($$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$4 per task)	Mean 91.3% (Wilson-95% CI: [84.7, 95.2]%)
Pixel Reprojection Error	$$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$5 for vision	Used for vision task validation
Physical Parameter Error	Drift in sim vs. measured parameters $$\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} \left[ E(\mathbf{x}) + \frac{\rho}{2h^2}\|\mathbf{x}-\mathbf{x}^t - h\,\mathbf{v}^t \|^2 \right]$$6	For monitoring sim–real parameter drift

Perfect simulation replay determinism (100/100 replays) is maintained, supporting rapid and reliable policy data generation at scale (2,824 usable trajectories/day on an 8×RTX 4090 server).

6. Integration in the SimWeaver Pipeline

SimWeaver-Sim is a central element within the SimWeaver system:

• SimWeaver-Asset provides physically measured asset meshes and material parameters.
• SimWeaver-Sim simulates deformable dynamics and photorealistic RGB outputs with domain and ISP-aware randomization.
• SimWeaver-Syn invokes SimWeaver-Sim for closed-loop, topology-driven demonstration synthesis, bypassing learned diffusion models for deterministic and interpretable grasp generation.
• Rendered and labeled demonstration trajectories are consumed by a learned policy trainer (typically for vision-language-action (VLA) models).
• Deployment occurs zero-shot on real hardware with RealSense cameras, exploiting the high measurement-fidelity and deterministic nature of SimWeaver-Sim.

The pipeline automates all data flow from asset instantiation to sim-augmented demonstration generation, policy training, and real-world evaluation, removing the need for manual teleoperation, per-task calibration, or synthetic proxy tuning (Hu et al., 13 Jun 2026).

7. Key Findings and Trade-offs

SimWeaver-Sim achieves superior sim-to-real transfer by enforcing strong shape fidelity, physically-meaningful parameterization, and multilayer contact determinism—all at moderate computational cost. The use of measurement-backed, continuous-energy modeling eliminates per-asset manual calibration, supporting generalization across asset types from thin shells to thick textiles. ISP-aware augmentation is empirically essential; disabling it leads to complete failure in sim-to-real policy transfer. The discrete, topology-based trajectory synthesis method yields demonstration pass rates exceeding 90% without requiring complex learned generative models.

A plausible implication is that this approach provides a path toward scalable, data-efficient, real-world deployment of deformable object manipulation policies, particularly in domains where real-world data collection is costly or unsafe (Hu et al., 13 Jun 2026).