SIM1 Architecture & Pipeline

• SIM1 Architecture is a physics-aligned real-to-sim-to-real data engine designed for scalable policy learning in deformable object manipulation.
• It integrates high-fidelity scene digitization, soft-body physics calibration, and diffusion-based trajectory synthesis to ensure sub-millimeter accuracy and robust sim-to-real transfer.
• Empirical results demonstrate up to 98% real-world success and a 27× cost reduction compared to traditional methods, facilitating effective synthetic data generation.

SIM1 is a physics-aligned real-to-sim-to-real (R2S2R) data engine designed to scale zero-shot policy learning for robotic manipulation of deformable objects. The architecture tightly integrates high-fidelity scene digitization, advanced soft-body physics calibration, generative diffusion models for behavior expansion, and rigorous synthetic-to-real filtering, addressing the data demands and sim-to-real transfer challenges intrinsic to deformable object domains such as garment folding (Zhou et al., 9 Apr 2026).

1. Scene Digitization and Metric Twin Generation

SIM1 initiates from a small number of real-world expert demonstrations, converting real deformable-object environments into digital twins with metric fidelity. The process consists of multi-view RGB and LiDAR scans of garments mounted on mannequins, yielding dense point clouds. These are reconstructed into watertight meshes via Poisson surface reconstruction [10.1145/2487228.2487237], followed by geometric post-processing (hole filling, fairing, remeshing) to achieve sub-millimeter fidelity.

Camera intrinsics and extrinsics $(K, R, t)$ are determined using checkerboard calibration, establishing correspondence: $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$ where $\mathbf{X} \in \mathbb{R}^3$ is a garment mesh point and $\mathbf{u} \in \mathbb{R}^2$ is its projected image coordinate. The robot’s URDF is imported directly, and an alignment transform ensures the simulation's kinematics are isomorphic to the real-world workspace. Environmental assets such as tables and fixtures are measured in situ and introduced as matching geometry, producing a fully populated, metric-accurate synthetic replica.

2. Deformable Physics Solver Calibration

Central to SIM1 is the calibration of a soft-body physics engine such that simulated cloth behavior matches real-world deformation under manipulation. The Augmented Vertex Block Descent (AVBD) solver, using a St. Venant–Kirchhoff elasticity model,

$$\Psi(F) = \mu \|G\|_{F}^{2} + \tfrac{1}{2}\hat{\lambda}(\mathrm{tr} G)^{2}, \quad G = \tfrac{1}{2}(F^T F - I),$$

is supplemented with dihedral-angle-based bending and edge-based strain constraints. Strain per edge $(i, j)$ is enforced as

$C_{ij} = \|x_i - x_j\| - (1 + \xi)l_{0} \le 0,$

with corresponding penalty energy active when constraints are violated.

Behavioral calibration proceeds by dual teleoperation—identical joint trajectories executed in real and simulated scenes—iteratively optimizing solver parameters $\Theta = \{\rho, E, \nu, \mu, \eta, \zeta\}$ by minimizing differences in visual feature statistics $\phi(\cdot)$ (e.g., drape, wrinkle): $$\Theta^* = \arg\min_{\Theta} \sum_{t=1}^T \|\phi_{\mathrm{sim}}(\tau_t^{\mathrm{sim}}; \Theta) - \phi_{\mathrm{real}}(\tau_t^{\mathrm{real}})\|^2 + \beta\|\Theta-\Theta_0\|^2,$$ thereby ensuring that synthetic trajectories are physically grounded.

3. Diffusion-Based Trajectory Expansion

To overcome scarcity of real and teleoperated examples, SIM1 employs generative modeling to synthesize large volumes of diverse, human-like manipulation trajectories. Expert demonstrations are temporally segmented into alternating “interaction” (stable grasp) and “movement” segments. Interaction segments are pooled as behavioral primitives.

For movement synthesis, a diffusion model learns to interpolate between key postures $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$0 given context $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$1. The forward noising process is

$\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$2

with a denoiser trained to recover the clean sequence from noise via

$\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$3

This enables the system to generate thousands of plausible, task-relevant manipulation trajectories from a limited set of seeds.

Quality control utilizes two filters: (1) a state-based “vibe coding” filter leveraging particle-level statistics to rapidly cull samples exhibiting excessive stretch or self-collision, and (2) a video discriminator (ResNet-18 + Transformer) evaluating rendered head-view videos, accepting only success-flagged, high-discriminator-score episodes for downstream use.

4. Synthetic Data Generation Pipeline

The end-to-end SIM1 pipeline transforms a small set of human trials into a large-scale synthetic dataset suitable for policy learning. The sequence is as follows:

1. Collect $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$4 expert trials in the real world.
2. Execute scene digitization and dynamics calibration as described in Sections 1 and 2.
3. Record a further $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$5 teleoperated episodes in simulation with calibrated physics.
4. Segment interactions, train the diffusion model, and synthesize novel trajectories.
5. Loop: sample segment skeletons, generate intermediate movements, simulate executions, apply filtering, and render $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$6 randomized RGB videos.
6. Export data tuples $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$7 in standard LeRobot format, aggregating approximately $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$8 filtered synthetic episodes into $\mathbf{u} \sim K\,[R\;\;t]\;\mathbf{X},$9.

This pipeline produces high-diversity, high-fidelity training data with near-expert behavior coverage, enabling effective downstream policy learning.

5. System Integration and Empirical Performance

The integrated SIM1 architecture forms a closed R2S2R loop: real demonstration $\mathbf{X} \in \mathbb{R}^3$0 digitization $\mathbf{X} \in \mathbb{R}^3$1 calibration $\mathbf{X} \in \mathbb{R}^3$2 simulation/synthesis (diffusion + filtering) $\mathbf{X} \in \mathbb{R}^3$3 rendering $\mathbf{X} \in \mathbb{R}^3$4 policy training $\mathbf{X} \in \mathbb{R}^3$5 real-world evaluation.

Key empirical findings on cloth manipulation (e.g., T-shirt folding) include:

• Zero-shot sim-to-real: Policies trained exclusively on SIM1 synthetic data ($\mathbf{X} \in \mathbb{R}^3$6 episodes) achieved 98% in-domain real-world success and 90% zero-shot transfer success, matching or slightly exceeding real-only data (97% at $\mathbf{X} \in \mathbb{R}^3$7 real episodes).
• Generalization: Under varying textures, lighting, and spatial configurations, SIM1-based policies averaged 50% higher generalization success rates compared to real-data baselines.
• Data efficiency: The saturated equivalence ratio is $\mathbf{X} \in \mathbb{R}^3$8 (one real demonstration $\mathbf{X} \in \mathbb{R}^3$9 15 synthetic episodes for performance parity).
• Cost efficiency:

$\mathbf{u} \in \mathbb{R}^2$0

• Summary Table (in-domain $\mathbf{u} \in \mathbb{R}^2$1):

Training Data	Success (%)	Equivalence
Real only (200 ex)	97	$\mathbf{u} \in \mathbb{R}^2$2 real
SIM1 synthetic (3k ex)	98	$\mathbf{u} \in \mathbb{R}^2$3 synthetic

This suggests SIM1 allows greater scaling at radically reduced per-trial cost without loss of effectiveness, due to direct physics-alignment, advanced generative expansion, and robust filtering.

6. Architectural Implications and Research Significance

SIM1 represents an explicit shift from rigid-body simulators and generic sim-to-real transfer toward physically-grounded, data-scaling pipelines specific to deformable manipulation regimes. By enforcing metric isomorphism and behavioral fidelity throughout the pipeline, SIM1 closes the sim-to-real gap for soft bodies. Its combination of expert-calibrated elastic modeling with generative, quality-filtered trajectory expansion yields a synthetic supervision engine that is empirically validated for zero-shot deployment and data-efficiency (Zhou et al., 9 Apr 2026).

A plausible implication is that the SIM1 pipeline can be extended to other physically complex, underconstrained domains where real data is expensive, provided sufficient scene digitization and physical model calibration are feasible. Its design further distinguishes between physical grounding (scene and dynamics alignment) and behavioral grounding (success-based, human-like policy generation), clarifying where future advances in generalizable sim-to-real robotics may occur.