SimWeaver-Real: Photometric Sim-to-Real Transfer

• SimWeaver-Real is a sim-to-real adaptation and photometric augmentation module that bridges simulated and real-world deformable object manipulation.
• It integrates physics-driven cloth state randomization, robot domain randomization, and calibrated ISP-aware augmentation to model RealSense D435i distortions.
• The framework achieves high task success rates (up to 91.3% average) without real-world fine-tuning, ensuring scalable and sample-efficient policy training.

SimWeaver-Real is the sim-to-real adaptation and photometric augmentation component of the SimWeaver stack, a framework enabling zero-shot policy transfer for deformable object manipulation from simulation to the real world using only RGB inputs. SimWeaver-Real systematically closes the sim-to-real gap in vision-language-action policy learning for challenging tasks involving cloth, garments, and bags by combining physics-aware state randomization, systematic domain randomization, and precisely calibrated ISP-aware photometric data augmentation. Unlike traditional domain randomization protocols, SimWeaver-Real is specifically engineered to absorb camera hardware-specific distortions, such as RealSense D435i color biases and gain-loop drift, which are otherwise ignored by default data augmentation routines. Comprehensive architectural integration across SimWeaver-Asset and SimWeaver-Sim results in a scalable protocol that achieves 91.3% average real-world task success without real-world fine-tuning or teleoperation, with consistent zero-shot transfer even under severe visual domain shifts (Hu et al., 13 Jun 2026).

1. Architectural Role and Workflow

SimWeaver-Real operates as the terminal stage within the SimWeaver sim-to-real data generation pipeline. Assets (meshes plus measured material parameters) are produced by SimWeaver-Asset through single-image generation workflows, using the RGBench fabric library for physical characterization. These asset descriptions are then consumed by SimWeaver-Sim, a physics-based deformable simulation layer that incorporates task-specific initializations (e.g., pin-and-release, random-fold-and-settle) and deterministic, topology-aware trajectory synthesis (SimWeaver-Syn).

SimWeaver-Real processes this simulated data episodically, introducing three orthogonal axes of augmentation:

1. Physics-driven cloth state randomization: Rather than static pose enumeration, cloth states are generated through primitives such as free-fall, two-point hang, and diagonal fold-and-settle, leveraging contact and bending dynamics inside the simulator to produce realistic, diverse configurations with representative folds, wrinkles, and self-contacts.
2. Robot and scene domain randomization: Each episode introduces randomized robot home-poses, table heights, background textures, illumination parameters, camera extrinsics, and physically measured cloth parameters (density, bending stiffness, friction), sampled either uniformly or via Sobol quasi-random sequences.
3. ISP-aware photometric augmentation: To match the statistical envelope and drift patterns of deployed RealSense D435i cameras, the pipeline applies brightness, contrast, saturation, hue, sharpness, affine, and optional noise transformations, with ranges empirically calibrated based on per-unit measurements.

The output per task consists of 200 high-fidelity, fully augmented RGB+state trajectories at 25 Hz, supporting sample-efficient, end-to-end VLA policy training (Hu et al., 13 Jun 2026).

2. ISP-Aware Photometric Augmentation

A central innovation in SimWeaver-Real is the ISP-aware photometric augmentation pipeline, which explicitly models systematic sensor-level distortions encountered during real deployment:

• Per-unit color bias: Empirical measurements establish that identical exposure, gain, and white-balance settings across different D435i units can yield color-channel offsets up to $\Delta(R-G)\approx39$ in 8-bit space.
• Gain-loop drift: Individual cameras exhibit per-frame mean channel drift up to 29 over dynamic scenes.

To absorb these phenomena, SimWeaver-Real fits augmentation ranges to the observed camera envelopes, departing from generic augmentation schemes. Given an 8-bit RGB image $I=(R,G,B)$, a random sequence of three transforms is applied (each weighted and parameterized according to Tab A.2):

• Brightness: $I' = \alpha I,\ \alpha \sim U[0.75,1.15]$ (weight 3.0)
• Contrast: $I' = (I - \mu)c + \mu,\ c \sim U[0.80,1.20]$ (1.0)
• Saturation (HSV): $S' = s S,\ s \sim U[0.50,1.50]$ (1.0)
• Hue (HSV): $H' = H + h,\ h \sim U[-0.15,0.15]$ (3.0)
• Sharpness: $$I' = I + \beta(I-\text{Blur}(I)),\ \beta \sim U[0.5,1.5]$$ (1.0)
• RandomAffine: $\pm5^\circ$ rotation, $\pm5\%$ translation (1.0)

Optional Gaussian noise can be injected to replicate sensor read noise but the principal transfer gains emerge from calibrated channel-jitter. Disabling photometric augmentation causes sim-to-real success to collapse to 0% across all tasks, establishing the necessity of these steps for zero-shot transfer (Tab A.3) (Hu et al., 13 Jun 2026).

3. Quantitative Sim-to-Real Transfer Performance

SimWeaver-Real enables training policies with only 200 simulated demonstrations per task, yielding the following real-world success rates over five deformable manipulation tasks (Tab 3):

Task	Real-World Success (%)
Snack packaging	86.96
Garment folding	91.30
Garment unfolding	82.61
Silk unfolding	95.65
Silk grasping	100.00
Average	91.30

Ablation studies reveal that removing ISP-aware augmentation universally destroys transfer. Under induced visual distribution shifts (texture, lighting, rotation), the sim-trained policy (ISP-DR enabled) sustains 100% silk grasping success, compared to 8.7–69.6% for real-data baselines ($n=23$, Wilson 95% CIs exclude overlap, Fisher test $I=(R,G,B)$0). Results further show that clean, synthesized trajectories confer superior sample efficiency—outperforming real-data baselines at matched demo budgets (Hu et al., 13 Jun 2026).

4. Integration with SimWeaver-Asset and SimWeaver-Sim

SimWeaver-Real implements a tightly coupled interplay with upstream asset and simulation modules:

• Asset Binding: SimWeaver-Asset provides triangle meshes and measured physical parameters directly from the RGBench fabric catalog. No hand-tuning of stretching or bending parameters is performed—faithful transfer hinges on accurate physical measurement.
• Simulation Steps: Sim initializations (cloth pose sampling, robot joint-noise, environmental perturbations) precede deterministic, topology-aware bimanual trajectory synthesis (Sec 4), conducted at 100 Hz and rendered to match real camera latency (40–80 ms).
• Visual Randomization: All domain randomization axes are applied prior to rendering, with the photometric pipeline operating online per frame.

A critical implementation tip is the use of the active-collision-region technique in SimWeaver-Sim to prevent gripper–cloth penetration without manual contact-tuning, further aligning simulated contacts with real-world outcomes (Hu et al., 13 Jun 2026).

5. Practical Reproduction and Hardware Considerations

The full SimWeaver-Real stack operates on 8× RTX 4090 GPUs for data generation and 4× RTX 4090 (48 GB) for policy fine-tuning (mixed-precision bfloat16, gradient checkpointing). Omniverse Kit serves as the renderer, with a kit_frame_update frequency adjusted to match real-world camera latency. Sobol quasi-random sampling governs domain randomization on silk tasks; uniform sampling is used elsewhere. Each usable trajectory requires 244.8 s wall-clock and costs \$I=(R,G,B)$1\$I=(R,G,B)$2/traj).

Recommended best practices include pre-calibration of camera white-balance and gain, strict adherence to measured RGBench fabric parameters, and aggressive photometric augmentation (±15% brightness, ±0.15 hue, up-weighted in transform sampling), as small systematic sensor errors can subvert policy transfer. Rendering latency is also explicitly matched to hardware characteristics (Hu et al., 13 Jun 2026).

6. Relation to Active Real-to-Simulation Cycles

SimWeaver-Real exemplifies the one-way sim-to-real paradigm. In contrast, the Real2Sim framework (Adami et al., 13 Jan 2026) introduces a vision-language active perception system whereby missing physical parameters (e.g., mass, friction, table height) are autonomously measured through VLM-generated Behavior Trees and compliant physical interaction. This produces a fast-start digital twin suitable for downstream sim-to-real learning, and supports automated refinement if sim–real discrepancies (e.g., force-profile mismatches) are detected during subsequent real-world trials. A plausible implication is that combining SimWeaver-Real's photometrically faithful simulation with VLM-driven active perception and closed-loop real2sim feedback could enable a robust, bidirectional Sim↔Real policy development workflow—mitigating the reality gap from both sides (Hu et al., 13 Jun 2026, Adami et al., 13 Jan 2026).