SimFoundry: Digital Twin Generation

• SimFoundry is a modular and automated system that creates simulation-ready digital twins from real-world videos with structured variations.
• It employs a three-stage pipeline—Extraction, Generation, and Augmentation—that integrates depth estimation, mesh synthesis, and affordance preservation.
• The system supports scalable policy training and sim-to-real transfer, achieving high correlation metrics and up to a 40% boost in task success through digital cousin augmentation.

SimFoundry is a modular and automated system designed for rapid, end-to-end generation of sim-ready digital twins and their structured variations from real-world video input, targeting policy learning and evaluation at scale. Leveraging modular foundation-model components for perception, reconstruction, and augmentation, SimFoundry enables zero-shot real-to-sim scene construction, affordance-preserving scene and object editing, and automated task proposal—substantially advancing the automation and fidelity of policy evaluation and transfer for robot manipulation tasks (Ranawaka et al., 26 Jun 2026).

1. System Architecture and Data Flow

SimFoundry's pipeline is structurally organized into three sequential stages—Extraction, Generation, and Augmentation—each comprising interchangeable modules. The pipeline is strictly feed-forward, taking as input a raw RGB or RGB-D video of a tabletop-style scene and producing a sim-ready digital twin suited for immediate evaluation and policy training.

• Extraction Stage: Processes the video via depth estimation ($V_{im2depth}$), typically using models like DepthAnything3 or FoundationStereo, to synthesize per-frame depth maps $D$. Keyframes are fused to produce a scene point cloud $\mathbf P_s$, while object segmentation is handled via SAM3, iteratively extracting object masks $m_i$ and using image/depth inpainting to remove segmented objects.
• Generation Stage: Each cropped object region ($p_i^{rgb}$) is passed to a mesh synthesis model ($V_{mesh}$), yielding textured mesh assets $\mathcal M_i$. Poses are aligned to the scene via $V_{pose}$ and Coherent Point Drift, solving for the similarity transform $\{\mathbf R_i,\mathbf t_i,s_i\}$. Articulation is detected by segmenting meshes and inferring joint parameters using a VLM loop, exporting URDFs. Physics properties are annotated using CoACD, and scenes are tested in PyBullet to ensure physical plausibility. For background reconstruction, SimFoundry provides both automatic (video inpainting, 3DGS training with photometric + depth supervision) and manual (COLMAP SfM) splat pipelines.
• Augmentation Stage: Starting from the baseline digital twin, SimFoundry autonomously generates "digital cousins" along three axes: object, scene, and task, while enforcing preservation of original affordances.

2. Algorithms and Mathematical Foundations

2.1 Reconstruction Losses and Alignment

Depth-supervised 3DGS training in SimFoundry minimizes the loss

$$L_{\rm splat} = \sum_{p\in \mathrm{pixels}} \|I_{\rm render}(p) - I_{\rm inpaint}(p)\|^2 + \lambda \sum_{p} w_p\, |D_{\rm render}(p) - D_{\rm inpaint}(p)|$$

where $D$0 is a depth-confidence mask and $D$1 weights depth consistency.

The similarity transform for mesh-scene registration uses the Umeyama approach: $D$2 with solution for $D$3 via SVD and direct computation of $D$4, $D$5.

2.2 Affordance-Preserving Object Variations

Generated object meshes $D$6 must satisfy

$D$7

where $D$8 denotes a set of action-enabling features (e.g., grasp patches), and $D$9 utilizes a mesh-based similarity such as Hausdorff distance.

2.3 Scene and Task Cousin Sampling

Scene cousins are generated by sampling semantic spatial predicates for each non-anchor object ($\mathbf P_s$0), followed by constrained optimization for placement. Task cousins are produced by prompting a vision-LLM (VLM) with the reconstructed scene; their outputs define new goal predicates, compiled as: $\mathbf P_s$1 used by the simulator for episode termination and automated demonstration collection.

3. Policy Training, Augmentation, and Evaluation

3.1 Data Collection Expansion

Policy learning begins with approximately 10 human teleoperated demonstrations per task, using JoyLo or VR. Data diversity is rapidly expanded via MimicGen (stitching sub-trajectories, domain randomization over textures, lighting, camera), as well as through the creation of object, scene, and task cousins; these augmentations scale dataset size multiplicatively, yielding a $\mathbf P_s$2–$\mathbf P_s$3 increase.

3.2 Policy Algorithms

SimFoundry supports training and evaluation of various policy architectures:

• Vision-language-action behavioral cloning models ($\mathbf P_s$4; Brohan et al.), GR00T N1.x, DreamZero
• FlowMatching policies (for YAM), mapping observations $\mathbf P_s$5 directly to robot actions $\mathbf P_s$6 (joint-positions, gripper commands).

3.3 Evaluation Metrics

Two core evaluation metrics are used:

• Pearson correlation ($\mathbf P_s$7) between real and sim success rates,

$\mathbf P_s$8

• Mean Maximum Rank Violation (MMRV):

$\mathbf P_s$9

which penalizes inversions in sim-vs-real task ranking.

4. Empirical Performance and Benchmarks

SimFoundry demonstrates strong correspondence between sim-based and real-world policy evaluation, achieving mean Pearson $m_i$0 and MMRV = 0.018 across 7 manipulation tasks and 5 policy architectures. Sub-task granularity (mid-task resetting) further elevates $m_i$1 to 0.95 for multi-step tasks.

For sim-to-real zero-shot transfer:

• YAM achieves 99% success on PotOnStove, 92% on StackDishware using object cousins.
• DROID achieves 100% on StackDishware with cousins, compared to 34% with only the digital twin.

Ablation studies for cousin generation yield average real-world success improvements of +17% for object cousins, +21% for scene cousins, and +40% for task cousins. Co-training in simulation and with 10–50 real demonstrations provides up to +36% on specific tasks such as ThrowAwayTrash.

Cousin Types and Performance Gains

Cousin Type	Mean Δ Success Rate (%)	Description
Object cousins	+17	Geometric/topological object variation
Scene cousins	+21	Spatial semantics and layout variation
Task cousins	+40	New goal predicates/task definitions

5. Interfaces, Integration, and Practical Application

Integration with SimFoundry follows a direct operational pipeline:

1. Record a slow panning video of the scene with minimal occlusion.
2. Launch Extraction, specifying video path and camera intrinsics ($m_i$2).
3. Optionally refine object alignments via the interactive GUI.
4. Select background reconstruction pipeline or provide custom mesh.
5. Invoke augmentation to generate digital cousins.
6. Export for simulation in IsaacLab, OmniGibson, or similar environments.

APIs and tools include:

• Modular Python API wrapping all core modules ($m_i$3, etc.)
• Interactive scene editor with 6-DoF and scale adjustment
• YAML-based task definition loader for both GUI and programmatic scenario setup
• Scriptable batch mode for automated large-scale policy evaluation, logging policy checkpoints, grids, and results

6. Significance and Operational Implications

SimFoundry enables scalable, modular, and fully automated real-to-sim scene generation, providing a platform where structured variation of scenes, objects, and tasks can be generated at scale while preserving critical affordances. This allows for reliable sim-based policy benchmarking and scalable sim-to-real transfer, demonstrated empirically via high sim-real correlation metrics and large gains in real-world task success from cousin-based augmentation. The strict modularity of each system component and the provision of programmatic, GUI, and batch interfaces facilitates flexible integration into diverse robotic research workflows (Ranawaka et al., 26 Jun 2026).