Real2Sim Data Generator Overview

• Real2Sim data generators are systems that convert diverse real-world inputs (e.g., videos, LiDAR, medical imaging) into simulation-ready representations.
• They employ multi-stage pipelines that include acquisition, geometric/dynamic reconstruction, simulator alignment, physics attribution, and large-scale data generation.
• These generators enable task-specific simulation improvements across robotics, medical imaging, and autonomous driving by adapting parameters to downstream objectives.

Real2Sim data generators are systems that convert real-world observations into simulation-ready representations, digital twins, or synthetic datasets. In the surveyed literature, the real input may be multi-view RGB videos, RGBD demonstrations, monocular video, LiDAR sweeps, echocardiogram videos, or an RGB scene image paired with a high-level task request; the simulation output may be Gaussian fields, collision meshes, convex planar primitives, inertial parameters, behavior trees for active sensing, patient-specific physiological simulators, or large-scale labeled data for downstream learning (Li et al., 2024, Pfaff et al., 1 Mar 2025, Kuang et al., 2024, Adami et al., 13 Jan 2026). A common motivation is that real-world data collection is costly, invasive, slow to scale, or insufficiently diverse, while naive simulation suffers from geometric, photometric, or dynamical mismatch.

1. Scope of the concept

The surveyed works do not present a single canonical architecture for a Real2Sim data generator. Instead, they instantiate the idea in several distinct ways. Some reconstruct a real scene or object as a simulation-ready digital twin. Others estimate physical parameters that are missing from a simulator. Others transform small real datasets into large synthetic corpora for policy learning, perception, or pre-training. A further variant maps real observations into simulation-style observations for policy adaptation rather than reconstructing the scene itself (Chen et al., 2022).

System	Real input	Primary simulation output
"RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator" (Li et al., 2024)	Multi-view RGB videos + MDH parameters	3DGS-based robotic simulator
"CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives" (Wang et al., 16 Dec 2025)	Casual monocular RGB video	Simulatable human motion and planar scene primitives
"Med-Real2Sim: Non-Invasive Medical Digital Twins using Physics-Informed Self-Supervised Learning" (Kuang et al., 2024)	Echocardiogram videos	Patient-specific cardiac hemodynamic simulator
"Scalable Real2Sim: Physics-Aware Asset Generation Via Robotic Pick-and-Place Setups" (Pfaff et al., 1 Mar 2025)	Robot torques + RGBD scans	Visual mesh, collision geometry, inertial parameters
"Real2Sim based on Active Perception with automatically VLM-generated Behavior Trees" (Adami et al., 13 Jan 2026)	RGB scene image + user request + partial simulator description	Physics-aware MuJoCo simulation
"Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data" (Mishra et al., 2021)	Downstream task representation	Optimal simulation parameters for synthetic pre-training

This diversity matters because the phrase “data generator” is used at different layers of the stack. In PeopleSansPeople, the generator emits rendered frames with perfect annotations for human-centric vision (Ebadi et al., 2021). In PCGen, it emits simulated LiDAR sweeps with coordinates, intensity, beam identifiers, and raydrop effects (Li et al., 2022). In ComSim, it emits pseudo-real videos paired with action sequences by passing classical simulation through a neural simulator (Qin et al., 13 Apr 2026). In Med-Real2Sim, the “generated data” are patient-specific simulated trajectories and pressure-volume loops rather than images (Kuang et al., 2024).

A frequent source of confusion is the relation between Real2Sim and Sim2Real. The distinction is not uniform across the literature. Some papers reconstruct simulation assets from real observations and then transfer policies back to reality, yielding explicit Real2Sim2Real loops (Li et al., 2024, Han et al., 12 Feb 2025). Another line instead translates real observations into simulation-style inputs at inference time, so that a policy trained in simulation can operate on real hardware through a Real2Sim observation adapter (Chen et al., 2022).

2. Canonical pipeline stages

Despite heterogeneity, many pipelines share a recognizable sequence: acquisition, geometric or dynamical reconstruction, alignment into a simulator, physics attribution, and large-scale data generation or evaluation.

Acquisition varies by domain. RoboGSim takes as input multi-view RGB videos of a static scene and Modified Denavit–Hartenberg parameters for the robot arm, then reconstructs a Gaussian scene and MDH-driven robot model (Li et al., 2024). Re$^3$Sim begins from multi-view RGB images with known intrinsics, optional depth for fine alignment, and ArUco fiducial markers on the table for coarse world-to-camera registration (Han et al., 12 Feb 2025). CRISP starts from a casual monocular RGB video, runs a visual-SLAM system to estimate camera intrinsics, per-frame poses, and depth, and lifts per-frame SMPL meshes into the world frame (Wang et al., 16 Dec 2025). PCGen begins from real LiDAR sweeps with ego-poses and reconstructs a dense static background before ray-based resimulation (Li et al., 2022).

Alignment into the simulator is often a technically separate stage. RoboGSim performs layout alignment by scaling the imported simulated table to true physical dimensions, placing a downward BEV camera at 1.6 m above the robot base in both Gaussian space and Isaac Sim, rendering segmentation masks, and shifting until the BEV layouts overlap (Li et al., 2024). ComSim similarly calibrates camera intrinsics and extrinsics, digitizes backgrounds and objects, and aligns object sizes and colors so that real action sequences can be replayed in a classical simulator to obtain paired simulated videos (Qin et al., 13 Apr 2026).

Some systems explicitly constrain reconstruction to only what the downstream objective requires. In the active-perception framework, a vision-LLM takes a system prompt, a partial simulation description, a user request, and an RGB image, then outputs detected objects, the minimal missing parameter set $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$, and a structured Behavior Tree for robotic interaction (Adami et al., 13 Jan 2026). This is a strong form of task-conditioned Real2Sim: the pipeline does not attempt full reconstruction if only a subset of parameters is needed.

Task2Sim pushes this logic further at the level of synthetic pre-training. It studies transferability from graphics-simulator data and reports that downstream performance on different tasks is favored by different configurations of simulation parameters such as lighting, object pose, and backgrounds, with “no one-size-fits-all solution” (Mishra et al., 2021). It therefore learns a mapping from downstream task representations to optimal simulation parameters and predicts parameters for unseen tasks in one shot (Mishra et al., 2021).

3. Scene and object representations

Representation choice is central because it determines both renderability and simulability. The surveyed papers employ 3D Gaussian Splatting, 4D Gaussian Splatting, NeRF-style volumetric fields, explicit meshes, convex decompositions, and planar primitives.

RoboGSim represents the reconstructed scene as a set of Gaussians

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$

where $\mu_i$ is the mean, $\Sigma_i$ the covariance, $o_i$ the opacity factor, and $c_i$ the 4-band spherical harmonics color (Li et al., 2024). Its radiance accumulation uses alpha blending along the ray,

$$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$

with

$$\alpha_i = o_i \cdot \exp\!\left[-\tfrac12\,\delta_i^T\,\Sigma_{2D}^{-1}\,\delta_i\right],$$

where $\delta_i$ is the 2D offset to the projected center (Li et al., 2024). Re$$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$0Sim also uses 3DGS for the background, but combines it with explicit mesh-based raytracing for foreground objects, then Z-buffer merges object rendering on top of the Gaussian-rendered background (Han et al., 12 Feb 2025).

The autonomous-driving Real2Sim framework extends this to 4D Gaussian Splatting. It reconstructs background with a single static Gaussian field and assigns each dynamic object its own 4D Gaussian cluster, with projected covariance

$$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$1

and front-to-back composition

$$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$2

Dynamic-object color coefficients evolve over time via a Fourier basis, and each cluster is placed in world coordinates by time-varying pose parameters $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$3 (Huang et al., 13 May 2026).

Not all Real2Sim systems favor dense radiance fields. CRISP deliberately fits planar primitives to the scene point cloud. It computes per-frame normals, clusters them with K-means, splits spatially with DBSCAN, merges across frames by optical flow and covisibility, and then fits planes with RANSAC. Each plane is converted into a thin convex cuboid primitive with default thickness $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$4, and the final scene is represented by $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$5 small convex cuboids that are “compact and cheap for collision” (Wang et al., 16 Dec 2025). This suggests that high-fidelity visual reconstruction and simulation-ready geometry can pull in different directions; CRISP explicitly prioritizes clean collision geometry over raw scene detail.

Object-centric pipelines often pass through an intermediate neural field but export explicit meshes. Scalable Real2Sim trains an object-centric NeRF-style model with alpha-transparent training and gripper-occlusion masking, then extracts a high-resolution triangle mesh by Marching Cubes, decimates it, transfers colors by sampling the learned field, and derives collision geometry by approximate convex decomposition into nearly convex components (Pfaff et al., 1 Mar 2025). SynthDrive, in contrast, reconstructs generic driving assets from a single image using Zero123++, StableNormal, InstantMesh, iterative normal-aware mesh optimization, remeshing operators, and high-resolution texture fusion before inserting the assets into reconstructed road scenes (Chen et al., 8 Sep 2025).

4. Physics attribution and parameter identification

A Real2Sim generator becomes physically usable only when geometry is paired with a parameterized dynamical model. The literature spans direct measurement, torque-based identification, physics-informed inverse problems, and reinforcement-learning-based plausibility checks.

RoboGSim extracts physical properties explicitly. Mass is obtained by weighing each object on a scale, friction is measured from sliding distance on an inclined plane of known angle, and the inertia tensor is approximated from CAD volume or measured via a compound pendulum test; these values populate ISO-standard URDF or MJCF files in Isaac Sim (Li et al., 2024). The simulator then uses NVIDIA PhysX with gravity $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$6, solver iteration count $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$7, and time step $$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$8 (Li et al., 2024).

Scalable Real2Sim pursues a more automated route. It models manipulator dynamics as

$$\Phi \subset \{\text{mass, dimensions, friction, \ldots}\}$$9

rewrites this in linear-in-parameters form $$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$0, and identifies inertial, friction, and reflected-inertia parameters from joint torques using excitation trajectories and a semidefinite program subject to pseudo-inertia positivity constraints (Pfaff et al., 1 Mar 2025). Object parameters are obtained by subtracting the identified robot-only parameters from the composite robot-plus-object parameters, followed by a renewed physical-feasibility constraint (Pfaff et al., 1 Mar 2025).

The active-perception framework estimates only the parameters requested by the task and couples reasoning to compliant interaction. It recovers the end-effector wrench from joint torques using

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$1

estimates mass from the vertical force component,

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$2

estimates friction coefficients during planar pushing via

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$3

and estimates a support-surface height by

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$4

These measurements are then written into a MuJoCo world description (Adami et al., 13 Jan 2026).

Med-Real2Sim casts the problem as a composite inverse problem. It assumes a known physics-based simulator

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$5

an unknown measurement process

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$6

and thus

$$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$7

A neural network $$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$8 is first pretrained on synthetic parameter–trajectory pairs to mimic the simulator, and a second network $$G=\{g_i\}_{i=1}^N,\qquad g_i=(\mu_i\in\mathbb{R}^3,\ \Sigma_i\in\mathbb{R}^{3\times 3},\ o_i\in\mathbb{R},\ c_i\in SH(4)),$$9 is then trained on real measurements by enforcing consistency through the frozen simulator surrogate (Kuang et al., 2024). In this formulation, Real2Sim is not mesh reconstruction but parameter inversion into a patient-specific ODE model of cardiac hemodynamics.

Human-interaction modeling provides another form of parameter identification. In the pHRI pipeline for a pelvis–strap interface, the coupling is modeled as a 6-DoF viscoelastic joint with

$\mu_i$0

or equivalently $\mu_i$1, and the parameters are identified in log-space via CMA-ES using a non-dimensional cost based on marker position and velocity residuals (Yang et al., 3 Jul 2026). An intraclass-correlation analysis then separates shareable from subject-specific parameters, permitting deployment on a new subject by refining only 5 of the 12 parameters (Yang et al., 3 Jul 2026).

CRISP validates geometry through policy learning rather than only geometric fit. After reconstructing planar primitives and contact-guided support surfaces, it trains a motion-tracking humanoid policy by PPO in Isaac Gym and reports that the approach reduces motion-tracking failure rates from $\mu_i$2 to $\mu_i$3 and delivers a $\mu_i$4 faster RL simulation throughput (Wang et al., 16 Dec 2025). The role of RL here is diagnostic as much as generative: physically implausible geometry is exposed by controller failure.

5. Data products, annotations, and downstream uses

The outputs of Real2Sim generators are often larger than the reconstructed simulator itself. Most systems are designed to produce reusable training corpora, automated annotations, or closed-loop evaluation environments.

RoboGSim’s Interactive Engine has a synthesizer mode that procedurally samples novel camera trajectories, object placements, and lighting variants, then drives robot and object motions to produce millions of RGB, depth, and segmentation frames for policy training (Li et al., 2024). Its evaluator mode inserts the policy into a loop in which Isaac Sim evolves kinematics and collision while the Gaussian renderer provides the observation, enabling “online, reproducible, and safe evaluation for different manipulation policies” (Li et al., 2024).

PeopleSansPeople is a fully parameterized, Unity-based synthetic data generator for human-centric vision. Each rendered frame produces 2D bounding boxes, 3D axis-aligned bounding boxes, semantic and instance segmentation masks, and COCO-style 17 keypoints with visibility flags $\mu_i$5, all emitted as COCO-structured JSON (Ebadi et al., 2021). On the reported benchmark, synthetic pre-training followed by fine-tuning improved keypoint AP from $\mu_i$6 to $\mu_i$7 in the 641-image few-shot regime and from $\mu_i$8 to $\mu_i$9 in the full-data regime (Ebadi et al., 2021).

SynthDrive targets autonomous driving rare cases. It automatically mines rare-case images from on-vehicle recordings and the Internet via CLIP-guided retrieval, reconstructs generic 3D assets from a single image, inserts them into real street-view footage or reconstructed scenes, and exports rare-case video frames, LiDAR point clouds, and automated annotations including 2D boxes, 3D boxes, and semantic masks (Chen et al., 8 Sep 2025). The reported asset bank contains more than 2,000 unique categories, and the synthesized data improve 2D and 3D perception performance, including a BEVFormer NDS increase from $\Sigma_i$0 to $\Sigma_i$1 and mAP increase from $\Sigma_i$2 to $\Sigma_i$3 on rare NuScenes classes (Chen et al., 8 Sep 2025).

PCGen emits simulated LiDAR with the same format as the real data, namely $\Sigma_i$4 (Li et al., 2022). Its FPA ray-casting computes a weighted return

$\Sigma_i$5

with inverse-distance weights in angle space, after which a learned surrogate raydrop model predicts a remaining ratio $\Sigma_i$6 and applies Bernoulli dropout per simulated hit (Li et al., 2022). The paper reports that adding raydrop closes part of the gap to real-data-trained 3D detection.

ComSim uses a classical simulator to generate action-aligned videos, then trains a diffusion-based neural simulator to map $\Sigma_i$7 and thereby produces pseudo-real videos paired with exact actions (Qin et al., 13 Apr 2026). This addresses a specific bottleneck in policy learning: classical simulation gives perfect action–video alignment but poor visual realism, while real video gives realism but is expensive to scale. The paper reports that with 10 real demonstrations plus 200 pseudo-real trajectories, average success is approximately $\Sigma_i$8 in-domain and $\Sigma_i$9 out-of-domain, compared with approximately $o_i$0 using only 10 real demonstrations (Qin et al., 13 Apr 2026).

Task2Sim generalizes the notion of generator control. Rather than reconstructing a specific scene, it predicts optimal synthetic-data simulation parameters for a downstream task and shows, over 20 diverse downstream tasks, that task-adaptive synthetic pre-training data significantly outperform non-adaptive parameter choice on both seen and unseen tasks, and are competitive with ImageNet pre-training (Mishra et al., 2021).

6. Performance, limitations, and recurrent debates

Reported performance indicates that Real2Sim can be quantitatively competitive, but the gains depend strongly on what is being reconstructed: appearance, contact geometry, dynamics, annotation coverage, or policy-relevant state.

In robotic manipulation, Re$o_i$1Sim reports background-only visual-gap metrics over wrist and third-person views, with PSNR improving from $o_i$2 for Polycam mesh rendering to $o_i$3 for Re$o_i$4Sim’s 3DGS rendering, and SSIM improving from $o_i$5 to $o_i$6 (Han et al., 12 Feb 2025). Using only simulated data, it reports zero-shot sim-to-real transfer with an average success rate exceeding $o_i$7 across three tasks (Han et al., 12 Feb 2025). RoboGSim reports novel-pose synthesis at PSNR $o_i$8, SSIM $o_i$9, average pixel $c_i$0, and rendering speed $c_i$1, alongside sim-to-real replay with less than $c_i$2 end-effector RMSE (Li et al., 2024).

For automated asset generation, Scalable Real2Sim reports millimeter-scale geometry accuracy, including Chamfer distance approximately $c_i$3 for the YCB mustard bottle, and inertial-parameter accuracy with mass error $c_i$4 and CoM error $c_i$5 on standard test payloads (Pfaff et al., 1 Mar 2025). The same paper also notes that inertia error remains substantially larger, at $c_i$6, than a wrist force-torque baseline’s $c_i$7 (Pfaff et al., 1 Mar 2025). This is an important corrective to the common assumption that torque-only identification is uniformly precise across all rigid-body parameters.

For human-scene modeling, CRISP’s main claim is not photorealism but simulation robustness: the failure rate reduction from $c_i$8 to $c_i$9, the increase in non-penetration from $$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$0 to $$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$1, and the throughput increase from approximately $$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$2 to approximately $$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$3 are all tied to replacing noisy geometry with planar primitives and adding contact-guided hallucination of occluded support surfaces (Wang et al., 16 Dec 2025). For pHRI, the standardized Real2Sim pipeline reports that only the identified prior model reproduces both the real interaction envelope and the correct gait adaptation, whereas overly compliant and overly stiff settings fail as extreme settings (Yang et al., 3 Jul 2026).

Several recurring limitations also appear across papers. SynthDrive explicitly states that LiDAR noise modeling $$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$4 is not yet parameterized, that current mesh optimization focuses on rigid objects, and that full inverse rendering of scene materials remains to be integrated (Chen et al., 8 Sep 2025). Re$$C = \sum_{i=1}^n c_i\,\alpha_i\,\prod_{j<i}(1-\alpha_j),$$5Sim lists highly specular or transparent objects, deformable or articulated objects, and manual assignment of physics parameters as current failure modes (Han et al., 12 Feb 2025). The driving-scene 4DGS+MPM Real2Sim reports qualitative fidelity and physics simulation examples on Waymo sequences but provides no numerical PSNR, IoU, or motion-prediction error metrics, and no ablation studies or quantitative physics-accuracy evaluations (Huang et al., 13 May 2026).

Two broader debates recur. The first concerns whether visual fidelity alone closes the sim-to-real gap. The surveyed evidence argues against that interpretation. RoboGSim motivates itself by noting that existing simulators fail at high-fidelity modeling on both texture and physics (Li et al., 2024). CRISP shows that noisy geometry with artifacts can cause motion-tracking policies with scene interactions to fail (Wang et al., 16 Dec 2025). The pHRI study shows that heuristic soft or stiff interface tuning is inadequate even when the simulator is otherwise functional (Yang et al., 3 Jul 2026). A plausible implication is that Real2Sim performance is often bottlenecked by contact geometry and physical parameterization rather than only by rendering quality.

The second debate concerns whether a single simulation configuration can be reused across tasks. Task2Sim answers negatively for synthetic pre-training data: downstream performance is favored by different configurations of lighting, object pose, backgrounds, and other simulator parameters, and there is “no one-size-fits-all solution” (Mishra et al., 2021). In encyclopedia terms, this marks a shift from scene reconstruction alone to task-adaptive simulator construction. Real2Sim, in that sense, is not merely the recovery of a digital twin; it is increasingly the recovery of the particular simulator that a downstream task actually needs.