Real-to-Sim-to-Real Training

• Real-to-sim-to-real training is an integrated approach that uses real-world data to create high-fidelity digital twins for scalable policy learning.
• It employs advanced reconstruction techniques such as 3D Gaussian Splatting and mesh extraction to ensure precise simulation of geometry and physics.
• The framework combines reinforcement learning, domain randomization, and online adaptation to reliably close the sim-to-real gap in robotics.

Real-to-sim-to-real (R2S2R) training is an integrated methodology for robotic and embodied AI systems that systematically closes the loop between the real world and simulation. Rather than relying solely on conventional sim-to-real transfer, R2S2R leverages real data to inform simulated reconstructions, employs simulation for scalable policy learning and evaluation, and ultimately deploys or adapts these policies back to reality, iteratively refining both simulation fidelity and policy robustness. R2S2R frameworks are increasingly critical for enabling robust, high-performance deployment in robotics, navigation, and other applications subject to visual, geometric, or dynamical domain gaps.

1. Core Framework and Methodological Principles

R2S2R frameworks proceed through a cyclical process, generally comprising three main phases:

1. Real-to-Sim Reconstruction: Real-world data (images, videos, sensor logs, or robot/scene scans) are used to generate a high-fidelity digital twin or simulation environment. Modern pipelines utilize dense 3D reconstructions with 3D Gaussian Splatting (3DGS), RGB-D scans, object-centric mesh extraction, or system identification for physical parameters (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025, Chhablani et al., 22 Sep 2025).
2. Simulation-based Policy Learning: Agents are trained in the reconstructed simulator, often with domain randomization and privileged information to maximize transferability. Training protocols include reinforcement learning (PPO, SAC, DD-PPO), behavior cloning, diffusion policy distillation, or mixtures thereof. The policy can be exposed to randomized visual features, physical properties, or stochasticity to enhance robustness (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025, Silveira et al., 21 Feb 2025, Dan et al., 11 May 2025).
3. Sim-to-Real Deployment and Adaptation: The learned policy is transferred to the real environment, possibly following fine-tuning or online domain adaptation. Real-to-sim parameter refinement and closed-loop correction are employed to align remaining variants in appearance or physics (Zhu et al., 3 Feb 2025, Chhablani et al., 22 Sep 2025).

Key to the R2S2R approach is the tight integration between real-world grounding, physically and visually realistic simulation, and policy learning strategies that account for and actively reduce the sim-to-real gap.

2. 3D Scene Reconstruction and Digital Twin Generation

Accurate real-to-sim translation requires scene reconstructions that capture both geometry and appearance. Several leading approaches include:

• Planar 3D Gaussian Splatting (3DGS): Real environments are represented as collections of anisotropic Gaussians, parameterized by $\mu \in \mathbb{R}^3$, $\Sigma \in \mathbb{R}^{3 \times 3}$, opacity $o$, and color coefficients $C$. These undergo multi-term loss minimization, combining photometric, depth/normal priors, and multi-view consistency:

$$\mathcal{L}_{\rm total} =\mathcal{L}_{\rm rec}+\lambda_{\rm scale}\mathcal{L}_{\rm scale} +\lambda_{\rm depth}\mathcal{L}_{\rm depth} +\lambda_{\rm normal}\mathcal{L}_{\rm normal} +\lambda_{\rm mv}\mathcal{L}_{\rm mv}$$

(Zhu et al., 3 Feb 2025, Chhablani et al., 22 Sep 2025).

• Hybrid 3DGS + Mesh: Scene surfaces are reconstructed using dense photogrammetric pipelines (COLMAP + OpenMVS) and object meshes (Polycam scans or ARCode). Objects are imported separately for individualized physical properties (Han et al., 12 Feb 2025, Chhablani et al., 22 Sep 2025).
• System Identification of Physical Parameters: Physical model parameters (mass, friction, velocity mapping) are inferred by fitting real-world trajectories to parametric models using least squares or differentiable simulation, yielding transition models for simulation environments (Silveira et al., 21 Feb 2025, Shi et al., 13 Mar 2025).
• Physics-Enabled Digital Twins: The reconstructed models are imported into high-fidelity simulators (Isaac Sim, MuJoCo, Habitat-Sim). Calibration routines align real and synthetic sensors, camera transforms, and physics models to minimize trajectory mismatches (Zhu et al., 3 Feb 2025, Wu et al., 6 Jul 2025, Chhablani et al., 22 Sep 2025).

These pipelines yield digital twins with photorealistic rendering and physically plausible dynamics, making them suitable for downstream policy training and reliable sim-to-real transfer.

3. Policy Learning in Simulation

With digital twins, policy training can utilize both model-free RL and imitation learning:

• Observation Spaces: Policies typically receive multi-modal input—ego-centric RGB, proprioceptive features, action history, and goal specifications. Visual input is processed with ViTs, DINOv2, or ResNet encoders; proprioception via MLPs (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025).
• Action Spaces: Raw velocity commands, SE(3) end-effector increments, or discretized navigation commands are standard, matching real robot actuation interfaces (Zhu et al., 3 Feb 2025, Wu et al., 6 Jul 2025).
• Domain Randomization and Calibration: To encourage policy generalization, simulators perform camera pose perturbations, varied lighting, object location and scale sampling, and photometric augmentations (brightness, Gaussian noise, blur) (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025, Silveira et al., 21 Feb 2025).
• RL Algorithms: PPO and SAC (with curriculum learning and HER for sample efficiency) dominate. Supervised behavior cloning or ACT transformers (for imitation) complement RL where expert data is available. Auxiliary objectives, such as InfoNCE alignment for domain adaptation, may be included (Zhu et al., 3 Feb 2025, Dan et al., 11 May 2025).
• Reward Structures: Dense rewards are crafted from geometric relationships (object-centric, keypoint-based, or VLM-generated/conditioned) and augmented with regularizers for domain shift robustness (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025, Patel et al., 12 Feb 2025, Sun et al., 29 Apr 2025).

Large-scale parallelism in simulators enables rapid policy convergence and extensive ablation studies.

4. Sim-to-Real Transfer, Adaptation, and Evaluation

The sim-to-real deployment of policies is supported by both rigorous simulation alignment and deployment-stage adaptation techniques:

• Zero-shot Transfer: High-fidelity visual and physical consistency, rigorous domain randomization, and careful system identification permit direct transfer of simulation-trained policies without additional real-world tuning in many navigation and manipulation tasks (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025, Chhablani et al., 22 Sep 2025).
• Online Domain Adaptation: Contrastive alignment (e.g., InfoNCE losses) between rendered simulated images and real observations enables continual policy adaptation post-deployment to further close sim-to-real discrepancies (Dan et al., 11 May 2025).
• Robustness via Hybrid Co-training: Co-training strategies such as RLinf-Co combine SFT on mixed real and synthetic data with RL in simulation and supervised anchoring on real examples to yield improved generalization and prevent catastrophic forgetting (Shi et al., 13 Feb 2026).
• Evaluation Metrics: Policies are evaluated on metrics such as success rate (SR), average return time (ART), trajectory alignment (RMS error), sim-to-real task progress, and correlation between simulated and real success rates ($\rho$ in [0.87, 0.97]) (Zhu et al., 3 Feb 2025, Chhablani et al., 22 Sep 2025, Shi et al., 13 Mar 2025).
• Ablative Analysis: Quantitative ablations demonstrate significant gains over baselines lacking photorealistic GS, domain randomization, or advanced policy architectures, with up to 100% simulation and real-world SR in achieved tasks (Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025, Wu et al., 6 Jul 2025, Chhablani et al., 22 Sep 2025).

A representative summary is provided below:

System & Task	Simulator	Methodology	Sim→Real Success SR	Key Advances
VR-Robo (locomotion/nav)	Isaac Sim + 3DGS	PPO+ViT, DR, mesh phys	100%/100%	GS-mesh, occlusion DR
Re³SIM (manipulation)	Isaac Sim + 3DGS	BC, privileged demos	75-58% (avg)	Hybrid GS-mesh render, DINOv2
EmbodiedSplat (navigation)	Habitat-Sim + 3DGS	DD-PPO, GS-mesh recon	+20–40% over ZS	iPhone GS capture, personalized
SimLauncher (dex/grasp)	Isaac/MuJoCo 3DGS	Pretrain BC+RL, RLPD	100%	RL bootstrapping w/ sim demos
RLinf-Co (VLA models)	Isaac Sim equiv.	SFT+RL co-training	+20–25% over SFT	RL fine-tune, real anchoring

5. Addressing the Sim-to-Real Gap: Quantitative and Practical Insights

The defining goal of R2S2R pipelines is to minimize the sim-to-real gap in both visual and physical domains. Salient empirical findings include:

• Photorealistic 3DGS-based simulation yields high sim-real correlation: Performance measured in GS-based simulators is predictive in the real world, with $\rho \approx 0.87-0.97$ (Chhablani et al., 22 Sep 2025, Zhu et al., 3 Feb 2025).
• Domain Randomization and Calibration are Necessary: Policies without DR or proper calibrated intrinsic parameters exhibit severe performance collapse (<10% SR in ablations), while GS+mesh-based policies with DR consistently yield close to perfect sim→real transfer (Zhu et al., 3 Feb 2025, Silveira et al., 21 Feb 2025).
• Robustness to Scene and Object Variation: Successful pipelines generalize to unseen objects, lighting, and random spatial initializations with only minor performance drops (≤10–15%), particularly in tasks like table clearing and bottle placement (Han et al., 12 Feb 2025, Zhu et al., 3 Feb 2025).
• Data Efficiency through Real-to-Sim: Mobile devices (iPhone/Polycam) plus automated 3DGS reconstruction enable sub-hour setup times and make end-to-end personalization feasible for navigation or manipulation agents (Chhablani et al., 22 Sep 2025).
• Quantitative Benchmarks: For challenging setups (e.g., quadruped locomotion, dense manipulation), GS+PPO+ViT achieves 100% SR and ≈5 s ART; ablated policies without these components record <25% SR and >12 s ART (Zhu et al., 3 Feb 2025).

6. Limitations, Open Challenges, and Future Directions

Despite clear empirical strengths, current R2S2R pipelines have the following limitations and open challenges:

• Deformable Objects and Complex Physics: Most R2S2R simulators target rigid objects; articulated, deformable, or fluidic manipulation remains an open problem due to absent physical system identification and mesh limitations (Han et al., 12 Feb 2025, Zhu et al., 3 Feb 2025, Patel et al., 12 Feb 2025).
• Scaling and Automation: Reconstruction fidelity is limited by completeness of scene scanning; single-image 3D geometry inference and automated mesh retrieval are under development but not yet fully robust (Patel et al., 12 Feb 2025, Sun et al., 29 Apr 2025).
• Policy Generalization: Single-scene fine-tuning risks overfitting; multi-scene augmentation and more aggressive domain randomization are promising directions to further widen transfer coverage (Chhablani et al., 22 Sep 2025).
• Integration of High-Dimensional Sensing: Vision remains dominant, but tactile sensing and cross-modal alignment pipelines are only partially explored in the R2S2R context (Church et al., 2021).
• Real-Time Correction and Residual Learning: Online correction currently leverages only visual feedback (photometric residuals); adaptive dynamics parameter learning and real-time planning inside the digital twin offer potential improvements (Abou-Chakra et al., 4 Apr 2025, Shi et al., 13 Mar 2025).
• Benchmarks and Reporting: Consistent reporting of sim→real correlations, ablation impacts, and robustness to environmental perturbations is critical for assessment and comparison (Chhablani et al., 22 Sep 2025, Zhu et al., 3 Feb 2025, Han et al., 12 Feb 2025).

Emerging work focuses on overcoming these boundaries through richer scene modeling, online adaptation, and integration with foundation models for semantic and instruction-grounded reward modeling (Patel et al., 12 Feb 2025, Sun et al., 29 Apr 2025).

7. Conclusion and Impact

R2S2R training unifies the strengths of real-world data grounding, high-throughput simulation-based learning, and robust deployment adaptation to enable high-fidelity, scalable robotic skill acquisition. Across navigation, dexterous manipulation, and multi-modal sensing tasks, the paradigm consistently narrows or eliminates the historical sim-to-real gap, delivering robust zero-shot or minimally-adapted real-world performance. Critical advances such as 3DGS-based scene capture, mesh-physics digital twins, domain randomization, and adaptive policy learning serve as the foundation for future, data-efficient, and generalizable embodied intelligence frameworks (Zhu et al., 3 Feb 2025, Chhablani et al., 22 Sep 2025, Han et al., 12 Feb 2025, Wu et al., 6 Jul 2025, Shi et al., 13 Feb 2026).