Real-to-Sim Policy Evaluation Framework

• Real-to-Sim Policy Evaluation Framework is defined as a methodology that uses calibrated simulation environments reconstructed with real-world data to reliably predict and rank policy performance.
• It employs controlled perturbations, Bayesian uncertainty models, and iterative alignment loops to minimize the gap between simulated outcomes and real-world behavior.
• The framework enables reproducible, cost-effective benchmarking across robotics and AI applications, ensuring robust policy selection and risk-aware performance evaluation.

A Real-to-Sim Policy Evaluation Framework provides a principled and scalable methodology for evaluating and selecting control or decision-making policies for robots and agents by leveraging high-fidelity simulation environments reconstructed, tuned, or otherwise calibrated using real-world data. In contrast to sim-to-real workflows—where the focus is on transferring policies trained in simulation to the real world—real-to-sim evaluation centers on predicting, ranking, and interpreting the real-world performance of policies by systematically running them in simulator instances as proxies, with the simulators periodically grounded, corrected, or enriched with real-world observations, demonstrations, or statistical feedback. Modern real-to-sim frameworks enable cost-effective, reproducible, and robust policy benchmarking, integrated confidence bounds, and detailed behavioral analysis across a variety of robotics and artificial intelligence domains.

1. Framework Motivation and Conceptual Foundations

The principal motivation for real-to-sim policy evaluation is to overcome the prohibitive cost, risk, and unreproducibility of large-scale or exhaustive real-world robotic experimentation, especially as policy complexity, generalizability, and environment diversity increase. Unlike pure sim-to-real transfer, which emphasizes training for robust deployment, the real-to-sim paradigm constructs validated simulation surrogates for critical evaluation, policy selection, and robust checkpointing.

Two conceptual premises support this approach:

• Simulators as Statistical Proxies: Simulators are explicitly calibrated, adapted, or reconstructed to minimize behavioral discrepancies relative to the real world (e.g., trajectory errors, success rates, sensory likeness, and critical distributions).
• Controlled and Reproducible Variability: Simulation enables controlled perturbations, domain randomization, and high-throughput experimentation to probe policy robustness, sensitivity, and coverage, impossible to achieve safely or consistently in physical systems.

This alignment is quantified by metrics such as the Pearson correlation coefficient $r$ between policy performances in real and simulation, Mean Maximum Rank Violation (MMRV), and more sophisticated divergence or transfer-coherence metrics (Li et al., 2024, Zhang et al., 6 Nov 2025, Yang et al., 14 Aug 2025).

2. Simulator Construction, Calibration, and Gap Mitigation

State-of-the-art real-to-sim frameworks employ a suite of methods for closing both the control and perception (visual) gap between real-world robotics and their virtual counterparts:

• Scene and Object Reconstruction: Photorealistic reconstruction is achieved through methods such as 3D Gaussian Splatting (3DGS) from consumer-grade RGB-D scans, producing digital twins with high-fidelity mesh or splat-based rendering (Chhablani et al., 22 Sep 2025, Zhang et al., 6 Nov 2025, Zhu et al., 3 Feb 2025).
• Physics and System Identification: Calibration of robot dynamics (e.g., mass, friction, damping coefficients) often uses offline system identification from trajectory data with parameter optimization (e.g., simulated annealing or differentiable physics gradients) (Li et al., 2024, Shi et al., 13 Mar 2025, Jangir et al., 27 Oct 2025, Yu et al., 11 Apr 2025).
• Visual Matching and Texture Baking: Realistic visuals are obtained by compositing green-screen backgrounds, baking real object textures onto sim meshes, and performing color alignment (e.g., polynomial color correction) (Li et al., 2024, Zhang et al., 6 Nov 2025).
• Residual Uncertainty Modeling: Advanced approaches model not just parametric mismatch but residual, unmodeled effects using Bayesian score-based diffusion models that infer both calibration and fidelity shifts from short real trajectories (Yu et al., 11 Apr 2025).
• Direct Real-to-Sim Alignment Loops: Iterative frameworks such as SimOpt (Chebotar et al., 2018), RSR loop (Shi et al., 13 Mar 2025), LoopSR (Wu et al., 2024), and bi-level RL (Anand et al., 20 Oct 2025) successively alternate real-world rollouts and simulation affinity optimization, updating simulator states and domains to more closely reflect observed real behavior.

Table 1 summarizes typical gap mitigation techniques.

Gap Type	Mitigation Strategy	References
Control	PD gains, system ID, sim param tuning	(Li et al., 2024, Jangir et al., 27 Oct 2025, Shi et al., 13 Mar 2025)
Visual/Percept.	3DGS, baking, compositing, color align	(Chhablani et al., 22 Sep 2025, Zhang et al., 6 Nov 2025, Li et al., 2024)
Residual/Noise	Bayesian/posterior calibration	(Yu et al., 11 Apr 2025, Wu et al., 2024)

3. Evaluation Protocols and Performance Metrics

Effective real-to-sim evaluation requires robust, multi-faceted metrics and protocols to ensure that performance in simulation tracks, predicts, or bounds behavior in the real world.

Key elements include:

• Paired Evaluation: Policies are evaluated on matched initial conditions in both sim and reality, with simulation platforms engineered to support overlays and replay for consistency (Zhang et al., 6 Nov 2025, Li et al., 2024, Jangir et al., 27 Oct 2025).
• Scalar and Ranking Metrics:
  • Pearson $r$: Measures linear correlation of success rates or other scalar scores across policies (Li et al., 2024, Zhang et al., 6 Nov 2025).
  • MMRV: Penalizes mis-ranking proportional to performance difference, protecting against misleading swaps (Li et al., 2024, Zhang et al., 6 Nov 2025).
• Behavioral and Robustness Analysis: Policy variants are ranked by completion, trajectory statistics, sensitivity to perturbations (lighting, texture, pose changes), and finer-grained behavioral breakdowns (Yang et al., 14 Aug 2025, Jangir et al., 27 Oct 2025).
• Human and VLM-Guided Scoring: Integration of vision-LLM (VLM)–based automated video assessment, and scalable human preference judgments for qualitative and nuanced behavioral evaluation (Jangir et al., 27 Oct 2025).
• Statistical Confidence: SureSim (Badithela et al., 5 Oct 2025) furnishes finite-sample confidence intervals for real-world mean performance $\mu^*$ by rectifying large-scale sim averages using paired real–sim evaluations, leveraging prediction-powered inference for Type–I error control.

4. Policy Evaluation Workflows and Trade-offs

Different frameworks instantiate real-to-sim evaluation at varying points along the robot learning pipeline:

• Pre-Deployment Selection: Validate-on-Sim/Detect-on-Real (VSDR) (Leibovich et al., 2021) uses out-of-distribution feature analysis of real images under trained policies to score and rank candidates without direct on-robot trials.
• Continuous/Looped Adaptation: LoopSR (Wu et al., 2024) encodes short real trajectories to recover environment parameters for online policy retraining, closing the occupancy-shift gap iteratively within a lifelong adaptation regime.
• Batch Policy Benchmarking: SIMPLER and RobotArena∞ simulate batch evaluation under controlled perturbations with quantitative and subjective scoring, supporting scalable comparison of collected and/or foundation policies (Li et al., 2024, Jangir et al., 27 Oct 2025).
• Bayesian and Model-Based Adaptation: Neural Fidelity Calibration (NFC) uses diffusion-based Bayesian inference to sample calibrated simulator-residual parameter sets from observed real transitions, enabling calibrated, risk-aware fine-tuning (Yu et al., 11 Apr 2025).

A fundamental trade-off highlighted in multiple studies concerns the proportion of in-domain (real-scene reconstructed) data used for fine-tuning: moderate inclusion (approximately 25–30%) increases real-world performance, but excessive use (>50%) may degrade cross-scene generalization (Cai et al., 13 May 2025).

5. Applications and Empirical Results Across Domains

Empirical validation of real-to-sim frameworks spans diverse robotics and agent-based domains:

• Manipulation: High-fidelity digital twins and adapted simulation enable accurate ranking and data-efficient improvement on grasping, packing, and pushing tasks involving both rigid and soft-body objects (Pearson $r$ > 0.90, MMRV < 0.10) (Zhang et al., 6 Nov 2025, Li et al., 2024, Shi et al., 13 Mar 2025).
• Navigation and Locomotion: In navigation, photorealistic reconstructions and diffusion-based policy evaluation yield strong real–sim alignment (SRCC ≈ 0.9), with Gaussian Splatting reconstructions supporting both policy adaptation and fine-tuning (Chhablani et al., 22 Sep 2025, Zhu et al., 3 Feb 2025, Cai et al., 13 May 2025).
• Policy Evaluation in Social Systems: Agent-based real-to-sim policy evaluation enables quantitative benchmarking on argument coverage, behavioral consistency, and outcome effectiveness, revealing substantial challenges in reaching expert-level decision support (coverage rates ≲25%) (Kang et al., 11 Feb 2025).
• Sim-to-Real Policy Tuning: AdaptSim (Ren et al., 2023) and bi-level RL (Anand et al., 20 Oct 2025) demonstrate that policy-driven and task-oriented sim calibration can achieve 1–3× higher asymptotic task performance compared to system identification or uniform domain randomization, with 2× data efficiency on real hardware.
• Open-Ended Benchmarking: RobotArena∞ enables reproducible comparison of policies trained under disparate training regimes on large, automatically constructed simulation testbeds, with performance scored by automated and human evaluators and stress-tested under targeted perturbations (Jangir et al., 27 Oct 2025).

6. Methodological Challenges, Limitations, and Future Directions

Despite significant advances, limitations persist. Notably:

• Residual perceptual and physical mismatch in unmodeled scenarios or over-parameterized environments remains a challenge, particularly for cluttered or highly articulated scenes (Zhang et al., 6 Nov 2025, Chhablani et al., 22 Sep 2025).
• Data acquisition and reconstruction (e.g., from phone scans or digital twins) incurs a one-time setup cost per new environment or object (Zhang et al., 6 Nov 2025, Chhablani et al., 22 Sep 2025).
• Excessive scene-specific tuning can induce overfitting and harm cross-scene generalization (Cai et al., 13 May 2025).
• Few frameworks currently address full uncertainty quantification or robust adaptation to rare, safety-critical anomalous states; recent work with Bayesian or score-based diffusion models provides a template for future efforts (Yu et al., 11 Apr 2025).

Future work is likely to:

• Further unify policy, model, and environment adaptation under Bayesian or contrastive learning schemes for efficient, risk-aware deployment.
• Extend 3DGS- and differentiable-simulation-based workflows to mobile manipulation, articulated and multi-agent systems.
• Explore human-in-the-loop real-to-sim policy evaluation at scale, leveraging VLMs for scalable, interpretable robotic benchmarking.
• Integrate continual and “lifelong” adaptation for autonomous systems functioning over prolonged deployments and changing environments, closing the sim-to-real/performance gap throughout the system’s operational lifetime.

7. Cross-Domain Extensions and Impact

The real-to-sim policy evaluation paradigm now underlies not just robot manipulation and navigation, but also extends to agent-based simulation in policy assessment, where frameworks such as PolicySimEval quantify the discrepancy between simulated and expert-driven reasoning for complex policy scenarios (Kang et al., 11 Feb 2025). These advances collectively accelerate safe, systematic, and statistically principled evaluation and selection of AI/robotics policies in open, dynamic, and uncertain real-world environments.