Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators (2510.04354v1)

Abstract: Rapid progress in imitation learning, foundation models, and large-scale datasets has led to robot manipulation policies that generalize to a wide-range of tasks and environments. However, rigorous evaluation of these policies remains a challenge. Typically in practice, robot policies are often evaluated on a small number of hardware trials without any statistical assurances. We present SureSim, a framework to augment large-scale simulation with relatively small-scale real-world testing to provide reliable inferences on the real-world performance of a policy. Our key idea is to formalize the problem of combining real and simulation evaluations as a prediction-powered inference problem, in which a small number of paired real and simulation evaluations are used to rectify bias in large-scale simulation. We then leverage non-asymptotic mean estimation algorithms to provide confidence intervals on mean policy performance. Using physics-based simulation, we evaluate both diffusion policy and multi-task fine-tuned (\pi_0) on a joint distribution of objects and initial conditions, and find that our approach saves over (20-25\%) of hardware evaluation effort to achieve similar bounds on policy performance.

• The paper's main contribution is the SureSim framework that integrates a small set of real evaluations with extensive simulations to correct for simulation bias.
• It demonstrates over a 20–25% reduction in hardware trials and efficient confidence interval tightening when real and simulated outcomes are well correlated.
• The framework provides finite-sample valid confidence intervals and highlights avenues for enhancing simulation fidelity and real-to-sim mapping.

Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators

Introduction and Motivation

The paper addresses the challenge of statistically rigorous evaluation of robot manipulation policies, particularly in the context of generalization across diverse tasks and environments. Unlike static benchmarks in computer vision and NLP, real-world robot policy evaluation is resource-intensive, requiring physical interaction and manual effort. Most studies rely on a small number of hardware trials, which lack statistical assurances and limit the reliability of policy comparisons. The central question is whether large-scale simulation, when combined with a small number of real-world evaluations, can yield valid and tight confidence intervals on real-world policy performance.

Figure 1: The evaluation goal is to compute bounds on a policy's mean real-world performance over a diverse environment distribution $\mathcal{D}$, leveraging both real and simulated trials.

Prediction-Powered Inference for Policy Evaluation

The core contribution is the SureSim framework, which formalizes the combination of real and simulated evaluations as a prediction-powered inference (PPI) problem. In this paradigm, a small set of paired real and simulation trials is used to correct for simulation bias, while large-scale simulation provides statistical power. The framework is agnostic to the underlying distribution of evaluation metrics, requiring only boundedness.

The PPI estimator for the mean policy performance $\mu^*$ is given by:

$$\mu^{\text{unif}_{\text{PPI}}} = \frac{1}{n}\sum_{i=1}^{n}(Y_i - f(\tilde{X}_i)) + \frac{1}{n+N}\sum_{i=1}^{n+N}f(\tilde{X}_i)$$

where $Y_i$ is the real evaluation, $f(\tilde{X}_i)$ is the simulation evaluation, $n$ is the number of paired trials, and $N$ is the number of additional simulation trials. The first term (rectifier) corrects for simulation bias, and the second term aggregates simulation outcomes. Confidence intervals are computed using non-asymptotic mean estimation algorithms (WSR), ensuring finite-sample validity.

Experimental Methodology

Real2Sim Pipeline

The real2sim pipeline constructs simulation environments that closely match real-world setups, using 3D object models generated from single images and calibrated robot/camera parameters. The evaluation metric is a partial success score, capturing nuanced task completion levels. Two policies are evaluated: a single-task diffusion policy and a multi-task finetuned $\pi_0$ foundation model.

Empirical Results

The experiments focus on three questions:

1. Interval Tightness and Hardware Savings: SureSim achieves over 20–25% reduction in required hardware trials compared to classical real-only evaluation, for equivalent confidence interval widths.
2. Scaling Simulations: As the number of simulation trials increases, interval widths decrease efficiently, converging to a lower bound determined by the real-simulation gap (rectifier variance).
3. Correlation Sensitivity: The benefit of simulation augmentation depends critically on the correlation between real and simulated outcomes. High correlation yields substantial interval tightening; low correlation negates simulation utility.

   Figure 2: Interval width decreases with scaling simulations under moderate correlation for $\pi_0$; SureSim approaches the rectifier lower bound efficiently.

   

   Figure 3: Under low correlation, interval width does not decrease with additional simulations, indicating limited utility of simulation data.

   

   Figure 4: In sim2sim experiments, prediction-powered inference consistently outperforms classical baselines across increasing confidence levels.

Theoretical Guarantees and Baseline Comparisons

SureSim and its variants provide finite-sample valid confidence intervals, controlling Type-I error at the specified significance level. The framework is compared against classical real-only methods and control variate approaches. The latter, while sometimes yielding tighter intervals, lacks finite-sample coverage guarantees and can be overconfident, especially in small-$n$ regimes.

Implementation Considerations

• Computational Requirements: Simulation scaling is computationally efficient compared to hardware trials, but the real2sim mapping and high-fidelity simulation setup require substantial initial effort.
• Limitations: The effectiveness of SureSim is bounded by the real-simulation gap. If simulation is poorly correlated with reality, interval tightening is not possible.
• Deployment: The framework is suitable for policy evaluation in manipulation tasks with access to physics-based simulators and representative real-world datasets. For broader deployment, large-scale paired datasets and improved real2sim pipelines are necessary.

Implications and Future Directions

Practical Implications

SureSim enables statistically rigorous policy evaluation with reduced hardware effort, facilitating scalable benchmarking and reliable policy comparisons. The approach is particularly valuable for evaluating generalist robot foundation models, where coverage of diverse environments is critical.

Theoretical Implications

The work demonstrates that prediction-powered inference can be effectively adapted to robotics, providing non-asymptotic guarantees and robust mean estimation. The necessity of high real-simulation correlation for interval tightening is a fundamental limitation, motivating research into improved simulators and real2sim techniques.

Future Developments

• Improved Simulators: Enhancing simulation fidelity and real2sim mapping will increase correlation and further reduce hardware requirements.
• Active Sampling: Developing active sampling strategies for real trials can optimize evaluation budgets.
• World Models: Action-conditioned video models offer promising alternatives for scalable simulation, though their reliability for policy evaluation remains to be established.
• Dataset Expansion: Community investment in large-scale, diverse real-simulation paired datasets will be essential for generalizing the framework.

Conclusion

The paper presents a rigorous framework for robot policy evaluation that combines small-scale real-world testing with large-scale simulation via prediction-powered inference. SureSim provides finite-sample valid confidence intervals, saving significant hardware effort when simulation is sufficiently predictive. The approach is robust to the real-simulation gap, with empirical results demonstrating strong savings and reliable inference in moderate-to-high correlation regimes. Future work should focus on improving simulation fidelity, expanding datasets, and exploring alternative simulation paradigms to further enhance scalable and reliable robot policy evaluation.