X4Val: Learning Neural Surrogates for Variance-Reduced Policy Evaluation

Abstract: Rigorous evaluation of learning-based robotic systems is an essential prerequisite for deployment. However, real-world test data is expensive to gather; moreover, in a typical iterative development context, data gathered from the latest policy is necessarily limited in scale. This motivates evaluation methodologies that make use of heterogeneous data sources, including simulation, historical policy logs, and data collected from related platforms or environments. While such auxiliary data are abundant and inexpensive, they are generally not directly representative of real-world outcomes -- for example, performance in simulation may differ substantially from performance in the real world -- making their principled use for high-confidence performance estimation challenging. In this paper, we introduce X4Val, a general framework for variance-reduced real-world metric estimation in the presence of non-paired, multi-domain data. X4Val embeds samples from real and auxiliary domains into a shared representation space and learns a transferable predictor of real-world metrics; this learned predictor is then incorporated into a control-variates estimator, enabling variance reduction even when paired samples are unavailable. We provide theoretical analysis and empirical evaluations on autonomous driving and real-world robot manipulation tasks, domains across which X4Val achieves up to 38.4% variance reduction and demonstrates consistent improvements over strong baselines. These results show that non-paired, heterogeneous data can be leveraged to substantially improve the sample efficiency of rigorous robotic system validation.

• The paper presents X4Val, a framework that leverages shared embeddings and neural surrogate models to achieve unbiased variance reduction in policy evaluation.
• It integrates multi-domain unpaired auxiliary data using cross-fitted estimation to produce efficient estimators in autonomous driving and robotic manipulation.
• Empirical results show up to 38.4% variance reduction over Monte Carlo methods, enabling more reliable performance validation for safety-critical robotics.

Variance-Reduced Policy Evaluation via Neural Surrogates: The X4Val Framework

Motivation and Context

Performance validation in robotics, especially in safety-critical domains like autonomous driving and mobile manipulation, demands accurate estimation of deployment metrics under realistic conditions. However, the cost and practicalities of real-world testing restrict the quantity of available data, impeding statistical confidence in evaluation. This paper introduces "X4Val: Learning Neural Surrogates for Variance-Reduced Policy Evaluation" (2606.05159), a general framework for leveraging heterogeneous, non-paired multi-domain data to reduce estimator variance during policy metric evaluation. Unlike classical control variates that require sample-wise pairing, X4Val projects real and auxiliary samples into a shared embedding space, learns a neural surrogate for the deployment metric, and incorporates this surrogate as a control variate, facilitating principled, unbiased variance reduction.

Figure 1: X4Val utilizes diverse unpaired data sources by projecting them to a shared space and learning a transferable metric predictor, enabling broader data usage and lower variance compared to standard pairwise control variate methods.

Methodological Advances

Shared Embedding and Neural Surrogate Learning

X4Val proceeds through three stages: shared embedding, surrogate training, and variance-reduced estimation. First, all available data (including scenario features, auxiliary signals, and metrics from various domains) are mapped to a unified representation space. This embedding is tailored to maximize correlation with the target metric, often using vision-language foundation model features or hand-engineered scenario features.

Second, a neural predictor for the real-world metric is trained via transfer or meta-learning, utilizing abundant surrogate data and minimally labeled deployment-domain samples. The surrogate is optimized either directly for variance reduction (by minimizing the empirical residual variance plus the variance in surrogate predictions) or via post-hoc scaling to align with classical control variate intuition. Notably, the surrogate does not require calibration; systematic bias is corrected with residuals computed on target-domain samples, preserving unbiasedness.

Third, the metric mean estimator combines the surrogate's predicted mean (over scenario-only data) with the average residual error (from labeled deployment samples), yielding strictly unbiased estimators with variance decomposed into surrogate estimation and residual correction components.

Statistical Guarantees and Cross-Fitted Estimation

To maximize sample efficiency, X4Val employs cross-fitting: the available labeled metric data are partitioned into folds, with surrogates trained excluding each fold, whose held-out residuals contribute to the final estimator. This approach leverages all labeled data for both surrogate improvement and residual correction, further reducing variance without introducing bias.

Empirical Evaluation: Autonomous Driving Case Studies

Geographic Transfer

A core use case is policy deployment to novel environments. The paper simulates transferring an AV policy trained and validated in the United States to deployment in Germany, leveraging extensive US evaluation logs as auxiliary data, limited metric-labeled data from Germany, and scenario-only data from the target region. X4Val uniquely combines neural predictors from both domains, utilizing cross-fitting and variance-oriented objectives to maximize estimator efficiency.

Figure 2: X4Val efficiently combines target-domain (Germany) evaluation data with auxiliary data from past US evaluations, reducing variance and yielding consistent improvements over baselines.

Strong numerical results are presented: X4Val consistently achieves 15–20% variance reduction in the mean Final Displacement Error (FDE) estimation, outperforming Monte Carlo and region-adapted control variate baselines. The method's efficacy is robust across varying amounts of available target-domain evaluation data.

Iterative Policy Development

In iterative autonomy programs, new policies are developed and evaluated using limited metric-labeled data, while historical evaluation logs exist for earlier policies. X4Val leverages this historical corpus for meta-learning, producing transferable predictors that inform current evaluation.

Figure 3: X4Val achieves maximal variance reduction by fusing current policy's limited evaluation data with historical logs, outperforming conventional and neural control variate baselines.

X4Val yields up to 38.4% variance reduction relative to Monte Carlo estimators, demonstrating significant efficiency even when classical control variate signals are already strongly correlated. Incorporation of auxiliary policy logs offers substantial gains beyond conventional methods.

Application to Robot Manipulation

X4Val is extended to block-stacking tasks using both simulation (ManiSkill) and physical hardware (Franka Panda). Here, substantial simulation data but scarce real-world metric-labeled samples are available, and paired rollouts are infeasible. Scenario-only images and learned DINOv2-based embeddings enable transferable surrogate learning.

Figure 4: X4Val leverages unpaired simulation and real-world data to reduce variance in block-stacking policy evaluation; variance reduction improves as cross-fitting folds increase.

Across experiments, X4Val consistently yields lower variance estimates for real-world policy success rate than standard Monte Carlo, with the variance reduction scaling with fold count in cross-fitting. This demonstrates the method's efficacy and generalization to domains where paired data cannot be obtained.

Practical and Theoretical Implications

X4Val generalizes classical control variate methods to realistic robotics settings characterized by abundant unpaired auxiliary data, multi-domain logs, and sharply limited real-world metric data. It strictly preserves unbiasedness by correcting surrogate bias, provides tighter confidence intervals, and enables statistically rigorous evaluation even in heterogeneous data regimes. Limitations are noted: all scenario-only and metric-labeled samples used for estimation must be drawn from the deployment distribution; auxiliary data inform surrogate learning but do not themselves define the target expectation.

Practically, X4Val offers a pathway for AVs and robots to be validated more efficiently and rigorously prior to real-world rollout, reducing the cost and risk associated with metric estimation. Theoretically, the work motivates future extension to other estimation targets (quantiles, conditional risks), active learning for prioritized data collection, and further integration with transfer, meta-learning, and off-policy evaluation techniques in RL.

Conclusion

X4Val provides a principled and scalable framework for variance-reduced policy evaluation using neural surrogates trained on multi-domain, unpaired auxiliary data. Empirical results across autonomous driving and robotic manipulation demonstrate consistent improvements in sample efficiency—up to nearly 40% variance reduction—over both traditional Monte Carlo and advanced control variate baselines. The approach is widely applicable to iterative, cross-environment, and cross-platform robotic system validation, and its methodology opens new directions for rigorous, cost-effective deployment metric estimation in machine learning for embodied systems.