Reliability of Zero-Shot Domain Evaluation
Determine when prediction-powered smoothing models can reliably estimate AI-system performance in an unseen domain with no observed outcome labels, using only available domain-level covariates.
References
Determining when such zero-shot domain evaluation is reliable remains an open question.
— Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
(2609.20758 - Kawano et al., 17 Sep 2026) in Section 5, Conclusion, final paragraph before Data Availability
No validated predictor of transfer success is available: the quantities examined---the share of total time spent in presolve, corpus richness, and domain-semantic similarity---do not separate the seven points, since the failure occurs on the domain with the richest corpus (CA) while the strongest gains occur on the most degenerate corpus (GISP) and on the most dissimilar domain (NNV).
— ORDO: Operation-level Round-aware Dynamic Ordering for MIP Presolve
(2610.11294 - Chen et al., 8 Oct 2026) in Appendix B.12.4, Boundary Observations on Cross-Domain Transfer