Resolve the calibration-dependent performance of the covariance-aware Deming correction

Determine whether the covariance-aware Deming slope correction (deming_cov_slope) outperforms or underperforms the uncorrected baseline on ranking accuracy, resolving the direction of its small performance gap across the two simulation calibration arms.

Background

The paper compares several proxy-ranking estimators under two simulation calibrations: a random-effects calibration and a tail-correlation calibration. The covariance-aware Deming estimator, deming_cov_slope, has a small negative gap relative to the uncorrected baseline in the random-effects arm but a small positive gap in the tail-correlation arm. Unlike the larger and more consistent losses observed for aggressive de-noising procedures, the sign of this mild correction’s advantage is not stable across calibration choices, leaving its ranking performance unresolved.

References

One comparison, at the mild end of the ladder, does not replicate: deming_cov_slope is $2.8$ points behind the baseline in the random-effects arm and $0.8$ points ahead of it in the tail-correlation arm, so the direction of that small gap remains open.

— The Noise Is the Signal: Correlated Sampling Error Is Rank-Informative for Proxy Metric Selection  (2610.08194 - Provenzano, 6 Oct 2026) in Section “Three loss functions and what they cannot settle,” subsection “Both calibration arms agree”

The natural next step, which we leave for future research, is to run them on archives with well-measured north stars and effective sizes in the hundreds, where the map predicts that the corrections should win; finding that they do would mark the boundary of our result from the other side.

— The Noise Is the Signal: Correlated Sampling Error Is Rank-Informative for Proxy Metric Selection  (2610.08194 - Provenzano, 6 Oct 2026) in Section 8, “Conclusion”