Determine the direction of bias caused by upstream training exposure

Determine whether the inclusion of calibration and validation images in the upstream experts’ training data introduces a systematic positive or negative bias in the conformal risk certificate when its scores are applied to expert-unseen official test images.

Background

The certificate relies on exchangeability between calibration scores and test scores. However, the calibration and validation images are included in the upstream experts’ training sets, whereas the official test images are not. Consequently, the benchmark split alone does not establish the exchangeability condition needed for the standard finite-sample guarantee.

The paper performs a two-sample check within the training-side data, but explicitly notes that this does not test the transition from expert-seen calibration data to expert-unseen test data. It further observes that the two experts in a pair share the same exposure and that validation-to-test shifts have opposite signs across experts, preventing a straightforward determination of the bias direction.

References

The sign of the bias is not determined --- both experts of a pair share the same exposure and harm is a comparison between them, so exposure does not enter it monotonically; the $D_{\mathrm{val}}$-to-$D_{\mathrm{test}}$ shift in mean expert $LPIPS$ even has opposite signs inside the v1 pair ($+0.00732$ for DiffLL, $-0.01822$ for Retinexformer), scene composition dominating the level.

— When Should a Low-Light Pipeline Generate? Auditable Risk Control for Selective Generative Enhancement  (2610.21335 - Xie et al., 15 Oct 2026) in Section statistical scope of the final-test claims, paragraph “What that costs the certificate” and following paragraph “What the record still supports”