Prospective OOD prediction under imperfect shortcut-label correlation

Determine whether prospective signals from source-side training dynamics persist when the shortcut is imperfectly correlated with the label and the relevant distribution shift is not specified in advance.

Background

The experiments use small CNN and MLP architectures trained on binary MNIST with a deliberately simple shortcut that is perfectly correlated with the label. The predictor successfully identifies source-side changes preceding persistent degradation on the clean OOD distribution in this controlled setting.

The paper explicitly leaves unresolved whether comparable prospective warning signals remain detectable when shortcut-label correlation is imperfect, as well as in larger models and settings where the shortcut and distribution shift are not specified in advance. The quoted sentence directly identifies the imperfect-correlation case as an unresolved issue.

References

Whether similar prospective signals persist when this correlation is imperfect remains to be determined.

— Prospective Prediction of OOD Degradation from Source-Side Training Dynamics  (2610.12397 - Sasha et al., 8 Oct 2026) in Discussion, paragraph beginning “There are several important limitations.”

It does not allow us to establish a calibrated decision threshold or to estimate the false-positive rate on MLP trajectories with no observed degradation.

— Prospective Prediction of OOD Degradation from Source-Side Training Dynamics  (2610.12397 - Sasha et al., 8 Oct 2026) in Discussion, paragraph beginning “A further limitation is that the MLP cohort contains no trajectories without observed degradation.”