Explain why proxy preferences emerge early in training

Determine whether the initial input-level advantage of proxy cues causes goal misgeneralization to be favored early in training, and characterize how that preference is preserved or reversed by goal-generalizing versus misgeneralizing training.

Background

The paper finds that randomly initialized networks already favor the proxy in the competition environments, largely because proxy cues are more visually salient or easier to extract from the input. Goal-misgeneralizing training preserves this initial preference, whereas diversified training reverses it in favor of the intended goal.

These results suggest, but do not establish, a mechanistic explanation for why proxy-based policies arise early in training. The open problem is to test this hypothesis directly through analyses of early optimization, cue learning order, and the evolution of proxy-versus-goal influence.

References

The proxy's advantage at initialization is therefore mostly an input effect, in line with networks preferring the more easily extracted feature citep{hermann2020shapes}, that GM training does not overturn, and the comparison with Non-GM agents carries the result. A prior of this kind may explain why misgeneralization is favoured early in training, which we leave as an exciting open direction.

— What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization  (2610.19731 - Barth et al., 14 Oct 2026) in Appendix A, Section “Why does the proxy win on the shared axis?”