Unknown optimization dynamics of hash-conditioning

Characterize the optimization dynamics by which hash-conditioning affects Transformer training, including identifying why and under what conditions varying random hash prefixes does not disrupt optimization or generalization, and formalizing convergence behavior of hash-conditioned training.

Background

The authors observe strong empirical benefits of hash-conditioning on algorithmic creativity and diversity but note that the optimization aspects of how this technique works are not understood.

They reference further discussion about noise-injection and its implications, highlighting an unaddressed gap in theoretical understanding of why hash-conditioning trains stably and generalizes.