Establish whether leakage is necessary for high-quality distillation

Determine whether evolution through, and termination in, the leakage space is necessary for the expressivity of exchange-only bubble-sort Born machines when distilling GPT-2 output distributions.

Background

The distillation experiments deliberately do not penalize leakage from the logical code space. The best solutions frequently have most of their probability outside that space, suggesting that leakage may provide additional degrees of freedom for matching the target readout distribution.

However, the reported experiments do not compare otherwise matched training with and without a leakage penalty. Consequently, the paper does not establish whether leakage is merely correlated with successful optimization or is fundamentally required for the observed expressivity.