Theoretical mechanism behind the data-dependent accuracy gap

Establish the theoretical mechanism linking the Stiefel isometry constraint on transformer query and key projection matrices to the empirically increasing generalization advantage over unconstrained AdamW as the amount of training data grows.

Background

The experiments report that the performance advantage of Stiefel-constrained attention over an unconstrained AdamW baseline increases across the tested CIFAR-10 training-set sizes, with no observed crossover. The discussion interprets the constraint as a persistent restriction that removes parameter directions irrelevant to the loss and may therefore reduce fitting of noise as more data become available.

However, the paper does not derive a formal explanation for why the accuracy gap grows with data. The authors explicitly identify the precise theoretical relationship between the isometry constraint and the increasing generalization advantage as unresolved.