Effect of the block-orthogonal multi-head constraint

Determine whether enforcing mutual orthogonality among all transformer attention heads through the block-orthogonal Stiefel variant improves or degrades performance relative to independent Stiefel frames.

Background

The paper introduces a block-orthogonal multi-head extension that stacks the query frames of all heads into a single orthogonal matrix, thereby forcing every head subspace to be orthogonal to every other head subspace. This is a stronger constraint than maintaining independent Stiefel frames and is motivated by the observed redundancy among attention heads.

Although the variant is implemented and unit-tested, it is not used to produce the reported experimental results. Its effect on model performance therefore remains unresolved: the stronger constraint could either help by preventing redundancy or hurt by restricting useful overlap between heads.