Universal interpolation below the single-head width threshold
Establish a universal interpolation theorem for noncausal self-attention with head width r<d using m=\lceil d/r\rceil independent Gaussian-initialized frozen blocks, thereby proving the conjectured threshold for generic full lifted Lie rank across batch sizes and sequence lengths.
References
For r < d, our numerical experiments (Section M) suggest that m ≥ ⌈d/r⌉ independent Gaussian-initialized frozen blocks suffice for generic full lifted Lie rank, independently of N and n, with n ≥ 2. With the standard width r = d/H (H ≥ 2), this gives the conjectured threshold m ≥ H, using one active head per block and setting the other heads to zero. A corresponding universal interpolation theorem remains open.
— Universal interpolation for deep residual self-attention networks
(2610.01981 - Marcotte et al., 1 Oct 2026) in Remark 3.6, Section 3.1