Universal interpolation below the single-head width threshold

Establish a universal interpolation theorem for noncausal self-attention with head width r<d using m=\lceil d/r\rceil independent Gaussian-initialized frozen blocks, thereby proving the conjectured threshold for generic full lifted Lie rank across batch sizes and sequence lengths.

Background

Theorem 3.5 proves that two frozen Gaussian single-head noncausal attention blocks suffice when the head width satisfies r\ge d. Remark 3.6 discusses the narrower-head regime r<d, where numerical experiments suggest that at least \lceil d/r\rceil independent frozen blocks may suffice for generic full lifted Lie rank, independently of the batch size N and sequence length n.

For the standard multihead width r=d/H with H\ge2, this suggests a threshold of at least H blocks, with one active head per block and the other heads set to zero. The numerical evidence does not establish the corresponding universal interpolation result, which would require a theorem proving the rank property and its implications for interpolation.

References

For r < d, our numerical experiments (Section M) suggest that m ≥ ⌈d/r⌉ independent Gaussian-initialized frozen blocks suffice for generic full lifted Lie rank, independently of N and n, with n ≥ 2. With the standard width r = d/H (H ≥ 2), this gives the conjectured threshold m ≥ H, using one active head per block and setting the other heads to zero. A corresponding universal interpolation theorem remains open.

— Universal interpolation for deep residual self-attention networks  (2610.01981 - Marcotte et al., 1 Oct 2026) in Remark 3.6, Section 3.1