Causal Effect of Q/K-Norm Growth on GPBand Shifts

Establish whether growth of query and key norms during training causally shifts the Global Positional Band toward lower frequencies in RoPE Transformers without QKNorm, using direct norm trajectories or equivalent training-dynamics measurements.

Background

The controlled experiments show that removing QKNorm shifts the observed frequency band toward lower frequencies. The authors propose that increasing query and key norms may require such a shift to preserve stable global positional representations, but they explicitly characterize this explanation as tentative. Direct measurements over training are needed to determine whether norm growth is the causal mechanism behind the observed displacement.

References

This suggests one possible explanation for part of this offset: without QKNorm, the growth of Q/K norms may need to be compensated for by shifting the band toward lower frequencies and smaller rotary angles to maintain stable global positional representations. This interpretation remains a hypothesis; direct norm trajectories would be needed to verify the causal training dynamics.

— Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design  (2609.02986 - Shi et al., 2 Sep 2026) in Appendix, Section 3.1, “Controlled Analysis of GPBand,” paragraph “Effect of QKNorm”