Incorporate additional normalization and activation symmetries

Develop a symmetry-aware treatment, in addition to the group mathcal{G} formalized for complete decoder-only transformer language models, of the diagonal rescaling symmetries of normalization layers and the scaling or sign symmetries induced by particular feed-forward activation functions, for example through weight canonicalization.

Background

The paper formalizes a symmetry group mathcal{G} that includes vocabulary, residual-stream, feed-forward-neuron, attention-head, and attention general-linear symmetries. It explicitly notes that this group is not maximal: normalization layers can admit diagonal rescalings, while some activation functions introduce scaling or sign symmetries among feed-forward neurons. These additional transformations are not incorporated into the presented GMT constructions, although the paper suggests that they could be addressed by extending the existing construction or by canonicalizing the weights.

References

In addition, some activation functions induce scaling or sign symmetries of the feed-forward neurons \citep{godfrey2022symmetries,tran2024monomial,kalogeropoulos2024scalegmn}. These symmetries are complementary to the ones we treat and can be handled on top of our constructions, e.g., by canonicalizing the weights; we leave this to future work.

— Graph Metanetworks for Transformer Based Language Models  (2610.18471 - Wolanowsky et al., 13 Oct 2026) in Section 3, subsection "Transformer Weight-Space and Symmetry Group," paragraph "Additional symmetries"