Signed-Permutation Coordinate Transport for RMSNorm Transformers

Abstract: Modern LLM workflows move coordinate-indexed objects across checkpoints: steering vectors, sparse autoencoders, top-$k$ neuron sets, attribution lists, and merge alignments. This is only well posed after fixing the model's residual-stream gauge, which we show is architecture-dependent: LayerNorm residual charts have permutation gauge $S_d$ (up to a global sign flip), while RMSNorm charts with generic per-channel gain have signed-permutation gauge $B_d = S_d \ltimes {\pm 1}^d$. Permutation-only alignment is therefore symmetry-incomplete for RMSNorm models. We introduce sign-marginalized Hungarian matching and prove a sharp failure mode: with decorrelated coordinates, raw signed-correlation matching has a structural permutation-accuracy ceiling at the positive-sign fraction of the true gauge, which sign-marginalization removes. We then make coordinate-preserving transport, not function-level merging, the primary object: composing saved-checkpoint local $B_d$ gauges along same-base fine-tuning trajectories recovers 91.1% of cross-run coordinates at 1500 steps versus 60.3% for endpoint matching, and the gain is not explained by merely routing through the base. The recovered gauge transfers tools that permutation-only alignment breaks: TinyLlama SAE reconstruction has NMSE 0.004 under $B_d$ versus 1.08 under $S_d$; Qwen sentiment steering preserves 95.8% of its effect versus 17.2%; refusal steering reverses sign under $S_d$; coordinate-preserving merges behave the same way. The same covariance governs stateful training: signed transport of AdamW state preserves the resumed trajectory, while permutation-only state follows a different one from a functionally identical checkpoint. Finally, gauge-sweep audits show index-level interpretability claims are reproducible only relative to an explicit gauge.

• The paper establishes that RMSNorm Transformers require a signed-permutation (B_d) gauge, refining the symmetry from S_d in residual streams.
• The methodology introduces sign-marginalized assignment and atlas-based transport, achieving significantly improved cross-run recovery and artifact consistency.
• The paper reveals that proper gauge-correct alignment enhances model merging, interpretability, and transfer of neuron-level artifacts.

Signed-Permutation Gauge and Coordinate Transport in RMSNorm Transformers

Overview

The paper "Signed-Permutation Coordinate Transport for RMSNorm Transformers" (2606.31963) critically re-examines the implicit symmetry group underlying coordinate-indexed operations in modern Transformer LLMs. The author demonstrates that, in RMSNorm-based architectures, the correct discrete gauge group for the residual stream is not the permutation group $S_d$ as in LayerNorm models, but the signed-permutation (hyperoctahedral) group $B_d = S_d \ltimes \{\pm 1\}^d$. This has broad implications for coordinate transport, transfer of neuron-level artifacts, and interpretability: in particular, approaches that ignore the sign symmetry provably destroy cross-checkpoint correspondence of coordinate-indexed objects.

Native Gauge Structure: From $S_d$ to $B_d$

The central technical result is an exact characterization of the residual-stream gauge in Transformers. For architectures with LayerNorm, the per-coordinate symmetry is $S_d$ (permutations) modulo a global sign. In contrast, for RMSNorm (as in Llama 2, Qwen2.5), the residual stream allows both permutations and independent sign flips of each coordinate; thus, the maximal discrete coordinate-preserving symmetry is $B_d$, with order $2^d d!$.

Figure 1: Symmetry-boundary validation for the native gauge. LayerNorm models are invariant to permutations but fail under independent sign flips; biases must transform under the residual gauge; RMSNorm models pass full signed-permutation gauges. Dashed lines indicate the $\sim10^{-4}$ numerical tolerance threshold.

This expanded symmetry group immediately implies that coordinate-indexed artifacts—such as neuron sets, steering vectors, sparse autoencoders (SAE) dictionaries—are identifiable only up to $B_d$. Failure to account for the sign flips causes index-level correspondence to collapse under even trivial gauge transformations. The theoretical analysis is complemented by comprehensive invariance audits across models, with RMSNorm models passing full $B_d$ tests to numerical tolerance (Figure 1, Table 1 in the original content).

Activation Matching and the Ceiling of Permutation-Only Alignment

Existing neuron alignment and weight matching procedures generally optimize over permutation assignments for coordinate recovery. However, in RMSNorm settings, signed permutations are needed for completeness. The author provides a precise population-theoretic result: when source coordinates are decorrelated and the true gauge includes sign flips, signed-correlation matching exhibits a strict accuracy ceiling governed by the fraction of positive signs—in expectation, exactly $B_d = S_d \ltimes \{\pm 1\}^d$0 for random sign configurations. Only sign-marginalized assignment, which matches by absolute cross-correlation, can recover all coordinates.

This is empirically validated: in 7B/8B RMSNorm models subjected to recorded $B_d = S_d \ltimes \{\pm 1\}^d$1 basis changes, sign-marginalized assignment achieves 100% accuracy, while traditional signed matching remains at ~50% (Table in the main text).

Coordinate Transport: Atlas Construction and Cross-Run Recovery

A major application is in the parallel transport of coordinates along fine-tuning trajectories. Rather than one-shot endpoint matching, the approach composes local $B_d = S_d \ltimes \{\pm 1\}^d$2 gauges along a sequence of checkpoints, forming an "atlas" that maintains coordinate correspondence across the trajectory. On Qwen2.5-1.5B fine-tuning runs, this method recovers $B_d = S_d \ltimes \{\pm 1\}^d$3 of cross-run coordinates at 1500 steps, compared to only $B_d = S_d \ltimes \{\pm 1\}^d$4 recovery for endpoint matching.

Figure 2: Per-pair transport advantage $B_d = S_d \ltimes \{\pm 1\}^d$5 for cross-seed and cross-dataset Qwen2.5-1.5B runs. Transport consistently improves cross-seed recovery; cross-dataset gains are heterogeneous.

Crucially, the improvement is not an artifact of routing through the base checkpoint—direct composition of endpoint matches through the base performs worse. Only sequential composition of local $B_d = S_d \ltimes \{\pm 1\}^d$6 matches supports robust, high-accuracy transport. Theoretical results establish necessary and sufficient conditions (zero cycle holonomy) for the existence of globally consistent coordinate atlases.

Tool Transfer and Artifact Preservation: The Impact of Gauge-Correct Alignment

The practical impact is sharp: steering vectors, SAE dictionaries, LoRA adapters, and other learned coordinate-indexed updates are completely mishandled by permutation-only alignment. For instance, transferring a sentiment steering vector or SAE dictionary across a $B_d = S_d \ltimes \{\pm 1\}^d$7-scrambled checkpoint via $B_d = S_d \ltimes \{\pm 1\}^d$8 matching reduces effectiveness or inverts the direction entirely. In Table~4 of the original text, signed ($B_d = S_d \ltimes \{\pm 1\}^d$9) recovery achieves an SAE NMSE of $S_d$0 on TinyLlama (matching upper bound), while $S_d$1-only transport yields an NMSE of $S_d$2. For LoRA adapters on Qwen2.5-1.5B, signed transport recovers nearly all accuracy (0.944 vs. reference 0.945), while permutation-only alignment collapses to the base value (0.523).

The transfer laws—exactly characterized for all artifact types—demonstrate that $S_d$3 alignment discards all sign orientation in residual-indexed vectors, destroying functional transfer. Crucially, optimizer state (such as AdamW moments) is also governed by the same gauge covariance, with the native $S_d$4 group constituting the maximal group for which AdamW state is representable under standard data structures.

Merging, Interventions, and Interpretability: Strong Numerical Effects

In coordinate-preserving model merging, the sign component is critical: omission leaves large loss barriers in interpolation. For Llama-2-7B, signed alignment eliminates the merge loss peak ($S_d$5); permutation-only alignment is not just suboptimal, but actively harmful in some settings.

Similarly, audit protocols for interpretability pipelines that name coordinates (e.g., "which neuron carries concept X?") must specify the gauge—raw coordinate names are non-reproducible otherwise. Audits applying exact $S_d$6 transformations show invariance in behavior, while coordinate-level indices move unless appropriately gauge-mapped.

Theoretical Implications and Connections

The work clarifies when more general continuous symmetry groups (orthogonal transformations, as in rotation-based fusion) are justified: only when dense, coordinate-indifferent bases are permitted. For native parameterizations and sparse coordinate-indexed interventions, $S_d$7 is both necessary and sufficient for gauge-consistent transport.

The structural theorems link group-theoretic properties to the viability of various merging, alignment, and interpretability approaches in LLMs. The negative result—non-existence of canonical coordinate transport between independently specified endpoints—highlights the need for explicit atlases (e.g., shared base checkpoints) when reproducibility is required.

Future Perspectives and Broader Impacts

The findings establish new standards for cross-checkpoint reproducibility, robust artifact transfer, and correctness in coordinate-indexed analyses. The insights extend naturally to other variants (e.g., gated FFNs for sign symmetry), and the general methodology can be adapted to future architectures with different normalization or symmetry properties.

Potential future developments include more robust parallel transport schemes for long trajectories, extensions to multi-modal or vision transformer settings, and investigation of the interplay between $S_d$8 gauge freedom and other symmetries in emergent behaviors or alignment.

Practically, gauge-correct alignment will be essential for reproducible interpretability claims, precise artifact transfer, and advanced checkpoint editing/auditing. Careful group-theoretic auditing can also identify non-reproducible claims and avoid spurious attributions.

Conclusion

The analysis presented in "Signed-Permutation Coordinate Transport for RMSNorm Transformers" establishes, with rigorous theoretical proofs and strong empirical results, that RMSNorm residual streams are coordinate-identifiable only up to signed permutations—not permutations alone. The missing sign component in prior alignment protocols causes irrecoverable transfer errors and non-reproducibility for coordinate-indexed artifacts. Comprehensive gauge audits, atlas-based transport, and sign-marginalized assignment methods provide a necessary foundation for future model analysis, merging, and interpretability work in LLMs using RMSNorm.

---

References: (2606.31963)