Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale

Abstract: Pretrained transformers sit near singular minima of the loss, where the Fisher information metric degenerates along dead directions: directions in parameter space along which the directional Fisher vanishes. Locating such a direction normally needs a forward pass and an eigendecomposition of activations, or a sampling-based complexity estimate; none returns a direction computable from the network's parameters alone. We give one, for LayerNorm transformers. The inverse-scale direction $γ^{-1}/|γ^{-1}|$ of the LayerNorm affine is an exact algebraic kernel of the post-final-norm centred activation covariance, for any input distribution, and induces a corresponding dead direction in parameter space. It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve: the cheapest dead-direction read, specific to LayerNorm. We test it on $14$ pretrained transformers ($9$ LayerNorm, $5$ RMSNorm; $160$M-$35$B; language and vision objectives). At random initialisation the predicted direction matches the measured bottom singular direction (one forward pass, direct SVD) to four decimal places on $9/9$ LayerNorm models, and is correctly absent on $5/5$ RMSNorm models, which lack the mean-subtraction projector that creates it. On the trained checkpoint the covariance eigenvalue along this direction deepens by ${\sim}10^3\times$ and further dead directions open; the random-init-to-trained gap is a one-forward-pass, per-checkpoint readout of singular structure along the predicted coordinate. Two consequences follow in closed form: the residual stream's smallest singular value is preserved block-to-block on $13/14$ transformers measured on their own input distribution, the one exception (Gemma$4$-$31$B) a genuine dead direction the same read pinpoints; and the kernel direction's presence classifies a transformer's normalisation from the parameters alone.

• The paper identifies an algebraic dead direction in LN transformers using a closed-form parameter derived solely from the LN scale, achieving near-perfect alignment in experiments.
• Empirical results over models from 160M to 35B parameters confirm that LN-equipped transformers exhibit a predictable singularity gap, unlike their RMSNorm counterparts.
• The work offers actionable insights for model probing and architecture design by leveraging forward-pass diagnostics to assess the post-LN activation covariance structure.

Algebraic Dead Directions in LayerNorm Transformers: Structure, Diagnostics, and Implications

Overview

This work, "Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale" (2606.19491), rigorously analyzes the emergence and structure of degenerate directions (“dead directions”) in the parameter space of large-scale transformers utilizing LayerNorm (LN). The authors provide a novel, architectural mechanism for identifying these dead directions—specifically, a closed-form direction in parameter space which reflects the exact kernel direction of the post-LayerNorm activation covariance. This diagnostic is remarkable in that it is computable from the LN affine parameter alone, without requiring data, gradients, or even a forward pass, achieving four-decimal precision on real LLM architectures. Comprehensive empirical validation is performed on a diverse set of models (spanning $160$M to $35$B parameters, with language, vision, and multimodal objectives).

Geometric and Algebraic Foundations

Trained overparameterized transformers are characterized by loss landscapes with degenerate minima, where the Fisher information metric loses rank along certain directions—these are the "dead directions" as formalized by singular learning theory [Watanabe09]. Locating such directions in practice has required computationally intensive activation spectral analysis or gradient-based estimators, which are impractical at LLM scale. The present work applies recent analytic advances in the theory of geometric singular learning to LayerNorm-equipped transformers, yielding for the first time a direction in original parameter coordinates (namely, the normalized reciprocal of the LayerNorm scale parameter $\gamma^{-1} / \|\gamma^{-1}\|$) that exactly identifies a structural kernel of the post-LN activation covariance.

Figure 1: Illustration of (a) dead directions in Fisher geometry, (b) measurement of the algebraic direction from $\gamma^{-1}$, and (c) dependence of Schur-ratio gap on training.

The algebraic mechanism is as follows. Consider LN with parameter $\gamma$. The mean-subtraction in LN ensures that the post-LN centered covariance is singular. The paper proves, for any input distribution, the kernel of this covariance lies exactly along $\gamma^{-1}$, a property unique to LN (and absent in RMSNorm, which lacks the mean-subtraction projector):

• LN algebraic kernel direction: $v^* = \gamma^{-1} / \|\gamma^{-1}\|$ (Prop. 63).
• Distinct RMSNorm behaviour: No universal kernel direction exists (Prop. 63 negative).

This structural result partitions Transformer architectures: only LN-equipped models possess a predictable algebraic dead direction, a conjecture empirically substantiated in the paper.

Empirical Validation at LLM Scale

The authors conduct detailed experiments over $14$ pretrained transformer checkpoints (language and vision; $9$ LN, $5$ RMSNorm). The principal findings can be summarized:

• LN models: At random initialization, the predicted algebraic direction aligns with the measured bottom singular direction of the post-final-norm centered covariance to $35$0 ($35$1 average). After training, this remains, but additional dead directions arise, as seen by a significant increase (about $35$2) in the "Schur-ratio” gap (the ratio of the minimum eigenvalue to the eigenvalue in the predicted direction) along the new data-driven dead directions.
• RMSNorm models: No such alignment. $35$3 has negligible projection onto the kernel direction at initialization or after training. This dichotomy aligns perfectly with the theoretical prediction.

  Figure 2: Distribution of $35$4 alignment across various LayerNorm and RMSNorm transformers, showing precise agreement with theory.

Further, through the measurement of the minimal singular value of the residual stream across layers ($35$5), the authors validate a robust depth-invariance corollary: under exact-identity residual skips, this ratio is always $35$6 unless catastrophic cancellation occurs, as seen in a unique pathology in Gemma-4-31B (a model exhibiting an intrinsic dead direction, serving as an exception that proves the rule).

Figure 3: Depth profile of $35$7 through the residual stream, confirming invariance except for identified model pathologies.

Results are consistent across model families, tasks, and training protocols.

Quantitative Diagnostics and Layer Structure

A critical quantitative tool in the paper is the "Schur-ratio" diagnostic:

• $35$8 where $35$9 is the per-layer input covariance, and $\gamma^{-1} / \|\gamma^{-1}\|$0.

At random initialization for LN models, $\gamma^{-1} / \|\gamma^{-1}\|$1 (with deviation solely from finite-sample effects), as predicted by the finite-$\gamma^{-1} / \|\gamma^{-1}\|$2 Marchenko–Pastur drift. At convergence, $\gamma^{-1} / \|\gamma^{-1}\|$3 and the gap, $\gamma^{-1} / \|\gamma^{-1}\|$4, quantifies the depth of learned singular structure inaccessible at initialization. The claim is tightly validated at scale, with RMSNorm models presenting high $\gamma^{-1} / \|\gamma^{-1}\|$5 even at initialization, in precise contrast to LN.

Figure 4: Empirical sweep of the Schur-ratio diagnostic shows architectural dead directions for LN, not RMSNorm, and substantial training-induced deepening.

Moreover, the per-layer rate structure predicted by singular learning theory (that shallow layers intrinsically accumulate more dead directions) is empirically validated: the predicted allocation for projection-compression preserves perplexity best at LLM scale.

Figure 5: Validation of per-layer rate structure; theorem allocation outperforms alternatives in terms of perplexity.

Theoretical and Practical Implications

The findings have several deep theoretical and applied implications:

• Architectural dichotomy: LayerNorm imposes an immutable, computable dead direction at each block, predictable for any pretrained checkpoint, informing precise understanding of the model’s singular geometry. RMSNorm does not.
• Trainability and capacity: LN's structural predictability trades off some residual-stream "capacity" for analytic anchoring of degenerate directions, while RMSNorm preserves all $\gamma^{-1} / \|\gamma^{-1}\|$6 directions but with less analytic structure.
• Forward-pass-only diagnostics: The central dead direction and the full singular-structure growth during training can be diagnosed via simple forward computation, making them uniquely accessible for model vetting, anomaly detection, and protocol validation even at LLM scale.
• Design and compression: In injection or pruning schemes (e.g., LoRA variants, affine folding for model compression, or selection of invariant subspaces for continual learning), knowledge of the architectural dead direction can be used to avoid wasting rank or capacity.

Limitations and Open Problems

Certain aspects remain open:

• The architectural coverage is limited to LN and RMSNorm; extension to BatchNorm, GroupNorm, and more exotic normalization schemes remains empirical.
• The source of the Gemma4-31B intrinsic dead direction is unresolved, highlighting the need for further study into the interaction of normalisation, residual MLPs, and training protocols.
• The measurement precision at the bottom of the spectrum is bounded by fp$\gamma^{-1} / \|\gamma^{-1}\|$7/fp$\gamma^{-1} / \|\gamma^{-1}\|$8 implementation details, though the authors address this thorougly.

Conclusion

This paper establishes a rigorous, architecture-dependent analytic diagnostic for dead directions in LayerNorm transformers. The existence, orientation, and singularity depth of kernel directions are read directly from LayerNorm affine parameters, with strong empirical confirmation and clean architectural prediction/falsification criteria. The work provides a practical forward-pass-only tool for measuring singular structure in LLMs and validates quantifiable differences between LN and RMSNorm. The analytic and empirical results yield actionable insights for model probing, architecture selection, and capacity control in modern deep networks.