Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment

Abstract: We give a descent-free, alignment-free measurement of singular structure on trained networks. At a single frozen checkpoint the read recovers the order $k$ of each dead direction from the directional-Fisher rate, the master invariant from which the per-direction learning coefficient $1/(2k)$ follows exactly, in whatever basis the optimizer left. The same read classifies each direction, separating a genuine singularity, whose order the architecture fixes, from a flat gauge symmetry; the directional-Fisher magnitude settles the cases the order cannot. A pluggable detector supplies the directions for transformer, convolutional, and normalisation layers. The read recovers the architecture-predicted order across constructed cells and trained networks, including a fine-tuned vision transformer whose dead structure is the LayerNorm-kernel gauge and a from-scratch one whose compressed MLP forms a node-death at its activation order. Where the singular structure enumerates, the per-direction orders assemble, through the typed intersection of the loci, into the global coefficient $(λ, m)$ matching the closed form. The method removes the canonical-alignment and descent preconditions of the underlying rate result, turning order-recovery into a deterministic, architecture-general reading. We then map its reach into the Watanabe triple: the order determines the universal singular fluctuation $ν(k)$, though a trained network's realized $ν$ falls below it as the live structure absorbs the dead direction's data fluctuation, and the multiplicity recovers from the dominant structure under a single-locus assumption.

• The paper presents a descent- and alignment-free framework that decomposes neural network singular structures, termed dead directions.
• It employs a two-stage detector-read pipeline that measures directional Fisher growth to recover analytic invariants like the local learning coefficient.
• Empirical validation across architectures confirms the taxonomy of singularities and gauges, linking singular learning theory to practical network complexity analysis.

Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment

Introduction

The paper "Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment" (2607.00603) presents a comprehensive descent-free and alignment-free framework for measuring and classifying singular structures—termed "dead directions"—in trained neural networks. The methodology addresses the limitations of previous approaches reliant on canonical alignment or optimizer-dependent descent directions, enabling architecture-agnostic characterization of singular structure directly at frozen checkpoints. The results unify perspectives from information geometry and singular learning theory, facilitate finer decomposition of the learning coefficient, and empirically validate the method across transformers, CNNs, LayerNorm, and other modern architectures.

Methodology: Detector-Read Pipeline and Analytic Invariants

The central primitive is a two-stage pipeline:

1. Detector: Architecture-dependent; identifies candidate dead directions as near-kernels of the layer's natural second-moment (e.g., K-FAC factors for dense and convolutional layers, closed-form kernels for LayerNorm).
2. Read: Architecture-invariant; performs a scan along the detected direction $u$ at the checkpoint $\theta_0$ via synthetic perturbations, measuring the growth rate of the directional Fisher information. The log-log slope encodes the key invariant—the KL order $k$—governing local flatness.

This read recovers, for each dead direction, the local learning coefficient $\lambda_\mathrm{dir} = 1/(2k)$, as per singular learning theory. Leveraging activation–gradient duality and an order-carrying mode construction (instead of naive Fisher spectrum extremization), the method robustly separates genuine singularities (finite $k$) from gauge directions (flat degeneracies).

Figure 1: Reading the order off canonical alignment: (a) a trained network with a dead direction $u$ rotated off the axes, read via K-FAC joint mode; (b) on a gelu transformer block, the off-canonical mode recovers $k \approx 2$, while coordinate scans are deviant.

To ensure robustness, a "purity-matched" window isolates the $t^{2(k-1)}$ regime, avoiding contamination from misalignment or lower-order mixtures. The per-direction order can then be assembled into a global learning coefficient by analytic rules (sum versus crossing minima), reconstructing the Watanabe triple $(\lambda, m, \nu)$ up to structure-dependent absorption effects.

Figure 2: The read in action, showing power-law growth of the directional Fisher in various architectural and activation settings, with the shaded window for slope estimation and clear separation from contamination floor.

Empirical Taxonomy of Singular Structure

The empirical section constructs a rigorous taxonomy, categorizing each dead direction as either a genuine singularity (finite $k$, e.g., node-death, depth singularity, convolutional channel death, unit overlap) or an architectural gauge (LayerNorm-kernel gauge, QK/VO attention rotations, ReLU rescaling, cross-entropy shift).

For each class, controlled experiments demonstrate the measured order matches analytic predictions:

• Node-Death: Recovered order equals activation analytic order ($\theta_0$0 for squared-ReLU, $\theta_0$1 for gelu/ReLU), both in constructed networks and fully-trained ViTs where dead subspaces can be significantly rotated.
• Depth Singularities: In deep linear networks, the measured order matches the network depth ($\theta_0$2).
• Gauges: LayerNorm-kernel and attention rotation directions are classified by a deep Fisher floor, not finite $\theta_0$3; only the Fisher magnitude disambiguates these curved orbits from low-order singularities.

  Figure 3: The recovered order aligns precisely with analytic structure: activation order for node-death, network depth for linear singularities, and convolutional factor adaptation for CNNs.

This per-direction classification facilitates not just measurement but diagnosis (e.g. architectural gauge versus true learning degeneracy), and enables reliable decomposition of the global model complexity into contributions from singular and symmetry structure.

Optimizer Geometry and Developmental Dynamics

An important set of results pertains to how optimizers and phase of training affect analytic decompositions:

• Orthogonalising Optimizers: Gauge-equivariant optimizers (e.g., scaled-polar orthogonaliser) produce axis-aligned, clean dead subspaces, making per-coordinate or off-canonical reads both effective.
• Standard Optimizers: Vanilla optimizers (e.g., Muon, Adam, SGD) may leave dead structure diffuse or substantially rotated, which can defeat coordinate-aligned scans but is handled by the presented pipeline. In deep transformers, this sometimes results in pre-asymptotic flatness with no finite $\theta_0$4 reachable.
• Developmental Emergence: Across grokking or compression transitions, the dead-subspace dimension and alignment increase, with the analytic order stable throughout.

  Figure 4: Orthogonaliser sets the basis—ordinal alignment and dead subspace dimensions by depth, revealing how optimizer choice critical influences the observable singular structure.

Singular Fluctuations and Data Absorption

The paper reports that along isolated dead directions, the empirical singular fluctuation $\theta_0$5 matches the theoretical universal $\theta_0$6, confirming the analytic relation between the KL order and Bayesian fluctuation scaling. However, real networks rarely isolate dead directions; partial overlap with live (non-singular) structure leads to systematic absorption of $\theta_0$7, decreasing its realized value compared to the theoretical maximum.

Figure 5: The order fixes $\theta_0$8 for isolated directions, while in the presence of live structure, significant absorption of the expected singular fluctuation is observed.

Decomposition, Global Assembly, and Cost

Unlike SGLD-based posterior samplers, which only yield a global scalar (with stochastic calibration costs), the per-direction geometry-based decomposition is deterministic, cheap relative to sampling, and readily interpretable. The method is validated using closed-form RLCTs on analytic models (Aoyagi, Watanabe) and provides explicit rules for assembling per-direction quantities into the model-level learning coefficient and multiplicity.

Figure 6: One-pass census and per-direction decomposition can be used to separate optimizer families and recover structure that aligns with, but is more fine-grained than, SGLD-LLC complexity estimates.

Diagnostic, Interpretability, and Limitations

The framework enables construction of per-model signatures, continuous deadness-depth spectra, and diagnoses for whether observed directions are true singularities, contaminated, or mere architectural gauges. These profiles have immediate uses in training diagnostics (e.g. monitoring singularity phase boundaries, optimizer design feedback, and liveness in modular or dynamic architectures).

However, the method's coverage is bounded by the detector family: structures unexposed by any detector are unmeasured, and global assemblies only lower bound the complexity if untyped singular structure remains. In determinantal singularities (e.g., wide deep linear nets), exact assembling of the learning coefficient from per-direction reads requires further structural resolution, currently beyond the simple pipeline.

Figure 7: Dead subspace dimension and alignment evolve over training trajectory, reflecting the developmental emergence of compressed solution manifolds.

Conclusion

This work establishes a rigorous, architecture-agnostic, and optimizer-robust framework for decomposing, classifying, and measuring singular degeneracies in modern neural networks. By directly associating observable local Fisher decay rates with analytic invariants from singular learning theory, the method yields precise per-direction learning coefficients and a taxonomy of degeneracy realized in deep models. The framework supports fine-grained model characterization, exposes the interplay between optimizer-induced geometry and network compression, and enables scalable, interpretable Bayesian complexity analysis without reliance on canonical alignment or stochastic posterior sampling.

Future work should address extension of the detector family, improved structural assembly in determinantal varieties, and potential application of the method to dynamic or sparsity-driven networks and ensembles. The fine-grained decomposition may inform both practical network design and deeper theoretical understanding of generalization and redundancy in high-dimensional learning systems.