Dead Directions: Geometric Singular Learning

Abstract: Singular learning theory and information geometry have studied the same parameter spaces in mostly separate vocabularies: the former computes Bayesian invariants in resolved coordinates, the latter works in original coordinates under a non-degeneracy assumption that overparameterised models routinely violate. We bridge them through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a tangent to the analytic singular set with a definite KL order, set by how fast the KL divergence vanishes. The two readings name the same vector; our central move shows its KL order is recoverable as the decay rate of the directional Fisher curvature approaching the singularity, in original parameter coordinates and without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and we extend the recovery to multi-component crossings, multiplicity $m$, the singular fluctuation $ν$ (universal in the KL order for 1D directions), prior-RLCT shifts, and tempered posteriors. We then lift this rate to a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at modern-network primitives (residual streams, layer normalisation, attention). A quotient theorem carries the rate to the gauge quotient $Θ/G$ under gradient flow on a $G$-invariant metric; SGD qualifies, standard Adam does not, and we construct a $G$-equivariant Adam-family preconditioner (DDCAdam) that does. The bridge yields a parameter-coordinate handle on singular geometry, closed-form per-architecture predictions, and a trajectory-rate readout of Watanabe's triple $(λ, m, ν)$ from one checkpoint's forward and backward passes, without posterior sampling.

• The paper introduces 'dead directions' as a unifying primitive linking Fisher metric degeneration with algebraic invariants in singular models.
• It leverages the asymptotic decay of directional Fisher information to deterministically recover key invariants such as the real log canonical threshold (RLCT).
• The framework offers practical insights for architecture design and optimizer adjustments to maintain geometric invariants in deep learning.

Dead Directions: Bridging Singular Learning Theory and Information Geometry

Overview and Motivation

"Dead Directions: Geometric Singular Learning" (2606.05957) systematically bridges two major analytic frameworks for parameter geometry in statistical learning: information geometry, which studies regular (non-singular) parameter regimes via the Fisher metric, and singular learning theory (SLT), which quantifies the complexity of highly overparameterized, non-identifiable models via algebraic invariants such as the real log canonical threshold (RLCT). The central innovation is the formalization of the dead direction—parameter-space directions along which the Fisher information degenerates and the Kullback–Leibler divergence (KL) vanishes to higher order—offering a unifying primitive that is simultaneously interpretable in both frameworks.

This primitive enables a geometric, coordinate-level characterization of degeneracies (singularities) in high-dimensional models, such as neural networks, without the explicit need for intractable algebraic resolutions. Using asymptotic decay rates of directional Fisher curvature, the KL order, and their implications for Bayesian complexity, the paper provides architectural predictions, optimizer-aware readouts, and numerical evidences directly in original coordinates.

Figure 1: The dead direction bridges the two traditions: the same unit vector is Amari's kernel-approaching direction of the Fisher metric $F$ and Watanabe's tangent to the singular set $\Sigma_T$, with KL order $k$ the shared invariant. The trajectory Fisher-rate exponent $\alpha = 2(k-1)$ recovers Watanabe's local RLCT $\lambda = 1/(2k)$ without a Hironaka resolution.

The Dead Direction as a Bridging Primitive

Formal Definition

A dead direction at a parameterization $\theta_0$ is a unit vector $u$ such that, along a path $\theta(t) = \theta_0 + t u$ approaching a solution set $\Sigma_T$, the Fisher information quadratic form $u^\top F(\theta(t)) u$ degenerates, vanishing as $\Sigma_T$0. The corresponding KL divergence satisfies $\Sigma_T$1 for some $\Sigma_T$2.

This vector $\Sigma_T$3 is:

• Amari's kernel-approaching vector: the direction along which the Fisher metric loses rank.
• Watanabe's singular-fiber tangent: the direction tangent to the analytic singular set in SLT, with KL order $\Sigma_T$4.

Reading Out the 'KL Order' via Fisher Decay

The directional Fisher information decays asymptotically as $\Sigma_T$5.

• Slope: The logarithmic slope, $\Sigma_T$6, allows for empirical estimation of $\Sigma_T$7 from forward and backward passes alone, circumventing the need for coordinate blowups or posterior sampling typical in SLT.
• RLCT Recovery: The local RLCT $\Sigma_T$8, which characterizes the leading correction to the Bayesian free energy, is recovered directly from the observed trajectory slope, thus tying the geometric and algebraic invariants together.

  Figure 2: Three views of the dead-directions framework: (a) rate primitive (Fisher decay), (b) K-FAC ladder (layerwise exponent factorization), (c) gauge quotient (loss symmetries under optimizer dynamics).

Fisher--Curvature--Volume Rate Chain

A profound consequence is the rate chain connecting three measurable geometric quantities to the KL order $\Sigma_T$9:

• Fisher’s minimal eigenvalue: Decays as $k$0.
• Fisher-Riemannian sectional curvature: Diverges as $k$1.
• Volume of regions of large curvature: Scales as $k$2 for curvature threshold $k$3.

  Figure 3: The Fisher--curvature--volume rate chain: three measurable faces of the single KL invariant $k$4.

Multi-layer and Architectural Extensions

Layer-Wise Rate Propagation: The K-FAC Bridge

By leveraging canonical-aligned K-FAC blocks, the primitive is extended to deep, layered architectures:

• The dead-direction rate ladder: For an $k$5-layer network, $k$6 for layerwise Fisher gradient factors, independent of non-singular block widths and activation details.
• The A-G duality: The product of the smallest eigenvalues of the activation and gradient covariances at layer $k$7 remains depth-invariant: $k$8.

Composition and Additivity

For heterogeneous block compositions (e.g., MLPs, residuals, attention), the per-block KL order composes additively along canonical-aligned paths under a scalar-transfer condition. Exceptions, such as non-elementwise nonlinearities (softmax in attention), introduce subtle context-dependent phenomena, including saturation effects and breakdown of naive additivity in deep attention chains.

Residual Connections and Depth-Invariance

In residual architectures, identity skips act as zero-length edges in the computational graph, resulting in invariance of the smallest activation singular value ($k$9) along the residual stream, independent of depth. This immediately implies that the geometric degeneracy is preserved under skip connections, contradicting the doubly-exponential decay observed in non-residual architectures.

Figure 4: Architectural freeze-probe roundup for the per-primitive lemmas of this section and the architectural catalogue. Primitives that preserve or shift the rate are annotated against predicted behaviors.

Normalization Primitives

For LayerNorm, the standardized output covariance always possesses a deterministic kernel direction given by $\alpha = 2(k-1)$0, determined purely by the affine scaling parameter. In contrast, RMSNorm lacks such a universal kernel direction, a distinction empirically validated across diverse transformer families.

Figure 5: LN kernel-direction alignment across 14 pretrained transformers. The direction $\alpha = 2(k-1)$1 is universal for LayerNorm, absent for RMSNorm.

Attention Mechanisms

Isolated (single-head) self-attention blocks have a forward and backward block rate $\alpha = 2(k-1)$2, with subleading corrections due to softmax. In pure attention chains, cross-block softmax Jacobian couplings introduce a saturation depth beyond which additional compositional rate does not accrue, a phenomenon quantitatively resolved for per-component rates in the paper.

Figure 6: Refined attention-chain composition rates for pure attention stacks, showing empirical validation of the predicted saturation and component-wise anomalous scaling.

Quotient Geometry and Optimizer Implications

Gauge Symmetries

The Fisher rate primitive descends to gauge-invariant quotients associated with model symmetries (e.g., weight rescaling, logit shifts). The intrinsic KL order and associated rate are well-defined (and observable) on quotient spaces.

Optimizer Equivariance

• SGD: On any $\alpha = 2(k-1)$3-invariant metric, gradient flow correctly realizes the quotient rate.
• Adam: The standard per-coordinate preconditioner is not $\alpha = 2(k-1)$4-equivariant, and the trajectory may drift along symmetry orbits, thereby disrupting direct empirical correspondence to RLCT-based predictions.
• DDCAdam: The authors construct a class of dead-direction-aware Adam-family optimizers (Dead Direction Conditioner; DDC), which are $\alpha = 2(k-1)$5-equivariant by design and hence preserve the bridge predictions along optimizer trajectories, closing the operational gap left by Adam.

Numerical and Empirical Validation

Extensive empirical studies are conducted both in controlled analytic models (Gaussian mixtures, reduced-rank regression) and modern deep neural architectures, validating the theoretical rates at high precision, across varying activation classes, singularity types, architectural primitives, and optimizer choices.

Figure 7: Rate validation with extended $\alpha = 2(k-1)$6 range shows the breakdown predicted by the theorem's asymptotic character, including tight slope matches in the predicted regime.

Figure 8: Per-seed view of the slope fits demonstrates robustness across initializations.

Implications and Future Directions

Practical Implications

• Empirical RLCT Readout: The trajectory rate readout provides a deterministic, sampling-free alternative to SGLD-based posterior estimation for Bayesian complexity.
• Architectural Diagnostics: Depth-invariance of $\alpha = 2(k-1)$7 serves as a structural health diagnostic for residual networks.
• Optimizer Design: DDCAdam generalizes preconditioners to enforce symmetry compliance, enhancing interpretability of optimizer dynamics in singular regimes.

Theoretical Implications

• Correspondence Principle: The paper formalizes a direct geometric–algebraic correspondence: observable Fisher curvature decay captures algebraic invariants (RLCT, multiplicity, singular fluctuation) in natural coordinates.
• Algebraic Geometry without Resolution: The approach obviates explicit Hironaka resolution, leveraging geometric decay directly for SLT quantities.
• Novel Universality: The observed universality of singular fluctuations ($\alpha = 2(k-1)$8) along 1D dead directions, as a function of KL order, is established theoretically and validated numerically.

Outstanding Problems and Future Work

• Multi-direction Couplings: Extension of trajectory-rate characterizations to multi-degenerate singularities with strong cross-direction couplings and unresolved normal crossing forms remains open.
• Non-i.i.d. Data and Realistic Loss Structures: Adapting the framework to non-i.i.d. setups and broader classes of loss functions is left for future work.
• Closed-Form Correction for Adam: Developing analytical correction terms for standard Adam-family optimizers in alignment-rotated manifolds is explicitly posed as an open technical challenge.

Conclusion

This work establishes dead directions as a geometric and operational bridge between information geometry and singular learning theory, enabling direct empirical and theoretical access to SLT invariants from forward and backward passes in deep models. Its general framework systematically connects local geometric degeneracy, Bayesian statistical complexity, and architectural dynamics in modern overparameterized networks with practical protocols that are scalable and theoretically principled. The interplay between geometric singularity, model complexity, and optimizer dynamics articulated herein sets an advanced foundation for both theoretical investigation and empirical methodology in modern statistical learning.