Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity

Abstract: Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities. The local learning coefficient (LLC), the standard estimator of Watanabe's real log canonical threshold (RLCT, $λ$), reads this geometry as an integrated Bayesian scalar through SGLD, which needs per-task calibration and $10^4$-$10^6$ forward-backward passes per checkpoint. We introduce Dead-Direction Signatures (DDS), a family of cheap closed-form spectral readings of singular structure: each reads a network's activation matrix or per-sample-gradient Fisher-Gram at a chosen layer, replacing the SGLD posterior chain with spectral linear algebra. The readings rest on a dead-direction framework that predicts a structural correlation between activation- and Fisher-side spectra at any singular minimum, and a rank-multiplicative volume identity that single-eigenvalue monitors cannot produce: the active-volume $\log\det^{+}(G)$ slope counts the dead directions, tracking the rank-deficit $r$ across $r \in {1,2,3,4}$ (slope ratios $2.0, 3.1, 4.0$ at $r{=}2,3,4$ against the predicted $2,3,4$), where the smallest eigenvalue is rank-blind. On reduced-rank regression with closed-form $λ$, calibrated LLC recovers $λ$ at $99\%$ mean and the DDS observables rank-track it at the framework-predicted sign; on a non-linear modular-addition transformer DDS separates $d_{\mathrm{model}}$ across eighteen orders of magnitude where calibrated LLC at the protocol budget is rank-flat. Complementary to LLC's integrated posterior reading, DDS gives a directional, layer-local handle on a network's dead directions, read in closed form from its activation and gradient spectra.

• The paper presents DDS, a closed-form method that detects and quantifies dead directions in deep neural networks via spectral analysis of activation and Fisher gradients.
• It demonstrates high-fidelity RLCT tracking with empirical validation showing strong alignment with theoretical predictions and a dramatic reduction in computational cost.
• The layer-local diagnostic approach enables real-time complexity monitoring and is effectively extended to modern transformer architectures, aiding model selection and bottleneck analysis.

Dead-Direction Signatures: Efficient Spectral Diagnostics of Singular Complexity

Introduction and Motivation

Understanding the intrinsic complexity of deep neural networks, particularly in overparameterized regimes, remains a central technical challenge. Singular learning theory, with the Real Log Canonical Threshold (RLCT) as its primary analytic, provides a powerful framework for measuring model complexity via the geometry of loss singularities. The dominant practical estimator for RLCT—the Local Learning Coefficient (LLC)—relies on computationally intensive Bayesian posterior sampling using SGLD, making its use prohibitive at scale and often requiring per-task calibration. The work introduces Dead-Direction Signatures (DDS), a suite of per-layer, closed-form spectral diagnostics that efficiently detect, count, and localize singular degeneracies using activation and Fisher gradient spectra. DDS leverages recent theoretical advances describing the local Fisher geometry at singular minima to provide an alternative to SGLD-based LLC, dramatically lowering compute requirements while maintaining layer-local sensitivity to structural complexity.

Theoretical Framework

At the core of the DDS approach is a characterization of dead directions—a subspace along which the Fisher information decays polynomially due to loss landscape degeneracies. Each dead direction is associated with a KL order $k$, such that along the path $\theta_0 + t u$, the KL divergence behaves as $c t^{2k} + O(t^{2k+1})$, encoding flatness transverse to the minimum. The DDS observables are built around the following key structural results:

• Directional Fisher Decay: Along a dead direction of KL order $k$, the smallest Fisher eigenvalue decays as $O(t^{2(k-1)})$—a direct spectral signature of the local learning geometry (2606.21158).
• Selection Rule and RLCT Recovery: This decay rate yields the RLCT's local contribution, $1/(2k)$, without explicit resolution of singularities, mapping observable rates directly to complexity invariants.
• Layerwise KFAC Bridge and A-G Duality: The lowest non-zero eigenvalues of activation and Fisher-Gram matrices at each layer are co-monotonic, with explicit rate relationships derived for both.
• Rank-Multiplicative Volume Identity: The log-volume of the active Fisher spectrum (via $\log\det_{+}(\mathrm{G})$) precisely counts the number of dead directions and scales linearly with the rank-deficit $r$; in contrast, single-rank monitors (smallest eigenvalue) are provably invariant to multiple degeneracies.

The DDS observables—smallest activation singular value $\sigma_\text{min}(X)$, smallest positive Fisher-Gram eigenvalue $\lambda_\text{min}(G)$, and the active spectrum log-volume $\theta_0 + t u$0—can be efficiently computed via forward and/or backward passes followed by linear algebra, bypassing any need for computational posterior sampling chains.

Empirical Validation and Diagnostic Strength

Detection and Counting of Dead Directions

Using parametric singular testbeds with closed-form RLCT (notably Aoyagi's reduced-rank regression), the collapse of $\theta_0 + t u$1 signals the onset of a dead direction and precisely localizes it to a network bottleneck. The volume observable $\theta_0 + t u$2 scales linearly in slope with the rank-deficit $\theta_0 + t u$3, empirically tracking theoretical predictions with high fidelity (ratios $\theta_0 + t u$4 for $\theta_0 + t u$5). Single-eigenvalue monitors (e.g., minimal Fisher eigenvalue) are invariant to $\theta_0 + t u$6 by construction and cannot resolve higher-order degeneracy structure.

Closed-form RLCT Tracking and Discriminative Capacity

On canonical closed-form RLCT grids, all DDS observables (including $\theta_0 + t u$7 and $\theta_0 + t u$8) rank-track complexity (Spearman $\theta_0 + t u$9 between $c t^{2k} + O(t^{2k+1})$0 and $c t^{2k} + O(t^{2k+1})$1) with the correct sign, matching the analytical RLCT almost exactly. However, this alignment, while necessary, is non-discriminative: even naive capacity proxies and calibrated LLC clear the same bar. The discriminative power of DDS is isolated to its ability to detect and count dead directions through the rank-multiplicative volume identity.

Transformer Extension and Robustness

On non-linear architectures, particularly width-varying modular-addition transformers, DDS observables robustly separate model widths across 5–15 orders of magnitude and retain sign coherence ($c t^{2k} + O(t^{2k+1})$2), whereas LLC estimation is flat across width, even under a $c t^{2k} + O(t^{2k+1})$3 increased SGLD budget. The structural correlation between activation and Fisher-side spectra persists (Spearman $c t^{2k} + O(t^{2k+1})$4 across several testbeds), confirming that the layer-local geometric signals detected by DDS generalize beyond strictly analytic settings.

Cost Efficiency

The DDS readouts are orders of magnitude more efficient than LLC: the activation singular value and its associated probe direction require only a single forward (and optionally backward) pass with SVD, while full Fisher-spectrum quantities scale cubically in layer width but remain far below SGLD calibration cost on practical models. This cost advantage brings real-time and periodic monitoring to checkpoints and architectures where Bayesian posterior sampling is infeasible.

Implications and Limitations

The DDS framework provides a practically tractable, directionally resolved, layer-local alternative to SGLD-based LLC for diagnosing singular complexity and parameter manifold degeneracies. Its empirical validation on parametric closed-form models and extension to modern transformer architectures position it as a diagnostic for model selection, bottleneck analysis, and dynamic complexity ranking in regimes where LLC estimation is computationally prohibitive.

However, the scope of DDS is limited to static geometric diagnosis: it does not yield posterior-invariant summaries (e.g., WAIC, singular fluctuation) and cannot detect "developmental plateaus" as identified by LLC. DDS structurally relies on the presence (and approach during training) of singular minima corresponding to dead directions—on fully regularized models or those without structural degeneracy, the signals are null. Moreover, trajectory-based rate analyses require canonical SGD-compatible training; non-G-invariant optimizers (e.g., Adam with CE loss) generally violate the required alignment for rate fitting, but static structure signals are robust.

Future Directions

This work opens avenues for several practical applications and further theoretical expansion. DDS provides the groundwork for low-cost complexity ranking, monitoring of rank-collapse during pretraining, and targeted intervention strategies (e.g., placing low-rank adapters along diagnosed dead directions). Theoretical extensions toward the remaining components of the RLCT triple (multiplicity, fluctuation) and portable diagnostic criteria for large-scale architectures are promising. Broadening coverage to deeper transformers, non-canonical trajectories, and diverse algorithmic tasks is required to fully characterize generalization.

Conclusion

Dead-Direction Signatures (DDS) constitute an efficient, closed-form, layer-local method for diagnosing the singular geometric complexity of deep neural networks. By leveraging a precise spectral characterization of dead directions, DDS delivers substantially lower computational cost compared to classical Bayesian RLCT estimation, while providing rich per-layer diagnostic capability. The empirical evidence demonstrates robust alignment with theoretical predictions on parametric models and reveals practical diagnostic value for complex architectures. Complementary to LLC, DDS expands the practical toolkit for contemporary complexity and degeneracy analysis in deep learning.