Curse of Depth in Deep Neural Architectures

• Curse of Depth (CoD) is a phenomenon where deep neural layers, particularly in Pre-LayerNorm models, contribute minimally to feature evolution.
• Empirical evidence from Transformers and GNNs demonstrates that many deep layers can be pruned with negligible performance loss, as measured by depth scores and relative change metrics.
• Mitigation strategies such as LayerNorm Scaling and depth-grown training effectively restore layer contributions and enhance overall model reasoning.

The Curse of Depth (CoD) denotes the phenomenon in deep neural architectures—especially Pre-LayerNorm Transformers and Graph Neural Networks—where a substantial fraction of deep layers contribute negligibly to feature evolution or task performance. Despite full architectural inclusion, these layers behave almost as identity mappings under both forward and backward passes, manifesting as both reduced representational learning and minimal response to layerwise pruning. The effect is empirically validated in contemporary LLMs such as Llama, Mistral, DeepSeek, and Qwen, as well as in extensive GNN benchmarks. Theoretical analysis traces CoD to compounding output variance in Pre-LN schemes and power-law growth of architectural energies, while remedies including LayerNorm Scaling and depth-grown training have recently demonstrated improved depth utilization, restored layer contributions, and enhanced reasoning capabilities.

1. Formal Definition and Observational Evidence

The Curse of Depth is characterized by an empirical deficit in layerwise feature evolution and output influence in deep neural networks built with Pre-Layer Normalization (Pre-LN). In LLMs, approximately half of the layers—primarily the deepest ones—can be pruned or perturbed without significant accuracy loss on benchmarks such as MMLU (Sun et al., 9 Feb 2025). In GNNs, the phenomenon is similarly identified via node-similarity energy measures $\mathcal{E}(X^k)$, where the relative change $\mathcal{R}(X^k)$ with layer depth $k$ vanishes, indicating negligible contribution from deep layers (Guan et al., 9 Dec 2025).

Depth specificity is quantified via influence metrics. For Transformers, "mean future effect" on logits is computed by skipping individual layers and measuring output deviation, yielding discrete "depth scores" centered in the initial segment of the network, far from its maximum possible value in deep architectures (Kapl et al., 9 Dec 2025). Early-exit classifiers, such as Tuned Lens, further confirm plateaued prediction accuracy within the shallow half of pre-LN networks.

2. Theoretical Mechanisms Underlying CoD

In Pre-LN Transformers, output variance recursions for activations $x_\ell$ scale as $$\sigma^2_\ell = \sigma^2_1\cdot\Theta\left( \prod_{k=1}^{\ell-1}(1 + 1/\sigma_k) \right)$$, admitting exponential growth with depth (Sun et al., 9 Feb 2025). The end-to-end Jacobian norm $\|\partial y_L/\partial x_1\|_2$ saturates, while individual block derivatives $\partial h_{\ell+1}/\partial h_\ell$ approach the identity matrix, causing deep blocks to enact no meaningful transformation.

For GNNs, Post-LN scheme yields classical over-smoothing (exponential decay of Dirichlet energy), whereas Pre-LN yields power-law growth of energy but vanishing relative per-layer updates, again resulting in minimal deep-layer impact (Guan et al., 9 Dec 2025). The relative change metric

$$\mathcal{R}(X^k) = \frac{\mathcal{E}(X^k) - \mathcal{E}(X^{k-1})}{\mathcal{E}(X^{k-1})}$$

tends to zero as $k$ grows, operationalizing CoD in message-passing networks.

3. Mitigation Strategies

The predominant architectural mitigation is LayerNorm Scaling (LNS), wherein the output of each layer normalization at depth $\ell$ is rescaled by $1/\sqrt{\ell}$, adjusting the Pre-LN block equations as follows: $$x'_\ell = x_\ell + \text{Attn}(LN(x_\ell)/\sqrt{\ell}), \qquad x_{\ell+1} = x'_\ell + \text{FFN}(LN(x'_\ell)/\sqrt{\ell})$$ This modification shifts output variance recursion to polynomial growth, $$\sigma^2_{\ell+1} = \sigma^2_\ell\cdot \Theta(1 + 1/(\sqrt{\ell}\sigma_\ell))$$, curbing the runaway variance and restoring Jacobian diversity (Sun et al., 9 Feb 2025). Table 1 from (Sun et al., 9 Feb 2025) demonstrates consistent perplexity reduction across scales:

Method	130M	250M	350M	1B
Pre-LN	26.73	21.92	19.58	17.02
DeepNorm	27.17	22.77	19.62	17.43
Mix-LN	26.07	21.39	19.54	diverged
Pre-LN + LNS	25.76	20.35	18.20	15.71

Similarly, in supervised fine-tuning, LNS yields a 1.8–2.2 point increase in average accuracy, outperforming all established normalization alternatives.

In GNNs, a nonlocal Post-LN message-passing scheme employing energy-dependent residual scaling achieves "algebraic smoothing"—energy decaying as $1/t$ rather than exponentially—eliminating both over-smoothing and CoD, with no additional learnable parameters and minimal computational overhead (Guan et al., 9 Dec 2025).

4. Depth-Growth and Residual Stream Shaping

Gradual depth expansion via middle stacking, as in MIDAS and LIDAS algorithms, overcomes CoD by encouraging distinct computational block formation within the residual stream (Kapl et al., 9 Dec 2025). Here, models begin shallow and incrementally insert blocks by duplicating parameter and optimizer states in the middle of the network. This drives cyclical feature amplification across blocks, as measured by:

• Relative-norm contribution: $\|a_i\|/\|h_i\|$
• Cosine-similarity to the stream: $\langle a_i, h_i\rangle / (\|a_i\|\|h_i\|)$

Newly grown blocks produce tight, symmetric clusters in FFN weight space and support block permutation (i.e., swapping central blocks yields minimal performance drop, unlike in static models).

Performance metrics from reasoning benchmarks show substantial gains:

Model (1.7B)	Math Word (Acc↑)	Primitives (Acc↑)
Baseline	13.75%	34.84%
LN-Scaling	11.00%	44.38%
MIDAS	16.07%	40.88%
LIDAS	18.59%	47.34%

A plausible implication is that depth-grown models, like LIDAS, achieve more robust blockwise computation and utilize entire layer stacks more effectively, avoiding deep-layer collapse even as network width and depth scale.

5. Empirical Validation and Limitations

Validation spans layer-pruning, perplexity, zero-shot/few-shot accuracy, and specialized diagnostics for LLMs and GNNs. Across LLaMA-130M, pruning deep layers under Pre-LN has negligible performance impact, while under LNS every layer mispruning results in nontrivial loss (Sun et al., 9 Feb 2025). In GNNs, the nonlocal Post-LN method maintains performance for up to 256 layers, with empirical Laplacian energy decay matching theoretical predictions (Guan et al., 9 Dec 2025).

Limitations of current analyses include exclusive focus on Pre-LN (for LLMs) and lack of unified treatment for Post-LN/hybrid architectures. Post-LN models inherently avoid exponential variance but introduce other training instabilities (Sun et al., 9 Feb 2025). Effects of LNS on implicit regularization, feature covariance, and interactions with architectural extensions (efficient attention, sparse FFNs, mixture-of-experts) remain open for research.

6. Broader Context and Related Phenomena

The Curse of Depth aligns with and diverges from other architectural pathologies such as the curse of dimensionality in scientific computation (LeFloch et al., 2016). While dimensionality-induced cost growth is polynomially mitigated via algorithms such as CoDeFi—employing Monte Carlo mesh and optimal transport localization—CoD specifically arises from the inability of deep models to leverage additional depth for representation learning, not from computational infeasibility.

Prior interventions for GNN over-smoothing, including wave equation propagation, gradient gating, and fractional diffusion, fall short of fully resolving the dual pathologies of excessive smoothing and depth collapse due to reliance on shared or fixed parameters (Guan et al., 9 Dec 2025). The parameter-free, data-dependent scaling of recent solutions marks a significant advance in depth-scalable network expressivity.

7. Future Directions

Open challenges include empirical scaling of LNS to very deep and large LLMs (multi-billion parameters and hundreds of layers), theoretical synthesis of normalization placement patterns, evaluation of interplay with task-specific regularization, and architectural integration with emerging sparsity and mixture-of-expert schemes (Sun et al., 9 Feb 2025, Kapl et al., 9 Dec 2025). In GNNs, algebraic smoothing via nonlocal message-passing offers a principled foundation for depth scalability without over-smoothing, yet generalization across diverse graph topologies and modalities warrants further study (Guan et al., 9 Dec 2025).

The mechanistic insight from depth growth protocols suggests broader applicability in neural architecture design, where gradual complexity expansion and residual block structuring may benefit not only natural language and graph modeling but also other domains sensitive to layerwise feature degradation. Expanding these protocols and normalization strategies to multi-modal and hierarchical networks represents a promising avenue for fully exploiting deep model capacity.