Curse of Depth in Neural Architectures

Updated 3 July 2026

The curse of depth is defined as a phenomenon in deep neural networks where layers beyond a threshold produce diminishing contributions and act as near-identity transformations.
Empirical analyses in Transformers, PLMs, and GNNs reveal that later layers can often be pruned without significant performance loss, indicating inefficient depth utilization.
Mitigation strategies such as sparsity-induced variance control, depth-growing methods, and LayerNorm Scaling effectively restore functional depth and enhance model efficiency.

The curse of depth describes a phenomenon in deep neural architectures—particularly Transformer-based models—where increasing depth beyond a threshold yields diminishing or vanishing per-layer contribution to representation learning or output prediction. This effect has been formalized and empirically documented in LLMs, protein LLMs (PLMs), graph neural networks (GNNs), and in the theoretical approximation properties of neural architectures. It is distinct from the curse of dimensionality but shares the theme that architectural parameters (here: depth) interact with expressivity and efficiency in nontrivial and sometimes pathological ways.

1. Formal Definition and Empirical Manifestation

The curse of depth typically refers to the observation that, in deep neural models, layers added beyond a certain depth contribute progressively less to the final model output, act as near-identity transformations, and can often be pruned without significant performance loss. In Pre-Layer Normalization (Pre-LN) Transformers, each layer ℓ computes a residual update:

$x_{\ell+1} = x_\ell + \mathcal{F}(\text{LN}(x_\ell))$

where $\mathcal{F}$ is attention or a feed-forward sublayer. Analyses show that, under typical initialization and training regimes, the variance of the residual stream grows (sub-)exponentially with depth:

$\operatorname{Var}(r_\ell) \approx \operatorname{Var}(r_0)\cdot O(1+\gamma)^{2\ell}$

For sufficiently large ℓ, the Jacobian $\left\| \frac{\partial x_L}{\partial x_1} - I \right\|_2 \rightarrow 0$ , so deep blocks behave as near-identities and do not contribute meaningful transformation (Muhtar et al., 16 Mar 2026, Sun et al., 9 Feb 2025). Empirical measures—pruning, causal interventions, permutation sensitivity, or usefulness scores—demonstrate a collapse in per-layer utility in models such as Llama, Mistral, DeepSeek, Qwen LLMs, and a wide range of PLMs (Sun et al., 9 Feb 2025, Siji et al., 25 Feb 2026).

2. Theoretical Underpinnings and Generalization

In theoretical learning settings, the curse of depth arises as strong lower bounds on the expressive efficiency of shallow networks. For example, for target functions such as the coordinate-wise product or its sine, any ReLU network with fixed or insufficient depth requires an exponential number of parameters in dimension $d$ :

$\text{Minimum size} \geq 2^{\frac{d}{L}}$

unless the network depth $L$ grows with $d$ (Gonon et al., 2023). Similarly, Safran et al. (Safran et al., 2024) demonstrate an exponential separation between depth-2 and depth-3 networks for $O(1)$ -Lipschitz functions on the unit hypercube, even with constant target accuracy and bounded weights:

Depth 2: width $\Omega(\exp(\Omega(d)))$ required for constant accuracy;
Depth 3: width $\mathcal{F}$ 0 suffices.

This distinguishes the curse of depth from the curse of dimensionality: removing the former (by increasing depth) can restore tractability for high-dimensional problems even when the class of target functions is restricted to smooth, Lipschitz, or bounded-support.

3. Manifestations in Modern Architectures

Large Language and Protein LLMs

The curse of depth characterizes a depth-inefficiency regime in both LLMs and PLMs: later layers deliver minimal functional change to representations or logits. Intervention-based diagnostics (skip-layer, future-setting, permutation, usefulness) and probing measures (KL divergence from final logits, early-exit overlap) consistently show that in deep Pre-LN Transformer stacks, only the first half to two-thirds of layers are essential. In multi-billion parameter PLMs (e.g. ESM2 15B), up to 50% of layers can be ablated without disrupting outputs (Siji et al., 25 Feb 2026). The phenomenon intensifies with increased model depth and parameter count.

Graph Neural Networks

In GNNs, a distinct but related phenomenon appears: Pre-LN GNNs experience the curse of depth (deep layers effecting vanishingly small relative feature change), while Post-LN GNNs are susceptible to over-smoothing (rapid convergence of node embeddings). In Pre-LN GNNs, the relative change in Laplacian energy per layer decays rapidly, so further stacking provides negligible added representational power, even as absolute feature variation may increase in a power-law fashion. Conversely, Post-LN design induces exponential decay (feature collapse), indicating over-smoothing but not the curse of depth (Guan et al., 9 Dec 2025).

Broader Theoretical Implications

The realization that the curse of depth is distinct from over-parameterization, vanishing/exploding gradient, or capacity mismatch is central. Denying sufficient depth can enforce strong lower bounds on function approximation, and, conversely, haphazard stacking of layers (in Pre-LN schemes or without variance control) may yield models where increases in nominal depth do not translate to effective computational depth (Gonon et al., 2023, Safran et al., 2024).

4. Mechanisms: Variance Propagation and Residual Stream

The mechanistic root of the curse of depth in Pre-LN Transformer-like architectures is exponential growth in the variance of the residual stream. The output variance at depth L satisfies:

$\mathcal{F}$1

which is observed to grow monotonically with depth, promoting identity-like Jacobians and thus layer inutility (Muhtar et al., 16 Mar 2026, Sun et al., 9 Feb 2025). Similar variance principles appear in GNNs, where over-smoothing and ineffective depth are explained by decay rates (exponential for Post-LN, algebraic or power-law for other configurations) in node-similarity or Laplacian energy.

5. Mitigation Strategies

Sparsity-Induced Variance Regulation

Both implicit and explicit sparsity can counteract the curse of depth by attenuating residual variance growth:

Implicit sparsity through weight decay or long-context training reduces parameter and attention variance, driving down per-layer residual variance (e.g., $\mathcal{F}$ 2 for sequence length $\mathcal{F}$ 3).
Explicit architectural sparsity from grouped-query attention (GQA) or mixture-of-expert (MoE) designs further regulates variance. For GQA with $\mathcal{F}$ 4 groups, per-coordinate variance is reduced by a factor $\mathcal{F}$ 5 (Muhtar et al., 16 Mar 2026).

The general variance upper bound for sparse-residual models is:

$\mathcal{F}$ 6

Controlling mask density $\mathcal{F}$ 7 through architectural or training choices directly controls variance escalation.

Depth-Growing and Structural Remedies

Gradual depth expansion protocols such as MIDAS (middle stacking) and LIDAS (layerwise insertion) inject new layers during training, cloned from central or midpoint blocks, enabling new computation pathways and preventing the formation of inert, post-processing upper layers. Such blockwise growth has been shown to raise depth-score metrics and downstream performance, forming “permutable computational blocks” that facilitate loop-like repeated computation (Kapl et al., 9 Dec 2025).

LayerNorm Scaling (LNS) and Normalization Adjustments

LayerNorm Scaling, which rescales Pre-LN output by $\mathcal{F}$ 8 at layer $\mathcal{F}$ 9, directly counters variance growth—yielding polynomially bounded, rather than exponential, variance propagation. This restores sensitivity to upstream representations throughout the depth of the stack, and experimental results consistently show that LNS-enhanced models outperform standard Pre-LN, DeepNorm, and Mix-LN configurations on both pretraining and supervised downstream metrics (Sun et al., 9 Feb 2025).

GNN Algebraic Smoothing

In deep GNNs, adopting nonlocal message passing with algebraic smoothing—modulating update magnitudes by layerwise Laplacian energy—prevents both over-smoothing and the curse of depth. This results in algebraic (rather than exponential) decay of node-similarity energies and well-conditioned gradients, supporting effective stacking up to hundreds of layers with all layers remaining functionally relevant (Guan et al., 9 Dec 2025).

6. Metrics and Experimental Evaluation

A wide array of diagnostics are used to formally assess the curse of depth:

Pruning-intervention metrics: Performance drop when ablating individual layers ( $\operatorname{Var}(r_\ell) \approx \operatorname{Var}(r_0)\cdot O(1+\gamma)^{2\ell}$ 0).
Causal/Permutation/Usefulness scores: Sensitivity of downstream state or loss to targeted interventions (e.g., layer skipping, permutation, or substitution by linearization).
Depth-Score: Quantifies the average depth at which significant causal computation is performed.
LogitLens and KL probes: Compare per-layer predicted distributions to the final output, quantifying layerwise token refinement.
Residual contribution/cosine similarity: Examines the relative norm or alignment between per-layer outputs and the state, (in)dicates diminishing update size or diversity.

Experimental interventions manipulating sparsity, normalization, or training regime routinely demonstrate that models with mitigated variance growth—via LNS, controlling sparsity, or depth-growing—redistribute functional contribution throughout the stack and achieve higher effective depth (Muhtar et al., 16 Mar 2026, Sun et al., 9 Feb 2025, Kapl et al., 9 Dec 2025).

7. Limitations, Open Questions, and Future Directions

Current mitigation frameworks have partial scope:

Excessive sparsity, aggressive over-regularization, or overshooting sequence length can “over-dampen” variance and collapse usable capacity (e.g., perplexity degradation when $\operatorname{Var}(r_\ell) \approx \operatorname{Var}(r_0)\cdot O(1+\gamma)^{2\ell}$ 1 or $\operatorname{Var}(r_\ell) \approx \operatorname{Var}(r_0)\cdot O(1+\gamma)^{2\ell}$ 2) (Muhtar et al., 16 Mar 2026).
Most analyses and interventions address only feature-wise (diagonal) sparsity; structured sparsity (block, neuron), dynamic vs. static masking, and residual vs. block-internal sparsity remain open areas.
Tight theoretical coupling between learned weights and sparsity masks is imperfectly analyzed—practical dependencies violate simplifying independence assumptions in variance propagation proofs.
The interaction of the curse of depth with alternative normalization (e.g., post-LN, RMSNorm), initialization (DeepNorm, Mix-LN), and with adaptive computation (early exit, layer skipping) is partially understood and a subject of active exploration (Siji et al., 25 Feb 2026, Guan et al., 9 Dec 2025).
For modal extensions (e.g., multimodal PLMs, structure-function joint models), heterogeneous depth utlilization profiles are observed, requiring new inductive biases and architectural variants.

Further work is warranted in adaptive, variance-aware scheduling, joint optimization of diverse sparsity patterns, extending mitigation to extreme depth ( $\operatorname{Var}(r_\ell) \approx \operatorname{Var}(r_0)\cdot O(1+\gamma)^{2\ell}$ 3), and leveraging depth growth curricula for synergy between capacity, efficiency, and functional depth utilization (Muhtar et al., 16 Mar 2026, Kapl et al., 9 Dec 2025, Siji et al., 25 Feb 2026).

Select References:

Area	Key Works	arXiv id
LLMs	The Curse of Depth in LLMs	(Sun et al., 9 Feb 2025)
	When Does Sparsity Mitigate the CoD in LLMs	(Muhtar et al., 16 Mar 2026)
PLMs	From Words to Amino Acids: Does CoD Persist?	(Siji et al., 25 Feb 2026)
GNNs	Algebraic Smoothing and Depth Scalability	(Guan et al., 9 Dec 2025)
Theory	Depth Separations in Neural Networks	(Safran et al., 2024)
	Necessity of Depth for High-dim Approx	(Gonon et al., 2023)
Depth Growth	Do Depth-Grown Models Overcome CoD?	(Kapl et al., 9 Dec 2025)

The curse of depth is now recognized as a canonical architectural inefficiency for deep neural networks, motivating a broad array of variance- and depth-aware strategies for harnessing the full representational and computational power of contemporary deep models.