Static Depth Pruning

• Static depth pruning is a structured compression method that permanently removes entire layers or blocks based on one-time calibration data.
• It uses criteria like cosine similarity, perplexity-based scoring, and Taylor expansions to rank and prune less essential components.
• Applied in transformers, CNNs, and retrieval systems, it improves runtime efficiency while maintaining acceptable performance levels.

Static depth pruning is a structured compression paradigm in which computational depth is reduced by a fixed, offline decision that is reused for all future inputs. In transformer and convolutional networks, it usually means permanently removing entire layers, blocks, or sublayers from a pretrained model; in tree ensembles, it means truncating individual trees by deleting lower depth layers; and in retrieval systems, closely related static pruning schemes reduce index or embedding complexity offline rather than adapting structure per query. The common invariant is that the pruned structure is determined once from calibration, corpus, or training-distribution data, and is not recomputed per prompt, token, or query (Ding et al., 26 Jan 2026, Siddiqui et al., 2024, Liu et al., 2022, Siciliano et al., 2024).

1. Definition and formal scope

In recent LLM work, static depth pruning is formalized as selecting a fixed subset of layers

$$\mathcal{L}' \subseteq \{1,\dots,L\}, \qquad |\mathcal{L}'|=L'<L,$$

so that only those layers remain active: $$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$ This is the “structured pruning” depth axis in a three-way comparison against static width pruning and dynamic depth pruning. Static depth pruning deletes entire layers; static width pruning shrinks hidden dimensions or prunes neurons or heads; dynamic depth pruning decides at inference time whether to execute a block for each input or token (Ding et al., 26 Jan 2026).

A closely related formulation represents pruning as an omission set. If a pretrained model has blocks

$\mathbf{W} = (W_1,\ldots,W_d),$

then an omission set

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$

defines the pruned model $\mathbf{W}^{\setminus \mathbf{b}}$. In the static case, the same omission set is used for every input, and the optimization objective is

$$\min_{\mathbf{b}:|\mathbf{b}| \ge k} L(\mathbf{W}^{\setminus \mathbf{b}}),$$

with $k$ determined by latency or memory constraints (Wee et al., 4 Feb 2025).

The defining contrast is input independence. Static depth pruning uses one fixed architecture for all inputs, whereas dynamic schemes learn or select different masks at inference time. In block-wise LLM pruning this contrast is often expressed with a binary mask

$z \in \{0,1\}^B,$

applied uniformly in the static case and adaptively in the dynamic case (Qiao et al., 4 Nov 2025).

Outside neural architecture compression, the term “static pruning” is used in an analogous sense for retrieval systems. In dense retrieval, PCA is computed once on document embeddings, a single offline projection is applied to the entire index, and queries are only projected into the same reduced space at inference time; in sparse neural retrieval, document-centric pruning retains only the top-$k$ term impacts per document before query processing (Siciliano et al., 2024, Lassance et al., 2023). This suggests that the notion of “static” is broader than literal layer deletion: it denotes fixed, offline structural reduction.

2. Importance estimation and pruning criteria

Most static depth pruning pipelines are ranking procedures: estimate importance once, sort layers or blocks, and remove the least important ones. In the controlled LLM study of instruction-following and reasoning-augmented models, three canonical scoring criteria are emphasized. Block Influence (BI) / ShortGPT scores a layer by the similarity between its input and output representations, treating high similarity as redundancy. Perplexity-based scoring (PPL) / Shortened-PPL measures how validation perplexity changes if a layer is removed. Taylor-based scoring / Shortened-Taylor estimates sensitivity of the loss to removing a layer via a gradient–weight criterion. The same study notes that many prior pipelines keep the first four and last two layers intact because one-shot scoring can underestimate early and late layers, and it identifies SLEB as an iterative variant that recomputes importance after each pruning step (Ding et al., 26 Jan 2026).

A deeper analysis of LLM depth pruning broadens this taxonomy. It compares static cosine block influence, relative $L_1/L_2$ norm metrics, and an adaptive Shapley-style metric

$$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$0

Cosine and relative norm metrics are local static proxies based on activation changes; Shapley is task- or loss-aware. The reported outcome is a cross-task trade-off: Shapley can improve the objective it is optimized for, but may sharply hurt transfer to other tasks because the induced block influences differ across objectives (Siddiqui et al., 2024).

SimDiff proposes a dual static criterion motivated by the residual form

$$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$1

It combines representational similarity with transformation difference. The similarity term is based on cosine dissimilarity between $$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$2 and $$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$3; the difference term uses either MSSD,

$$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$4

or MASD,

$$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$5

and fuses them as

$$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$6

The motivation is that cosine similarity alone captures direction but not magnitude, which can produce unstable pruning across architectures (Chen et al., 21 Apr 2026).

Another static line replaces hand-designed criteria with policy search. In the static mode of PPF, the sampling window collapses to a fixed target ratio,

$$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$7

and a DDPG agent chooses both an importance scoring method and a scaling factor to generate layer-wise structured pruning ratios. The framework therefore treats static pruning as optimization over a fixed layer-wise policy under a single target budget, rather than a one-shot rule (Ma et al., 4 Aug 2025).

3. Architectural granularity and compensation mechanisms

Although the canonical form removes whole transformer blocks, static depth pruning is not restricted to that granularity. One study explicitly evaluates pruning at three levels: complete blocks, self-attention layers only, and feed-forward layers only. Its empirical conclusion is that self-attention layers are more amenable to pruning than feed-forward layers, with Mistral 7B tolerating removal of up to 33% of self-attention layers without any performance degradation on MMLU; the deployment significance is that self-attention pruning also reduces costly KV-cache maintenance during autoregressive decoding (Siddiqui et al., 2024).

ReplaceMe modifies the usual “delete layers and heal later” pattern. It statically removes a contiguous span of transformer blocks and replaces the removed sequence with a learned linear transformation. For a kept block $$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$8 followed by a removed span $$\mathbf{H}_{\mathcal{L}'} = \{\mathbf{H}_l \mid l \in \mathcal{L}'\}.$$9, the approximation is

$\mathbf{W} = (W_1,\ldots,W_d),$0

and $\mathbf{W} = (W_1,\ldots,W_d),$1 is obtained from a calibration set by solving

$\mathbf{W} = (W_1,\ldots,W_d),$2

With an $\mathbf{W} = (W_1,\ldots,W_d),$3 objective, the solution becomes a closed-form least-squares estimate,

$\mathbf{W} = (W_1,\ldots,W_d),$4

The learned map is then merged into the second FFN/down-projection of the kept block, so no additional module remains at inference time. This yields a training-free static depth pruning method that reports up to 25% pruning with about 90% relative performance on open benchmarks, and up to 37.5% compression on Llama 3 70B while retaining at least about 90% relative performance (Shopkhoev et al., 5 May 2025).

Static pruning can also be paired with lightweight recovery. In the LLM study centered on block importance metrics, two repair mechanisms are examined after pruning: an emulated update, which adds the average block delta computed on a calibration set, and low-rank linear adapters, trained independently for each removed block. The reported result is that emulated update can yield at least 5% absolute improvement on MMLU at some pruning points and is competitive with, or superior to, the learned adapter approach (Siddiqui et al., 2024).

In tinyML, depth pruning is defined as retaining the first $\mathbf{W} = (W_1,\ldots,W_d),$5 layers of a trained model and attaching a new classifier head after layer $\mathbf{W} = (W_1,\ldots,W_d),$6. The proposed variation replaces the usual single dense head with a small auxiliary network built from depthwise separable convolutions, batch normalization, ReLU, global average pooling, and a dense output layer. The rationale is that intermediate feature maps are insufficiently interpreted by a shallow dense head; a small auxiliary network is a more effective interpreter of intermediate feature maps (Leon et al., 2022).

4. Empirical behavior in LLMs

The empirical profile of static depth pruning in LLMs is strongly regime dependent. In the 17-task controlled study comparing instruction-following models (LLM-instruct) and reasoning-augmented models (LLM-think), depth pruning outperforms width pruning on classification tasks, while width pruning is more robust for generation and reasoning. Among static depth methods, Shortened-Taylor is usually the best depth pruner for LLM-instruct. At 20% pruning, it records average performance decline below 7.04% on generation tasks, but at higher pruning ratios depth pruning degrades more quickly than width pruning on generation. For LLM-think, the picture is substantially harsher: even at 20% pruning, all static depth methods suffer severe degradation, with average performance decline exceeding 42%, and by 40% pruning performance collapses much further (Ding et al., 26 Jan 2026).

A separate analysis of LLaMa-2 7B and Mistral 7B reproduces earlier findings that about 10% of blocks can be removed without significant degradation in downstream metrics when static cosine- or norm-based scoring is used. It also shows that apparent robustness on MMLU can conceal cross-task damage: GSM-8k can degrade significantly even after pruning a single block, and ARC-Challenge is also notably sensitive. Static depth pruning in LLMs is therefore not well summarized by a single benchmark, even when that benchmark is widely used (Siddiqui et al., 2024).

More recent static criteria aim to stabilize the high-ratio regime. SimDiff reports that on LLaMA2-7B at 25% pruning, the MSSD variant reaches an average score of 44.74 and 91.44% retained performance, while on LLaMA3.1-8B it remains better than ShortGPT and EntroDrop from 4 to 12 removed layers. The same study highlights architecture sensitivity of cosine-only pruning: on Mistral-7B-v0.3, pruning 12 layers drives ShortGPT perplexity from 42.2 to 1,833.0, whereas the MASD variant remains at 47.7 (Chen et al., 21 Apr 2026).

Static depth pruning is also notable for how well nominal computation reduction translates into real runtime gains. Under a GEMM-centric taxonomy, static depth pruning is classified as static M pruning, because removing a layer or sublayer reduces token-axis work in every forward pass by the same amount. On Llama3.1-8B, it is reported as the strongest Pareto-optimal baseline and the method closest to its theoretical acceleration upper bound in memory-bounded scenarios. The benchmark places static depth on the prefill Pareto frontier at low quality loss, with the frontier transitioning from static depth at 0%–4% loss, to dynamic depth at 5%–16%, and finally to static width pruning at 17%–26%. At 25% sparsity, static M reaches about 1.29× prefill speedup and 1.32× decoding speedup; at 50% sparsity, it reaches 1.88× prefill and 1.91× decoding (Hu et al., 8 Jun 2026).

5. CNNs, diffusion U-Nets, tinyML, and tree ensembles

In CNNs, static depth pruning is often framed as layer-level pruning. A comprehensive three-dimensional pruning framework defines model depth as the number of blocks, with normalized depth ratio

$\mathbf{W} = (W_1,\ldots,W_d),$7

Its central claim is that pruning along one dimension only—depth, width, or resolution—often over-prunes that dimension and causes avoidable accuracy loss. The framework therefore optimizes

$\mathbf{W} = (W_1,\ldots,W_d),$8

with $\mathbf{W} = (W_1,\ldots,W_d),$9. The reported comparisons show that depth-only DBP often loses more accuracy than the joint approach; for example, on ResNet-50/ImageNet the depth-only baseline records 73.92% accuracy, versus 75.90% for the joint method (Wang et al., 2020).

LayerMerge pursues static depth compression by jointly pruning activation layers and convolution layers, then merging the remaining consecutive convolutions. Its key observation is that naive layer merging inflates effective kernel size,

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$0

which undermines latency gains. LayerMerge formulates a surrogate optimization problem over kept activations, kept convolutions, and merged kernel sizes, and solves it exactly after discretization via dynamic programming with recurrence

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$1

It reports consistent Pareto improvements over prior depth compression and layer pruning baselines on ResNet-34, MobileNetV2, and DDPM (Kim et al., 2024).

In tinyML deployment, hardware friendliness is a primary reason for using depth pruning: unlike magnitude pruning, it does not require sparse matrix computation and compression support. On MLPerfTiny Visual Wakewords, the auxiliary-network variant reports 93% parameter reduction and 51% FLOPS reduction with 0.65% accuracy drop at block5; on Cortex-M0, pruning at block6 reduces model size from 336 KB to 71 KB and latency from 904 ms to 551 ms while increasing accuracy from 80% to 81% (Leon et al., 2022).

Static depth pruning also appears outside neural networks. ForestPrune compresses tree ensembles by pruning entire depth layers from individual trees. Each tree $$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$2 receives a binary vector

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$3

with monotonicity constraints

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$4

so valid solutions are contiguous from the root downward. The pruned ensemble is then obtained by solving a regularized least-squares problem over these depth-selection vectors. The motivation is structural: the number of nodes in a tree increases exponentially with depth, so removing a few bottom layers drastically compactifies the ensemble (Liu et al., 2022).

6. Retrieval and index-oriented variants

Although not layer pruning in the transformer sense, retrieval research uses closely related static pruning principles. In dense retrieval, PCA-based static pruning computes a single offline projection from the document embedding matrix $$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$5 via

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$6

retains the first $$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$7 principal dimensions, and builds a reduced index

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$8

Queries are projected into the same reduced space,

$$\mathbf{b} = \{b_1,b_2,\ldots,b_k\} \subseteq \{1,2,\ldots,d\}$$9

and ranking is still performed by dot products,

$\mathbf{W}^{\setminus \mathbf{b}}$0

The method is static because PCA is learned once from document embeddings and applied offline to all documents. Across TAS-B, Contriever, and ANCE, it reports more than 50% dimensionality reduction with up to about a 5% reduction in nDCG@10, often less (Siciliano et al., 2024).

Sparse neural retrieval offers a different static analogue. Document-centric pruning retains only the top-$\mathbf{W}^{\setminus \mathbf{b}}$1 learned term impacts per document, with $\mathbf{W}^{\setminus \mathbf{b}}$2; term-centric pruning removes low-scoring postings from each posting list; agnostic pruning removes postings below a global threshold. Across MS MARCO, TREC-DL, and TripClick, the study reports 2× speedup with negligible effectiveness loss of at most 2%, and 4× speedup with minimal impact of at most 8% (Lassance et al., 2023).

These retrieval variants share the same operational property as neural static depth pruning: the reduction is determined before online inference, stored in the index, and reused uniformly. A plausible implication is that “static depth pruning” functions as part of a wider design pattern in which representational capacity is reduced globally, rather than selected adaptively per input.

7. Limitations, misconceptions, and current directions

A recurring misconception is that static depth pruning is uniformly the best pruning strategy. The current evidence is more conditional. For LLM-instruct, dynamic pruning methods such as SkipGPT and MOD outperform static methods on classification and generation, while for LLM-think static pruning preserves reasoning better overall because dynamic methods often produce repetitive, incoherent outputs and converge to higher training loss. The mechanistic interpretation offered in that study centers on path continuity: reasoning-heavy models appear to benefit from continuous, stable computation paths, making a smaller fixed graph preferable to unstable token-wise routing (Ding et al., 26 Jan 2026).

Another misconception is that one universal static heuristic is reliable across architectures and tasks. Prompt-based and input-guided studies argue the opposite. PuDDing is motivated by the observation that a transformer block important for one task may be removable without degrading another; on LLaMA-3.1-8B, swapping omission of block 29 for block 30 causes BoolQ accuracy to fall from about 62.2% to 38.0%, while PIQA and WinoGrande change only slightly. IG-Pruning makes a related claim at the block level: static fixed masks are brittle because different inputs prefer different block configurations, and masks with similar perplexity can differ sharply in downstream performance (Wee et al., 4 Feb 2025, Qiao et al., 4 Nov 2025).

Static depth pruning also remains sensitive to calibration and recovery data. The controlled LLM pruning study finds that calibration and post-pruning recovery should match the model’s original training distribution—using tulu-3-sft-mixture for LLM-instruct and OpenThoughts for LLM-think—because mismatched datasets such as C4, PTB, WikiText2, or Alpaca can preserve basic linguistic ability while failing badly on harder capabilities (Ding et al., 26 Jan 2026).

Methodologically, the field is moving from manually designed rules toward learned static policies and deployment-aware benchmarking. PPF shows that static pruning at a fixed target ratio can be cast as policy search over layer-wise structured pruning ratios, guided by a predictor that evaluates pruning masks in seconds rather than minutes. The GEMM-centric acceleration benchmark adds a separate caution: nominal sparsity alone is a weak predictor of real speedup, because non-GEMM overheads, kernel shape constraints, and decode-time behavior can erode theoretical gains (Ma et al., 4 Aug 2025, Hu et al., 8 Jun 2026).

The accumulated evidence therefore supports a narrower characterization. Static depth pruning is most effective when the deployment constraint rewards fixed, compiler-friendly structure; when calibration data are distribution matched; and when pruning ratios stay in ranges that do not destroy the sequential computation path the model depends on. A plausible implication is that future work will continue to combine static structural simplicity with more informed scoring, lightweight compensation, and stronger deployment realism, rather than treating fixed layer deletion as a universally sufficient compression primitive.