---
title: Layer Skipping in Neural Networks
url: https://www.emergentmind.com/topics/layer-skipping
type: topic
---

# Layer Skipping in Neural Networks

Layer skipping denotes a family of methods that selectively bypass neural network layers or sublayers, either statically or adaptively, in order to alter computation, communication, or internal predictive behavior. In the literature covered here, the term spans several distinct mechanisms: inference-time identity bypass in residual networks, early-exit-style truncation, per-input or per-token dynamic routing, sublayer skipping, self-speculative draft construction, selective modality-token processing in vision-language models, and partial parameter updating in federated learning. Across these settings, the common primitive is that some subset of transformations is omitted while the remainder of the network continues to operate; what changes is the objective. In some works, the primary goal is latency or FLOP reduction [2404.06954; 2501.02336; 2505.17420]. In others, skipping is used to construct a more useful auxiliary distribution for contrastive decoding rather than to save compute [2407.10795], or to reduce communication in federated fine-tuning by freezing most layers [2504.10536].

## 1. Conceptual scope and formalizations

Layer skipping is not a single algorithmic pattern. In decoder-only transformers, a standard formalization replaces selected blocks by the identity map. If \(G^l \in \{0,1\}\) is the execution gate for layer \(l\), the skipped model can be written as
\[
T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),
\]
with compute cost
\[
c = \sum_{l=1}^L G^l.
\]
This formulation, used to compare layer skipping against early exit in decoder-only generation, makes explicit that skipping preserves the full model depth graph while omitting selected blocks, whereas early exit removes the entire suffix of upper layers [2410.20022].

Several later works adopt the same identity-bypass semantics at finer granularity. AdaSkip skips attention and FFN sublayers independently rather than whole transformer blocks, using cosine similarity between input and output vectors as an importance signal [2501.02336]. LayerRoute uses a hard binary gate at each transformer block,
\[
\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,
\]
with \(g_i = \mathbf{1}[\sigma_i > 0.5]\), so a skipped block incurs zero block compute [2606.01838]. DASH generalizes the action space further: a layer can be skipped, executed in INT4, executed in INT8, or executed in FP16, with the selection framed as a Markov Decision Process [2505.17420].

A distinct formalization appears in multilingual contrastive decoding. There, skipping does not mean pruning or speed-oriented routing; it means bypassing a contiguous span \([m,n)\) in an auxiliary forward pass while still running the upper layers. For a transformer with \(N\) layers,
\[
h_0 = Emb(x), \qquad
h_i = L_i(h_{i-1}) \quad i \in (0,m) \cup [n, N], \qquad
h_i = h_m \quad i \in [m,n),
\]
followed by output projection to amateur logits. The expert branch remains the ordinary full forward pass. This construction is used specifically to repair the language mismatch of early-exit contrastive decoding in multilingual generation [2407.10795].

A broader classification emerges from the surveyed papers.

| Category | Representative mechanism | Example papers |
|---|---|---|
| Static skipping | Fixed retained/omitted layers chosen offline | [2404.06954], [2505.17626], [2603.07475] |
| Dynamic skipping | Input-, token-, frame-, or action-conditioned routing | [2503.23798], [2505.17420], [2606.01838], [2401.02046], [2602.22896] |
| Early-exit-related | Stop at depth \(E\) or use prefix as draft | [2404.16710], [2410.20022], [2604.14612] |
| Non-efficiency use | Construct alternative decoding distributions | [2407.10795] |
| Training/communication use | Freeze subsets of layers during optimization | [2504.10536] |

This suggests that “layer skipping” is best understood as a control primitive over depth, rather than as a synonym for acceleration.

## 2. Static skipping and balanced depth reduction

Static skipping fixes one skip pattern for all inputs. In decoder-only LLM inference, Unified Layer Skipping (ULS) determines the retained layers solely from a target speedup ratio \(r\), retaining
\[
M = \left\lfloor \frac{N}{r} \right\rfloor
\]
layers, always including the bottom and top layers, and selecting intermediate retained layers in a spread-out pattern rather than dropping a contiguous top or bottom chunk [2404.06954]. For BLOOMZ-7B with 30 layers, the 2x ULS pattern retains \(\{0,2,4,\dots,26,29\}\), while skip-top and skip-bottom retain contiguous prefixes or suffixes. The paper reports that this balanced pattern improves both inference performance and throughput relative to Early Exit and SkipDecode on machine translation and text summarization, while also matching the target activated-layer count much more closely [2404.06954].

A related decoder-only study finds that pre-trained models are substantially more robust to layer removal via skipping than to early exit. Uniform layer skipping preserves final hidden states better than early exit at matched layer budgets, and an oracle over static skipped subnetworks can match full-model average ROUGE-L on Alpaca using only \(5.6/24\) layers on average, i.e. \(23.3\%\) of the layers; on CNN-DM the corresponding figure is \(5.5/24\), or \(22.9\%\) [2410.20022]. The same paper reports that small token-level controllers conditioned on hidden states do not outperform constant-input controllers, suggesting that naive per-token dynamic routing is harder than per-sequence depth allocation in decoder-only generation [2410.20022].

Static skipping also appears in diffusion language models. A task-agnostic method selects non-consecutive layers whose adjacent-layer cosine similarity exceeds a threshold \(\theta = 0.95\), while forbidding adjacent skips. Native diffusion LLMs, especially LLaDA, show substantial early-layer redundancy and tolerate skipping markedly better than autoregressive models: the reported headline is up to \(18.75\%\) FLOPs reduction while preserving over \(90\%\) performance on reasoning and code generation benchmarks, with 6 skipped layers as the main operating point [2603.07475]. By contrast, AR models degrade sharply, and AR-initialized diffusion models retain AR-like brittleness, which the paper interprets as persistent initialization bias [2603.07475].

In residual CNNs, a related static idea is to exploit stochastic-depth-trained robustness. A design-time search ranks skippable residual blocks by sensitivity and stores a Pareto front of configurations, then runtime switches among those prevalidated masks without learned gates [2505.17626]. On ResNet-110/CIFAR-10, accuracy stays within about \(1\%\) of the original up to around 20 skipped blocks, and the runtime scheduler can improve processed inferences per Watt by up to \(2.00\times\) relative to the original baseline in the reported setting [2505.17626]. Proportional Layer Skipping (PLS) pursues an even simpler static policy: retain a ratio \(r\) of layers within each residual group and combine that with processor frequency scaling under a latency constraint. On ResNet-152/CIFAR-10, PLS-50 reduces parameters from 58.16M to 29.04M while accuracy drops from 87.64% to 77.87% [2503.24014].

## 3. Dynamic and adaptive skipping during inference

Dynamic skipping conditions depth on the current sample, token, frame, or action. FlexiDepth is a token-wise, layer-wise adaptive scheme for frozen LLMs. It converts the latter 16 layers of Llama-3-8B-Instruct into routable layers, where each token is sent either through the full frozen layer or through a lightweight adapter. The router is a bottleneck MLP,
\[
G = \sigma(\text{Router}(\text{Norm}(X))),
\]
with threshold-based path selection, and the model includes a skip regularizer
\[
\mathcal{L}_{skip} = \frac{1}{T}\sum_{t=1}^{T}\left( \sum_{l=1}^{L} g^l_t \right)^2,
\qquad
\mathcal{L} = \alpha \cdot \mathcal{L}_{skip} + \mathcal{L}_{lm}.
\]
At the main operating point, FlexiDepth skips 8 out of 32 layers on Llama-3-8B-Instruct while retaining \(100.7\%\) average benchmark performance, substantially outperforming LayerSkip, ShortGPT, LaCo, and MindSkip at the same skip budget [2503.23798]. The same study reports that repetitive tokens and predictable collocations use fewer layers, whereas arithmetic results and uncertain continuations use more, and that removing the adapter or omitting KV-cache updates for skipped tokens causes severe degradation [2503.23798].

AdaSkip addresses long-context inference by selecting attention and FFN sublayers independently based on input-output cosine similarity. For historical tasks \(T_i\), average sublayer similarity is
\[
\bar{\mathrm{Similarity}}_j =
\frac{\sum_{i=1}^{N}\sum_{t=1}^{|T_i|}\mathrm{Similarity}(\vec a_{it}^j,\vec b_{it}^j)}
{\sum_{i=1}^{N}|T_i|},
\]
and skipped sublayers are approximated by a scale-corrected identity
\[
\hat{\vec b}_{it}^j = \bar{\mathrm{Scale}}_j \cdot \vec a_{it}^j.
\]
AdaSkip uses offline importance learning for prefilling and online FFN similarity from the first \(P\) decoded tokens to refine decoding skips [2501.02336]. Across long-context benchmarks such as GovReport, MultiNews, TriviaQA, and TREC, it preserves quality much better than SkipDecode, Unified Skipping, and Early Exit when skipping is applied to both prefilling and decoding [2501.02336].

LayerRoute pursues sequence-level adaptive depth in agentic language models. Each transformer block receives a tiny Linear(896,1) router and LoRA adapters on Q/K/V/O projections. Routing is hard-gated with a straight-through estimator, trained with a gate regularization term
\[
\mathcal{L} = \mathcal{L}_{\text{LM}} + \lambda \cdot \frac{1}{L}\sum_{i=0}^{L-1}\sigma(s_i),
\]
and initialized so middle layers start closed while early and late layers start open [2606.01838]. On a mixed tool-call/planning corpus, tool calls skip \(15.25\%\) of FLOPs while planning skips only \(2.34\%\), for a \(12.91\%\) skip differential, using 1.10M trainable parameters, i.e. \(0.223\%\) of the 494M frozen backbone [2606.01838].

DASH frames layer skipping as sequential decision-making over a multi-state action space \(\{0,1,2,4\}\), meaning skip, INT4, INT8, and FP16. The policy
\[
s_{i+1} \sim \pi_\theta(s_{i+1}\mid s_i, h_i)
\]
is trained with RL and cross-entropy, and skip-induced drift is compensated by layer-specific scaling plus mixed-precision execution [2505.17420]. The reported results show that at 1.33x on Qwen-2.5-7B, MMLU falls only from 70.2 to 69.7, while at 2.0x DASH still retains 61.0 versus far lower scores for Early-Exit, RandomSkip, SkipDecode, and AdaSkip [2505.17420].

Outside text-only LLMs, dynamic skipping appears in CTC ASR and robot manipulation. In CTC-based speech recognition, the first \(K\) encoder layers are always executed, then frames whose intermediate blank probability exceeds \(\tau = 0.99\) for the current and previous two frames skip the last \(L-K\) layers:
\[
h_t =
\begin{cases}
h_t^{in}, & \text{if blank confidence exceeds threshold with spike extension} \\
Enc_{K+1}^{L}(h_t^{in}), & \text{else.}
\end{cases}
\]
This yields about 29% inference acceleration with minor degradation, and KL distillation aligns intermediate and final spikes enough to recover near-baseline WER [2401.02046]. In VLA models, DySL-VLA partitions layers into always-executed informative layers and skippable incremental layers, and uses continuity of recent predicted actions as a prior guidance signal for when to enable skipping. On CALVIN, DySL-VLA attains average length 2.89 at 13.6 ms latency versus RoboFlamingo’s 2.92 at 51.0 ms and DeeR-VLA’s 2.83 at 19.3 ms [2602.22896].

## 4. Early exit, self-speculation, and draft construction by skipping

Early exit is the oldest depth-reduction pattern in this set of papers, but it is not identical to general layer skipping. The distinction is central in decoder-only transformers: early exit truncates the suffix, whereas skipping can preserve upper layers while omitting selected internal blocks [2410.20022]. LayerSkip trains models so that every intermediate layer can be unembedded by a shared LM head,
\[
e_k = g(x_k),
\]
and uses progressive layer dropout together with a shared early-exit loss across all layers [2404.16710]. The dropout rate factorizes as
\[
p_{l,t} = S(t)D(l)p_{\max},
\]
with higher dropout at deeper layers, and the loss sums weighted token-prediction losses from all exits [2404.16710]. This makes intermediate exits much more accurate than in standard training, enabling plain early exit and self-speculative decoding.

LayerSkip’s more consequential inference contribution is self-speculation: use layers \(1{:}E\) as a draft model, then layers \(E{+}1{:}L\) as a verifier within the same model, sharing compute and KV cache. The paper reports speedups up to \(2.16\times\) on CNN/DM summarization, \(1.82\times\) on coding, and \(2.0\times\) on TOPv2 semantic parsing [2404.16710]. The broader lesson is that later-layer skipping is much more reliable when followed by verification than when used as a standalone approximation.

Several later papers revisit draft construction without retraining. CLaSp builds a self-speculative draft model by dynamically skipping intermediate layers of the verifier and optimizing the skipped-layer set after each verification stage using approximate dynamic programming over hidden-state cosine similarity [2505.24196]. Its hidden-state recurrence searches over keep/skip choices to maximize similarity between sparse-path and full-model hidden states, and it reports speedups around \(1.3\times\) to \(1.7\times\) on LLaMA3-series models without altering the output distribution [2505.24196]. ConfLayers also targets self-speculative decoding, but selects skipped intermediate layers using entropy-based confidence from intermediate attention and FFN outputs projected through the shared LM head. It standardizes per-layer confidence, uses an adaptive local window and a threshold
\[
\hat c_i < \mu_{i,\text{local}} - \lambda \sigma_{i,\text{local}},
\]
and iteratively refines the skip set online. The reported average speedups are 1.15x on LLaMA-2-13B, 1.35x on LLaMA-2-70B, 1.11x on LLaMA-3-8B, and 1.31x on LLaMA-3-70B, with “up to 1.4x” overall [2604.14612].

These results clarify a recurring misconception. Skipping for self-speculation does not need to preserve the full model’s behavior by itself; it needs to produce a draft that yields a favorable acceptance-versus-cost trade-off under verification. This is why ConfLayers emphasizes accepted drafted token count \(M\), rather than acceptance rate alone, as the relevant optimization target [2604.14612].

## 5. Non-efficiency uses: decoding quality, multilinguality, and communication reduction

A notable strand of work uses skipping for reasons other than raw inference speed. In multilingual contrastive decoding, the problem is that DoLa-style early exits produce amateur logits biased toward English even when the final output is in another language. The proposed remedy is to skip a contiguous span of lower layers, then continue through the upper layers so the amateur branch still passes through the target-language realization machinery [2407.10795]. Two selection rules are proposed: heuristic layer skipping
\[
m \sim U(4, N/2 - 1), \qquad n = m + \mathrm{round}(N/8),
\]
and dynamic entropy-based skipping,
\[
m = n - \mathrm{round}(N/8), \qquad
n = \min \{ i : e_i - e_{i-1} > \delta \},
\]
with \(\delta = 0.1\) and smoothing heuristics in the appendix [2407.10795]. On MGSM multilingual reasoning, skip-layer contrastive decoding consistently improves over direct inference and DoLa; for example, on LLaMA3-8B the average rises from 38.5 to 42.7 with SL-H, while DoLa slightly degrades to 37.9 [2407.10795]. The same paper explicitly identifies extra inference cost as a limitation, since both expert and amateur branches are still required [2407.10795].

In federated fine-tuning, “layer skipping” refers instead to freezing lower layers and communicating only upper layers. Layer-Skipping FL partitions parameters into frozen \(\theta^f\) and trainable \(\theta^t\), updates only \(\theta^t\) on each client,
\[
\theta_i^{t,r+1} = \theta^{t,r} - \eta \nabla_{\theta^t}\mathcal{L}_i(\{\theta^f,\theta^t\};\mathcal D_i),
\]
and aggregates only that subset with FedAvg [2504.10536]. For LLaMA 3.2-1B with 32 layers, the main configuration freezes layers 1–24 and trains layers 25–32, reducing communication to 31% of full-model FL. Reported performance is 88.7 i2b2 F1 and 84.7 MIMIC Micro-F1, versus 87.1 and 82.8 for full-model FedAvg, and within roughly 2% of centralized training [2504.10536]. This is a different use of the same underlying control over depth: not skipping execution at inference, but skipping participation in optimization and synchronization.

The same broad pattern appears in multi-stage reasoning. LiteStage argues that conventional training-free skipping fails because skip sensitivity varies strongly across stages and because skipping can induce redundant output tokens that erase latency gains. It therefore performs an offline stage-wise search for layer budgets and combines it with confidence-based generation early exit [2510.14211]. On StrategyQA, LiteStage reaches \(1.70\times\) speedup at 62.0% accuracy, while UnifiedSkip and AdaSkip collapse at comparable skip budgets in the reported setup [2510.14211]. A plausible implication is that in staged pipelines, the correct control variable is not only depth per token, but budget per reasoning stage.

## 6. Empirical regularities, theory, and limitations

Several regularities recur across architectures and modalities. First, early and late layers often matter more than middle layers. LayerRoute converges to a clear pattern where layers 0–7 and 17–23 are usually run and layers 8–16 are comparatively skippable [2606.01838]. DASH similarly reports that many middle layers are relatively redundant, while early and late layers are more important [2505.17420]. In VLMs, theory and experiments suggest that early vision layers and late layers may be skippable under redundancy and weak cross-modal interaction, but middle layers should not be skipped because they concentrate cross-modal integration [2509.25584].

Second, skipping contiguous blocks is frequently worse than dispersed skipping. ULS outperforms contiguous top-layer and bottom-layer removal by maintaining a balanced sub-network across depth [2404.06954]. In native diffusion LLMs, non-consecutive skipped layers are far safer than consecutive skips: for LLaDA on GSM8K, non-consecutive 8-layer skipping retains 91.8% relative performance, while consecutive skips reduce this to 45.8 [2603.07475]. This suggests that representational continuity across depth, rather than raw skip count alone, is often the decisive variable.

Third, late-layer preservation can be semantically necessary even when lower layers are damaged. In multilingual contrastive decoding, skipping lower layers helps but skipping top layers collapses performance, confirming that top layers are essential for target-language conversion [2407.10795]. In self-speculative decoding, LayerSkip’s verifier is exactly the skipped suffix; if that suffix is removed entirely, quality loss becomes much larger [2404.16710].

Fourth, similarity-based criteria recur in both empirical heuristics and theory. AdaSkip uses input-output cosine similarity as a direct skippability signal for attention and FFN sublayers [2501.02336]. LiteStage reuses cosine similarity between residual input and sublayer output as an importance proxy [2510.14211]. In VLMs, a theoretical framework relates low adjacent-layer cosine distance and high probability of small distance to functional and informational redundancy, and argues that skipping is safe when redundancy coincides with weak cross-attention [2509.25584]. This suggests that geometric redundancy is becoming a unifying lens across otherwise disparate skipping methods.

The limitations are equally consistent. Many methods provide FLOP reduction without proportional wall-clock gain. FlexiDepth explicitly notes that its token-wise dynamic routing does not improve throughput on existing GPU hardware because of irregular control flow and branch imbalance [2503.23798]. LayerRoute reports FLOP reduction but not measured serving latency [2606.01838]. ULS, by contrast, is designed precisely to avoid this issue by making skip patterns static and batch-compatible [2404.06954]. Training-free methods often depend on heuristic thresholds or profiling environments, while learned methods introduce extra modules, training cost, or adaptation overhead [2505.17626; 2604.14612]. Finally, architecture dependence is substantial: native diffusion LLMs are notably more skippable than AR models [2603.07475], and TinyLlama-1.1B is far more amenable to stage-aware skipping than Qwen2.5-0.5B in LiteStage’s experiments [2510.14211].

Taken together, these results support a general but qualified interpretation. Layer skipping is most effective when it exploits an existing redundancy pattern—across depth, modalities, token types, action phases, or communication pathways—without destroying the specific transformations performed by the layers that remain essential. The papers surveyed here show that those essential transformations may correspond to language conversion [2407.10795], multimodal fusion [2509.25584], late-stage refinement [2603.07475], high-uncertainty generation [2503.23798], or task adaptation in federated learning [2504.10536]. The practical question is therefore not whether layers can be skipped in the abstract, but which layers, under which signals, and for which objective.

Source: https://www.emergentmind.com/topics/layer-skipping