Layer Skipping in Neural Networks

• Layer Skipping is a technique that selectively bypasses neural network layers or sublayers to control computation without removing essential transformations.
• Static and dynamic approaches employ fixed skip patterns, adaptive per-input routing, and multi-precision methods to optimize latency, FLOPs, and output quality.
• Applications span accelerating inference in LLMs, improving multilingual decoding, and reducing communication in federated learning, demonstrating its versatility across contexts.

Layer skipping denotes a family of methods that selectively bypass neural network layers or sublayers, either statically or adaptively, in order to alter computation, communication, or internal predictive behavior. In the literature covered here, the term spans several distinct mechanisms: inference-time identity bypass in residual networks, early-exit-style truncation, per-input or per-token dynamic routing, sublayer skipping, self-speculative draft construction, selective modality-token processing in vision-LLMs, and partial parameter updating in federated learning. Across these settings, the common primitive is that some subset of transformations is omitted while the remainder of the network continues to operate; what changes is the objective. In some works, the primary goal is latency or FLOP reduction (Liu et al., 2024, He et al., 4 Jan 2025, Yang et al., 23 May 2025). In others, skipping is used to construct a more useful auxiliary distribution for contrastive decoding rather than to save compute (Zhu et al., 2024), or to reduce communication in federated fine-tuning by freezing most layers (Zhang et al., 13 Apr 2025).

1. Conceptual scope and formalizations

Layer skipping is not a single algorithmic pattern. In decoder-only transformers, a standard formalization replaces selected blocks by the identity map. If $G^l \in \{0,1\}$ is the execution gate for layer $l$, the skipped model can be written as

$$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$

with compute cost

$c = \sum_{l=1}^L G^l.$

This formulation, used to compare layer skipping against early exit in decoder-only generation, makes explicit that skipping preserves the full model depth graph while omitting selected blocks, whereas early exit removes the entire suffix of upper layers (Glavas et al., 2024).

Several later works adopt the same identity-bypass semantics at finer granularity. AdaSkip skips attention and FFN sublayers independently rather than whole transformer blocks, using cosine similarity between input and output vectors as an importance signal (He et al., 4 Jan 2025). LayerRoute uses a hard binary gate at each transformer block,

$$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$

with $g_i = \mathbf{1}[\sigma_i > 0.5]$, so a skipped block incurs zero block compute (Sikdar, 1 Jun 2026). DASH generalizes the action space further: a layer can be skipped, executed in INT4, executed in INT8, or executed in FP16, with the selection framed as a Markov Decision Process (Yang et al., 23 May 2025).

A distinct formalization appears in multilingual contrastive decoding. There, skipping does not mean pruning or speed-oriented routing; it means bypassing a contiguous span $[m,n)$ in an auxiliary forward pass while still running the upper layers. For a transformer with $N$ layers,

$$h_0 = Emb(x), \qquad h_i = L_i(h_{i-1}) \quad i \in (0,m) \cup [n, N], \qquad h_i = h_m \quad i \in [m,n),$$

followed by output projection to amateur logits. The expert branch remains the ordinary full forward pass. This construction is used specifically to repair the language mismatch of early-exit contrastive decoding in multilingual generation (Zhu et al., 2024).

A broader classification emerges from the surveyed papers.

Category	Representative mechanism	Example papers
Static skipping	Fixed retained/omitted layers chosen offline	(Liu et al., 2024, Korol et al., 23 May 2025, Goel et al., 8 Mar 2026)
Dynamic skipping	Input-, token-, frame-, or action-conditioned routing	(Luo et al., 31 Mar 2025, Yang et al., 23 May 2025, Sikdar, 1 Jun 2026, Hou et al., 2024, Yang et al., 26 Feb 2026)
Early-exit-related	Stop at depth $E$ or use prefix as draft	(Elhoushi et al., 2024, Glavas et al., 2024, Amer et al., 16 Apr 2026)
Non-efficiency use	Construct alternative decoding distributions	(Zhu et al., 2024)
Training/communication use	Freeze subsets of layers during optimization	(Zhang et al., 13 Apr 2025)

This suggests that “layer skipping” is best understood as a control primitive over depth, rather than as a synonym for acceleration.

2. Static skipping and balanced depth reduction

Static skipping fixes one skip pattern for all inputs. In decoder-only LLM inference, Unified Layer Skipping (ULS) determines the retained layers solely from a target speedup ratio $l$0, retaining

$l$1

layers, always including the bottom and top layers, and selecting intermediate retained layers in a spread-out pattern rather than dropping a contiguous top or bottom chunk (Liu et al., 2024). For BLOOMZ-7B with 30 layers, the 2x ULS pattern retains $l$2, while skip-top and skip-bottom retain contiguous prefixes or suffixes. The paper reports that this balanced pattern improves both inference performance and throughput relative to Early Exit and SkipDecode on machine translation and text summarization, while also matching the target activated-layer count much more closely (Liu et al., 2024).

A related decoder-only study finds that pre-trained models are substantially more robust to layer removal via skipping than to early exit. Uniform layer skipping preserves final hidden states better than early exit at matched layer budgets, and an oracle over static skipped subnetworks can match full-model average ROUGE-L on Alpaca using only $l$3 layers on average, i.e. $l$4 of the layers; on CNN-DM the corresponding figure is $l$5, or $l$6 (Glavas et al., 2024). The same paper reports that small token-level controllers conditioned on hidden states do not outperform constant-input controllers, suggesting that naive per-token dynamic routing is harder than per-sequence depth allocation in decoder-only generation (Glavas et al., 2024).

Static skipping also appears in diffusion LLMs. A task-agnostic method selects non-consecutive layers whose adjacent-layer cosine similarity exceeds a threshold $l$7, while forbidding adjacent skips. Native diffusion LLMs, especially LLaDA, show substantial early-layer redundancy and tolerate skipping markedly better than autoregressive models: the reported headline is up to $l$8 FLOPs reduction while preserving over $l$9 performance on reasoning and code generation benchmarks, with 6 skipped layers as the main operating point (Goel et al., 8 Mar 2026). By contrast, AR models degrade sharply, and AR-initialized diffusion models retain AR-like brittleness, which the paper interprets as persistent initialization bias (Goel et al., 8 Mar 2026).

In residual CNNs, a related static idea is to exploit stochastic-depth-trained robustness. A design-time search ranks skippable residual blocks by sensitivity and stores a Pareto front of configurations, then runtime switches among those prevalidated masks without learned gates (Korol et al., 23 May 2025). On ResNet-110/CIFAR-10, accuracy stays within about $$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$0 of the original up to around 20 skipped blocks, and the runtime scheduler can improve processed inferences per Watt by up to $$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$1 relative to the original baseline in the reported setting (Korol et al., 23 May 2025). Proportional Layer Skipping (PLS) pursues an even simpler static policy: retain a ratio $$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$2 of layers within each residual group and combine that with processor frequency scaling under a latency constraint. On ResNet-152/CIFAR-10, PLS-50 reduces parameters from 58.16M to 29.04M while accuracy drops from 87.64% to 77.87% (Chan et al., 31 Mar 2025).

3. Dynamic and adaptive skipping during inference

Dynamic skipping conditions depth on the current sample, token, frame, or action. FlexiDepth is a token-wise, layer-wise adaptive scheme for frozen LLMs. It converts the latter 16 layers of Llama-3-8B-Instruct into routable layers, where each token is sent either through the full frozen layer or through a lightweight adapter. The router is a bottleneck MLP,

$$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$3

with threshold-based path selection, and the model includes a skip regularizer

$$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$4

At the main operating point, FlexiDepth skips 8 out of 32 layers on Llama-3-8B-Instruct while retaining $$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$5 average benchmark performance, substantially outperforming LayerSkip, ShortGPT, LaCo, and MindSkip at the same skip budget (Luo et al., 31 Mar 2025). The same study reports that repetitive tokens and predictable collocations use fewer layers, whereas arithmetic results and uncertain continuations use more, and that removing the adapter or omitting KV-cache updates for skipped tokens causes severe degradation (Luo et al., 31 Mar 2025).

AdaSkip addresses long-context inference by selecting attention and FFN sublayers independently based on input-output cosine similarity. For historical tasks $$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$6, average sublayer similarity is

$$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$7

and skipped sublayers are approximated by a scale-corrected identity

$$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$8

AdaSkip uses offline importance learning for prefilling and online FFN similarity from the first $$T_{LS}(\cdot) = \left(G^L d^L + (1-G^L)I\right)\circ \dots \circ \left(G^1 d^1 + (1-G^1)I\right)\circ e(\cdot),$$9 decoded tokens to refine decoding skips (He et al., 4 Jan 2025). Across long-context benchmarks such as GovReport, MultiNews, TriviaQA, and TREC, it preserves quality much better than SkipDecode, Unified Skipping, and Early Exit when skipping is applied to both prefilling and decoding (He et al., 4 Jan 2025).

LayerRoute pursues sequence-level adaptive depth in agentic LLMs. Each transformer block receives a tiny Linear(896,1) router and LoRA adapters on Q/K/V/O projections. Routing is hard-gated with a straight-through estimator, trained with a gate regularization term

$c = \sum_{l=1}^L G^l.$0

and initialized so middle layers start closed while early and late layers start open (Sikdar, 1 Jun 2026). On a mixed tool-call/planning corpus, tool calls skip $c = \sum_{l=1}^L G^l.$1 of FLOPs while planning skips only $c = \sum_{l=1}^L G^l.$2, for a $c = \sum_{l=1}^L G^l.$3 skip differential, using 1.10M trainable parameters, i.e. $c = \sum_{l=1}^L G^l.$4 of the 494M frozen backbone (Sikdar, 1 Jun 2026).

DASH frames layer skipping as sequential decision-making over a multi-state action space $c = \sum_{l=1}^L G^l.$5, meaning skip, INT4, INT8, and FP16. The policy

$c = \sum_{l=1}^L G^l.$6

is trained with RL and cross-entropy, and skip-induced drift is compensated by layer-specific scaling plus mixed-precision execution (Yang et al., 23 May 2025). The reported results show that at 1.33x on Qwen-2.5-7B, MMLU falls only from 70.2 to 69.7, while at 2.0x DASH still retains 61.0 versus far lower scores for Early-Exit, RandomSkip, SkipDecode, and AdaSkip (Yang et al., 23 May 2025).

Outside text-only LLMs, dynamic skipping appears in CTC ASR and robot manipulation. In CTC-based speech recognition, the first $c = \sum_{l=1}^L G^l.$7 encoder layers are always executed, then frames whose intermediate blank probability exceeds $c = \sum_{l=1}^L G^l.$8 for the current and previous two frames skip the last $c = \sum_{l=1}^L G^l.$9 layers: $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$0 This yields about 29% inference acceleration with minor degradation, and KL distillation aligns intermediate and final spikes enough to recover near-baseline WER (Hou et al., 2024). In VLA models, DySL-VLA partitions layers into always-executed informative layers and skippable incremental layers, and uses continuity of recent predicted actions as a prior guidance signal for when to enable skipping. On CALVIN, DySL-VLA attains average length 2.89 at 13.6 ms latency versus RoboFlamingo’s 2.92 at 51.0 ms and DeeR-VLA’s 2.83 at 19.3 ms (Yang et al., 26 Feb 2026).

4. Early exit, self-speculation, and draft construction by skipping

Early exit is the oldest depth-reduction pattern in this set of papers, but it is not identical to general layer skipping. The distinction is central in decoder-only transformers: early exit truncates the suffix, whereas skipping can preserve upper layers while omitting selected internal blocks (Glavas et al., 2024). LayerSkip trains models so that every intermediate layer can be unembedded by a shared LM head,

$$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$1

and uses progressive layer dropout together with a shared early-exit loss across all layers (Elhoushi et al., 2024). The dropout rate factorizes as

$$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$2

with higher dropout at deeper layers, and the loss sums weighted token-prediction losses from all exits (Elhoushi et al., 2024). This makes intermediate exits much more accurate than in standard training, enabling plain early exit and self-speculative decoding.

LayerSkip’s more consequential inference contribution is self-speculation: use layers $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$3 as a draft model, then layers $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$4 as a verifier within the same model, sharing compute and KV cache. The paper reports speedups up to $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$5 on CNN/DM summarization, $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$6 on coding, and $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$7 on TOPv2 semantic parsing (Elhoushi et al., 2024). The broader lesson is that later-layer skipping is much more reliable when followed by verification than when used as a standalone approximation.

Several later papers revisit draft construction without retraining. CLaSp builds a self-speculative draft model by dynamically skipping intermediate layers of the verifier and optimizing the skipped-layer set after each verification stage using approximate dynamic programming over hidden-state cosine similarity (Chen et al., 30 May 2025). Its hidden-state recurrence searches over keep/skip choices to maximize similarity between sparse-path and full-model hidden states, and it reports speedups around $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$8 to $$\mathbf{h}_{i+1} = g_i \cdot \text{Block}_i(\mathbf{h}_i) + (1-g_i)\cdot \mathbf{h}_i,$$9 on LLaMA3-series models without altering the output distribution (Chen et al., 30 May 2025). ConfLayers also targets self-speculative decoding, but selects skipped intermediate layers using entropy-based confidence from intermediate attention and FFN outputs projected through the shared LM head. It standardizes per-layer confidence, uses an adaptive local window and a threshold

$g_i = \mathbf{1}[\sigma_i > 0.5]$0

and iteratively refines the skip set online. The reported average speedups are 1.15x on LLaMA-2-13B, 1.35x on LLaMA-2-70B, 1.11x on LLaMA-3-8B, and 1.31x on LLaMA-3-70B, with “up to 1.4x” overall (Amer et al., 16 Apr 2026).

These results clarify a recurring misconception. Skipping for self-speculation does not need to preserve the full model’s behavior by itself; it needs to produce a draft that yields a favorable acceptance-versus-cost trade-off under verification. This is why ConfLayers emphasizes accepted drafted token count $g_i = \mathbf{1}[\sigma_i > 0.5]$1, rather than acceptance rate alone, as the relevant optimization target (Amer et al., 16 Apr 2026).

5. Non-efficiency uses: decoding quality, multilinguality, and communication reduction

A notable strand of work uses skipping for reasons other than raw inference speed. In multilingual contrastive decoding, the problem is that DoLa-style early exits produce amateur logits biased toward English even when the final output is in another language. The proposed remedy is to skip a contiguous span of lower layers, then continue through the upper layers so the amateur branch still passes through the target-language realization machinery (Zhu et al., 2024). Two selection rules are proposed: heuristic layer skipping

$g_i = \mathbf{1}[\sigma_i > 0.5]$2

and dynamic entropy-based skipping,

$g_i = \mathbf{1}[\sigma_i > 0.5]$3

with $g_i = \mathbf{1}[\sigma_i > 0.5]$4 and smoothing heuristics in the appendix (Zhu et al., 2024). On MGSM multilingual reasoning, skip-layer contrastive decoding consistently improves over direct inference and DoLa; for example, on LLaMA3-8B the average rises from 38.5 to 42.7 with SL-H, while DoLa slightly degrades to 37.9 (Zhu et al., 2024). The same paper explicitly identifies extra inference cost as a limitation, since both expert and amateur branches are still required (Zhu et al., 2024).

In federated fine-tuning, “layer skipping” refers instead to freezing lower layers and communicating only upper layers. Layer-Skipping FL partitions parameters into frozen $g_i = \mathbf{1}[\sigma_i > 0.5]$5 and trainable $g_i = \mathbf{1}[\sigma_i > 0.5]$6, updates only $g_i = \mathbf{1}[\sigma_i > 0.5]$7 on each client,

$g_i = \mathbf{1}[\sigma_i > 0.5]$8

and aggregates only that subset with FedAvg (Zhang et al., 13 Apr 2025). For LLaMA 3.2-1B with 32 layers, the main configuration freezes layers 1–24 and trains layers 25–32, reducing communication to 31% of full-model FL. Reported performance is 88.7 i2b2 F1 and 84.7 MIMIC Micro-F1, versus 87.1 and 82.8 for full-model FedAvg, and within roughly 2% of centralized training (Zhang et al., 13 Apr 2025). This is a different use of the same underlying control over depth: not skipping execution at inference, but skipping participation in optimization and synchronization.

The same broad pattern appears in multi-stage reasoning. LiteStage argues that conventional training-free skipping fails because skip sensitivity varies strongly across stages and because skipping can induce redundant output tokens that erase latency gains. It therefore performs an offline stage-wise search for layer budgets and combines it with confidence-based generation early exit (Kang et al., 16 Oct 2025). On StrategyQA, LiteStage reaches $g_i = \mathbf{1}[\sigma_i > 0.5]$9 speedup at 62.0% accuracy, while UnifiedSkip and AdaSkip collapse at comparable skip budgets in the reported setup (Kang et al., 16 Oct 2025). A plausible implication is that in staged pipelines, the correct control variable is not only depth per token, but budget per reasoning stage.

6. Empirical regularities, theory, and limitations

Several regularities recur across architectures and modalities. First, early and late layers often matter more than middle layers. LayerRoute converges to a clear pattern where layers 0–7 and 17–23 are usually run and layers 8–16 are comparatively skippable (Sikdar, 1 Jun 2026). DASH similarly reports that many middle layers are relatively redundant, while early and late layers are more important (Yang et al., 23 May 2025). In VLMs, theory and experiments suggest that early vision layers and late layers may be skippable under redundancy and weak cross-modal interaction, but middle layers should not be skipped because they concentrate cross-modal integration (Hartman et al., 29 Sep 2025).

Second, skipping contiguous blocks is frequently worse than dispersed skipping. ULS outperforms contiguous top-layer and bottom-layer removal by maintaining a balanced sub-network across depth (Liu et al., 2024). In native diffusion LLMs, non-consecutive skipped layers are far safer than consecutive skips: for LLaDA on GSM8K, non-consecutive 8-layer skipping retains 91.8% relative performance, while consecutive skips reduce this to 45.8 (Goel et al., 8 Mar 2026). This suggests that representational continuity across depth, rather than raw skip count alone, is often the decisive variable.

Third, late-layer preservation can be semantically necessary even when lower layers are damaged. In multilingual contrastive decoding, skipping lower layers helps but skipping top layers collapses performance, confirming that top layers are essential for target-language conversion (Zhu et al., 2024). In self-speculative decoding, LayerSkip’s verifier is exactly the skipped suffix; if that suffix is removed entirely, quality loss becomes much larger (Elhoushi et al., 2024).

Fourth, similarity-based criteria recur in both empirical heuristics and theory. AdaSkip uses input-output cosine similarity as a direct skippability signal for attention and FFN sublayers (He et al., 4 Jan 2025). LiteStage reuses cosine similarity between residual input and sublayer output as an importance proxy (Kang et al., 16 Oct 2025). In VLMs, a theoretical framework relates low adjacent-layer cosine distance and high probability of small distance to functional and informational redundancy, and argues that skipping is safe when redundancy coincides with weak cross-attention (Hartman et al., 29 Sep 2025). This suggests that geometric redundancy is becoming a unifying lens across otherwise disparate skipping methods.

The limitations are equally consistent. Many methods provide FLOP reduction without proportional wall-clock gain. FlexiDepth explicitly notes that its token-wise dynamic routing does not improve throughput on existing GPU hardware because of irregular control flow and branch imbalance (Luo et al., 31 Mar 2025). LayerRoute reports FLOP reduction but not measured serving latency (Sikdar, 1 Jun 2026). ULS, by contrast, is designed precisely to avoid this issue by making skip patterns static and batch-compatible (Liu et al., 2024). Training-free methods often depend on heuristic thresholds or profiling environments, while learned methods introduce extra modules, training cost, or adaptation overhead (Korol et al., 23 May 2025, Amer et al., 16 Apr 2026). Finally, architecture dependence is substantial: native diffusion LLMs are notably more skippable than AR models (Goel et al., 8 Mar 2026), and TinyLlama-1.1B is far more amenable to stage-aware skipping than Qwen2.5-0.5B in LiteStage’s experiments (Kang et al., 16 Oct 2025).

Taken together, these results support a general but qualified interpretation. Layer skipping is most effective when it exploits an existing redundancy pattern—across depth, modalities, token types, action phases, or communication pathways—without destroying the specific transformations performed by the layers that remain essential. The papers surveyed here show that those essential transformations may correspond to language conversion (Zhu et al., 2024), multimodal fusion (Hartman et al., 29 Sep 2025), late-stage refinement (Goel et al., 8 Mar 2026), high-uncertainty generation (Luo et al., 31 Mar 2025), or task adaptation in federated learning (Zhang et al., 13 Apr 2025). The practical question is therefore not whether layers can be skipped in the abstract, but which layers, under which signals, and for which objective.