LayerSkip: Accelerating Early-Exit Inference

• LayerSkip is an innovative method that trains decoder-only LLMs with depth-aware dropout and a shared LM head to enable reliable early exits.
• It combines early exit with self-speculative decoding, where early layers draft tokens and later layers verify and correct them within a single model.
• Empirical results demonstrate speedups up to 2× while maintaining competitive accuracy, highlighting its efficiency in adaptive-depth inference.

LayerSkip is an end-to-end method for accelerating inference in decoder-only LLMs by training the model to remain predictive at intermediate depths and then exploiting those intermediate predictions during decoding. In its canonical form, introduced as “LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding,” the method combines depth-dependent layer dropout with a shared-head early-exit objective, then performs self-speculative decoding in which early layers draft tokens and later layers verify and correct them, all within a single model (Elhoushi et al., 2024). Within the broader literature, “LayerSkip” is also used more generically as a label for layer-skipping or early-exit baselines, so the term can denote either the specific 2024 LLM method or a wider family of adaptive-depth inference schemes.

1. Conceptual basis and motivation

LayerSkip targets a central inefficiency of autoregressive decoding: standard decoder-only transformers execute all layers for every generated token, even though token prediction quality is not uniform across depth. By probing each layer’s hidden state through the shared LM head, the method’s authors report that early layers often produce weak or irrelevant predictions, later layers converge to the final token, and intermediate layers can be unstable, repeatedly revising predictions. In one Llama1 7B example, the average token needed 23.45 out of 32 layers, implying that even a perfect early-exit detector would save only about 26% of compute unless the model itself were retrained to make earlier layers more useful (Elhoushi et al., 2024).

This framing distinguishes LayerSkip from purely heuristic early-exit rules. The objective is not merely to decide when to stop computation, but to reshape training so that earlier hidden states become compatible with the final unembedding and can serve as credible partial predictors. The method is therefore best understood as a joint training-and-inference system: training alters the depthwise distribution of predictive burden, and inference converts that altered distribution into speedup through early exit and self-speculation (Elhoushi et al., 2024).

2. Training recipe: depth-aware dropout and shared early exits

The training procedure has two coupled components. First, LayerSkip applies layer dropout to the residual update

$x_{l+1,t} = x_{l,t} + M(p_{l,t}) f_l(x_{l,t}),$

where $M(p)$ is a Bernoulli mask and the dropout probability is

$p_{l,t} = S(t) D(l) p_{\max}.$

The defining choice is that dropout is lower in earlier layers and higher in later layers. The depth schedule reported to work best is exponential,

$D(l)=e^{\frac{l\ln 2}{L-1}-1},$

so earlier layers are more reliably trained while later layers are more aggressively perturbed. For continual pretraining and finetuning, the schedule is $S(t)=1$; for pretraining from scratch, the paper uses an exponential curriculum,

$S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$

Implementation is per sample within a batch, with dropped samples removed for the layer computation and then concatenated back; the authors also seed GPUs so each layer drops the same number of samples, improving training efficiency (Elhoushi et al., 2024).

Second, LayerSkip adds an early-exit loss at each layer, but without auxiliary per-layer heads. All transformer layers share the same LM head $g$, and the total objective is written as

$$J(X,Y,t)=\sum_{l=0}^{L-1}\tilde e(t,l)\,J_{\text{BCE}(g(x_{l+1}),Y)},$$

with normalized exit weights

$$\tilde{e}(t,l)=\frac{C(t,l)e(l)}{\sum_{i=0}^{L-1} C(t,i)e(i)}.$$

Here $C(t,l)$ is a curriculum over active exits and $M(p)$0 is a depth-dependent weighting scheme. The paper emphasizes that training all exits at every iteration slows training and harms final-layer quality, so it uses either a rotational curriculum $M(p)$1, which activates only every $M(p)$2-th layer at a time, or a gradual curriculum $M(p)$3, which enables exits progressively from later layers toward earlier ones (Elhoushi et al., 2024).

The significance of this design is architectural simplicity. LayerSkip does not add auxiliary exit heads, confidence modules, or external routers. The shared-head formulation trains the model to behave like an implicit ensemble of sub-models of different depths, all realized within a single backbone and a single unembedding function (Elhoushi et al., 2024).

3. Inference procedure: early exit and self-speculative decoding

At inference time, LayerSkip first supports conventional early exit: decoding can stop at a chosen exit depth $M(p)$4 and produce a token from the intermediate representation via the shared LM head. The method’s more distinctive inference mode is self-speculative decoding. In the draft stage, only the first $M(p)$5 layers are run autoregressively to generate draft tokens. In the verify stage, the remaining $M(p)$6 layers process those drafted tokens in parallel. In the correct stage, the system accepts the longest prefix that matches the verified output; once a mismatch is found, it emits the verified token at that position and resumes generation from there (Elhoushi et al., 2024).

This design differs from standard speculative decoding because it does not use a separate draft model. The draft and verifier are the same model split by depth. The paper highlights two implementation consequences. First, LayerSkip uses a single KV cache shared across draft and verification stages. Second, it caches the query vector at the exit layer, referred to as the “KVQ cache,” so verification can resume directly from layer $M(p)$7. In the reported CPU appendix, reusing KVQ saves about 9–20 ms/token on TOPv2 and CNN/DM tests, indicating that cache reuse is a material contributor to realized speedup rather than a purely conceptual convenience (Elhoushi et al., 2024).

Because draft and verify share parameters, activations, and the early-layer computation graph, the memory footprint is lower than two-model speculative decoding. The paper presents this as a core systems advantage: self-speculation retains the draft-then-verify logic while avoiding duplicated weights and duplicated caches (Elhoushi et al., 2024).

4. Empirical results across regimes and tasks

LayerSkip was evaluated under continual pretraining, pretraining from scratch, domain finetuning on code, and task finetuning on semantic parsing. The reported results show that speedups depend on both the training regime and the chosen exit depth, and that aggressive exits reduce token acceptance even when they maximize speed (Elhoushi et al., 2024).

Setting	Task / model	Reported result
Continual pretraining	CNN/DM, Llama2 7B	$M(p)$8 speedup; ROUGE-2 $M(p)$9
Continual pretraining	CNN/DM, Llama2 13B	$p_{l,t} = S(t) D(l) p_{\max}.$0 speedup; ROUGE-2 $p_{l,t} = S(t) D(l) p_{\max}.$1
Continual pretraining	XSUM, Llama2 7B / 13B	$p_{l,t} = S(t) D(l) p_{\max}.$2 / $p_{l,t} = S(t) D(l) p_{\max}.$3
Continual pretraining	HumanEval, Llama2 7B / 13B	$p_{l,t} = S(t) D(l) p_{\max}.$4 / $p_{l,t} = S(t) D(l) p_{\max}.$5
Pretraining from scratch	CNN/DM, Llama2 1.5B / 7B	$p_{l,t} = S(t) D(l) p_{\max}.$6 / $p_{l,t} = S(t) D(l) p_{\max}.$7
Code finetuning	HumanEval, Llama1 7B at $p_{l,t} = S(t) D(l) p_{\max}.$8	$p_{l,t} = S(t) D(l) p_{\max}.$9 speedup
Task finetuning	TOPv2, Llama 1.5B	$D(l)=e^{\frac{l\ln 2}{L-1}-1},$0 at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$1, $D(l)=e^{\frac{l\ln 2}{L-1}-1},$2 at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$3, $D(l)=e^{\frac{l\ln 2}{L-1}-1},$4 at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$5

On TOPv2, the paper reports token acceptance rates of 98.9% at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$6, 97.6% at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$7, and 76.0% at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$8. Exact match on the test set stays near the autoregressive baseline but is lower: autoregressive decoding reaches 85.9%, while self-speculative decoding at $D(l)=e^{\frac{l\ln 2}{L-1}-1},$9, $S(t)=1$0, and $S(t)=1$1 is reported as 82.9% in each case. The pattern illustrates the characteristic LayerSkip trade-off: smaller $S(t)=1$2 yields more speed but lower acceptance, while larger $S(t)=1$3 preserves agreement with the full-depth model (Elhoushi et al., 2024).

The paper also stresses an important negative result. Early exit alone can be extremely fast, but accuracy collapses on generation tasks. The empirical claim is therefore not that any intermediate layer suffices, but that the LayerSkip training recipe materially improves the utility of earlier exits, making self-speculative decoding practical on summarization, code generation, and semantic parsing (Elhoushi et al., 2024).

5. Relation to adjacent layer-skipping methods

Subsequent work situates LayerSkip within a rapidly diversifying design space for dynamic-depth inference. “Accelerating Inference in LLMs with a Unified Layer Skipping Strategy” replaces sample-dependent early exit with an input-independent, target-speedup-driven policy that retains the first and last layers and distributes the remaining retained layers uniformly through the middle. Its central critique is that sample-dependent methods such as Early Exit cannot guarantee a precise acceleration ratio and can interfere with batch decoding and KV caching, whereas a fixed retained-layer pattern yields stable speedup and a balanced subnetwork (Liu et al., 2024).

Other work changes the granularity rather than the control rule. “FFN-SkipLLM” argues that whole-layer skipping can cause generation collapse, hallucination of wrong facts, and noticeable performance drop even at the trivial exit ratio of $S(t)=1$4 of layers, attributing these effects primarily to ineffective handling of the KV cache through state copying during early-exit. It therefore skips only FFN blocks, keeps attention intact, and reports that 25–30% of FFN blocks can be skipped with marginal change in performance on knowledge-intensive generation tasks (Jaiswal et al., 2024). “FlexiDepth” instead attaches a router and adapter to a frozen Llama-3-8B-Instruct model, equips the latter 16 layers with adaptive skipping, and reports that skipping about 8 of 32 layers maintains 100.7% benchmark retention; in its comparisons, LayerSkip is described as performing very poorly on multi-token reasoning when aggressively skipping (Luo et al., 31 Mar 2025).

A separate line of work uses layer skipping as the drafter in self-speculative decoding, but without LayerSkip’s retraining recipe. “CLaSp” dynamically recomputes the skipped-layer set from the last verification stage via an approximate dynamic programming procedure and reports speedups of $S(t)=1$5 to $S(t)=1$6 on LLaMA3-series models without altering the original text distribution (Chen et al., 30 May 2025). “Balcony” freezes the pretrained backbone, inserts lightweight exit modules, trains them with KL self-distillation, and is explicitly positioned as a lighter alternative to LayerSkip; for LLaMA3-8B it reports 31B training tokens for Balcony versus 419B for LayerSkip, and states that Balcony outperforms LayerSkip across multiple model scales and benchmarks (Jamialahmadi et al., 6 Mar 2025).

LayerSkip also persists as a baseline in later architectural analyses. In hybrid LLMs, “Component-Aware Self-Speculative Decoding in Hybrid LLMs” treats LayerSkip as the generic early-exit self-speculative baseline. On the sequential hybrid Qwen3.5-0.8B at $S(t)=1$7 under greedy decoding, the paper reports acceptance $S(t)=1$8 for LayerSkip 33% versus $S(t)=1$9 for the component-aware strategy, a $S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$0 difference (Borobia et al., 1 May 2026). More generally, this suggests that LayerSkip remains a reference point even when the preferred acceleration abstraction shifts from layers to architectural components.

6. Limitations, misconceptions, and terminological scope

A common misconception is that LayerSkip is a purely post-hoc inference trick. In fact, the canonical method requires retraining or finetuning with layer dropout and early-exit supervision. The paper is explicit that hyperparameters such as $S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$1, $S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$2, and the rotational-curriculum parameter $S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$3 require tuning, that pretraining from scratch with layer dropout needs a higher learning rate, and that more aggressive exits trade higher speed for lower token acceptance and possible quality loss (Elhoushi et al., 2024). Later work reinforces these deployment caveats from other angles: Unified Layer Skipping argues that input-dependent early exit yields unstable realized speedup (Liu et al., 2024), while FFN-SkipLLM argues that whole-layer early-exit schemes can damage generation because of KV-cache handling (Jaiswal et al., 2024).

The term itself is also broader than the 2024 LLM method. In “Dual Skipping Networks,” a 2017 vision model uses a learned gating network inside skip-dense blocks to decide whether individual dense layers are executed or skipped, with a hard sigmoid gate

$S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$4

straight-through estimation, and $S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$5 possible hidden paths for $S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$6 dense layers (Cheng et al., 2017). A different and unrelated use appears in “Rethinking Skip Connection with Layer Normalization in Transformers and ResNets,” which proposes recursive skip connection with layer normalization,

$S_{\exp}(t)=e^{\frac{t\ln 2}{T-1}-1}.$7

as a modification of residual blocks rather than an inference-time early-exit scheme (Liu et al., 2021).

This terminological plurality matters because LayerSkip now denotes both a specific training-and-inference recipe for LLM self-speculation and, in some papers, a generic layer-skipping baseline. In the LLM literature, however, the defining contribution of the 2024 method is precise: it made early exit usable for self-speculative decoding without auxiliary exit modules or an external draft model, and it established a concrete template against which later dynamic-depth methods have continued to compare (Elhoushi et al., 2024).