---
title: 'ConfLayers: Confidence-Based Skipping'
url: https://www.emergentmind.com/topics/confidence-based-skipping-conflayers
type: topic
---

# ConfLayers: Confidence-Based Skipping

Confidence-Based Skipping (ConfLayers) refers to a family of adaptive computation techniques for deep neural networks—especially large language models (LLMs)—wherein individual layers are dynamically skipped at inference time according to a measure of confidence computed at intermediate network states. The core objective is to reduce inference latency and computational cost while minimally impacting output quality. ConfLayers comprises hard, data-dependent, and token-specific routing policies, typically derived from statistics over per-layer model confidences, and admits both plug-and-play inference-time adaptations (requiring no retraining) and fully end-to-end trainable variants. Current instantiations of ConfLayers are employed in state-of-the-art LLM speculative decoding, token-wise conditional computation, and adaptive routing frameworks.

## 1. Theoretical Foundation and Formalism

The principal mechanism underlying ConfLayers is the computation of a per-layer confidence score that reflects the network’s certainty in its intermediate activations. In Transformer-based LLMs with $L$ layers, let $h^i \in \mathbb{R}^d$ be the hidden state after layer $i$. Projecting $h^i$ through the language model head yields intermediate logits $\ell^i \in \mathbb{R}^K$ over the vocabulary of size $K$.

The softmax-normalized probabilities,
\[
p^{i}_j = \frac{\exp(\ell^i_j)}{\sum_{k=1}^K \exp(\ell^i_k)}
\]
allow computation of the entropy $H_i = -\sum_{j=1}^K p^{i}_j \log(p^{i}_j + \epsilon)$, with entropy complement $c_i = 1 - H_i / \log K$ serving as the normalized per-layer confidence ($c_i \in [0,1]$). High values indicate peaked distributions and high certainty.

Layer $i$ is skipped if its normalized confidence $\hat c_i$ falls below a locally-adaptive threshold $\tau_i$, which is determined using statistics from a local window of layers:
\[
\tau_i = \mu_{i,local} - \lambda \cdot \sigma_{i,local}
\]
where $\mu_{i,local}$ and $\sigma_{i,local}$ are the mean and standard deviation of normalized confidences in the window $W(i)$, and $\lambda$ is a tunable sensitivity parameter [2604.14612].

## 2. Iterative Adaptive Skipping Algorithm

ConfLayers is instantiated via an iterative search procedure that greedily optimizes the skip set $S \subset \{1, \ldots, L\}$ to maximize a downstream acceptance criterion (such as accepted tokens per speculative decoding window in self-speculative generation). The full procedure is as follows [2604.14612]:

1. **Initialize** with an initial skip set $S_0$ (e.g., uniform random skip ratio $\beta = |S| / L$).
2. **Draft Generation**: Use the model with layers $N = [L] \setminus S$ to speculatively generate $M$ tokens.
3. **Verification**: Validate generated tokens with the full model; record accepted tokens $A_t$.
4. **Selection and Update**: If $A_t$ exceeds current best, update the skip set. Compute per-layer confidences, normalize globally, compute local statistics, and update $S$ for next round:
   \[
   S_{t+1} = \{i : \hat{c}_i < \mu_{i,local} - \lambda \sigma_{i,local}\}
   \]
5. **Termination**: Stop once acceptance exceeds target $A^*$ or after $T_{max}$ rounds. Use $S_{best}$ for the remainder of inference.

This algorithm is executed every $\text{Opt\_Interval}$ tokens to amortize computation. Typically, $\beta \in [0.4, 0.6]$ yields the best empirical trade-offs.

## 3. Empirical Performance and Trade-Offs

Quantitative evaluation establishes that ConfLayers achieves consistent end-to-end inference speedup of $1.1$–$1.4\times$ across a broad range of models and tasks, including LLaMa-2 (13B, 70B), LLaMa-3 (8B, 70B), CodeLLaMa-34B, and Qwen-2.5-Math-72B on summarization, math reasoning, translation, and code synthesis [2604.14612]. Output quality, measured via metrics such as ROUGE-2 (summarization) and exact match (math), is preserved within $1$–$2\%$ of vanilla decoding.

For instance:

| Model/Task         | DEL   | SWIFT | ConfLayers |
|--------------------|-------|-------|------------|
| LLaMa2-13B         | 0.89× | 0.92× | 1.16×      |
| LLaMa2-70B         | 0.95× | 1.30× | 1.37×      |
| LLaMa3-8B          | 0.77× | 1.08× | 1.10×      |
| LLaMa3-70B         | 0.89× | 1.26× | 1.38×      |
| **Average**        | 0.93× | 1.03× | 1.15×      |

On CodeLLaMa-34B (HumanEval) and Qwen2.5-Math-72B (GSM8K), ConfLayers delivers speedups of $1.24\times$ and $1.22\times$ respectively at skip rates $\beta$ of $44\%$, confirming that the method provides practical gains on large models and diverse domains.

## 4. Implementation Details and Variants

ConfLayers requires no retraining: a forward pass is instrumented to extract per-layer logits, upon which skipping logic is applied according to local-adaptive thresholds. Its computational overhead is minimal ($O(L)$ per search interval), and inference-time integration entails only an index mask $S$ in the decoding loop.

Variants include:

- **Token-Wise Binary Routing**: Each token at each layer is routed via a binary gate (e.g., a small router MLP) using the straight-through Gumbel-Softmax trick, enabling per-token, per-layer granular control [2311.15436].
- **Plug-in Adapter Approaches**: A light-weight adapter is substituted for the original FFN in skipped layers, controlled by a continuous gating score $g^l_t \in (0,1)$ (e.g., FlexiDepth [2503.23798]). Skipping is thresholded at $g^l_t \le \tau$.
- **Speculative Decoding Integration**: ConfLayers forms an adaptive subnetwork ("draft model") in self-speculative decoding, optimizing the acceptance rate of speculative tokens to maximize end-to-end throughput [2604.14612].

Crucially, hard gating (true layer skipping in the forward pass) differentiates ConfLayers from prior soft gating and early-exit schemes, which did not provide real computation savings [2311.15436].

## 5. Comparative Evaluation and Related Methods

In contrast to non-adaptive baselines (uniform skipping, random gating) and soft early-exit methods (e.g., DeeBERT, Right-Tool), ConfLayers assigns computation on a per-token basis throughout all layers. Conditional Mixture-of-Experts approaches also gate computation, but typically involve expert modules rather than within-layer skipping [2311.15436]. Compared to them, ConfLayers entails minimal overhead and is compatible with frozen pretrained weights.

Prior work such as SkipNet learned to conditionally skip convolutional blocks via supervised and reinforcement learning to optimize for both accuracy and reduced computation in vision models, yielding $30$–$90\%$ computation savings without accuracy loss [1711.09485]. However, SkipNet did not employ entropy-based confidence as the skip criterion, in contrast to later ConfLayers instantiations in language modeling.

A summary distinguishing features:

| Method               | Routing Granularity      | Confidence Metric        | Training Required | FLOP Reduction  |
|----------------------|-------------------------|-------------------------|-------------------|-----------------|
| ConfLayers [2604.14612, 2311.15436] | Layer/token | Entropy-complement (confidence) | No / Optional    | Hard, per-layer |
| FlexiDepth [2503.23798]     | Layer/token           | Router MLP ($g^l_t$)           | Yes              | Hard, per-layer |
| LiteStage [2510.14211]      | Generation (stage)    | Logit max/$p_{t,v}$, Sliding   | No               | Early-exit (token) |
| SkipNet [1711.09485]        | Convolution block     | Activations, learned gating     | Yes              | Hard, per-block |

## 6. Limitations and Extensions

Several operational caveats apply to ConfLayers use:

- Highly adversarial or out-of-distribution inputs can attenuate the informativeness of intermediate confidences, degrading skip reliability.
- Very short decoding intervals or minimal window sizes can induce noisy statistics; increasing window sizes or interval frequency can partially mitigate this.
- Wall-clock runtime improvements may lag behind theoretical FLOP savings for fine-grained skipping due to caching, memory bandwidth, or control-flow limitations on standard hardware, especially in token-wise conditional computation modes [2503.23798].
- For very tall models ($L > 100$), retuning of window sizes and sensitivity parameters may be required.

ConfLayers naturally extends to encoder-decoder and encoder-only architectures, as well as non-autoregressive settings or dynamic head pruning (i.e., routing over heads within layers).

## 7. Practical Deployment and Research Directions

The plug-and-play nature of ConfLayers allows direct insertion into inference pipelines for LLMs of arbitrary scale; per-layer confidences are readily computable from existing model outputs. The adaptive, statistics-driven windowing and thresholding yield robust performance across tasks and model sizes. Typical hyperparameter ranges are: $\lambda \in [0.25, 0.5]$, skip rates $\beta$ in $[0.4, 0.6]$, base window $w_{base}=2$ and $w_{max} \sim L/5$.

Open research questions include:

- Theoretical analysis of worst-case quality loss as a function of confidence dynamics.
- Fusing confidence-adaptive skipping with speculation length optimization and hardware-efficient control flow.
- Extension to context-aware dynamic routing and efficient implementation under quantization or kernel sparsification regimes.

ConfLayers currently represents a state-of-the-art, general-purpose, adaptive compute-control facility in large-scale language model inference, consistently delivering strong latency-compute savings with empirically negligible quality deficits [2604.14612, 2311.15436, 2503.23798, 2510.14211].

Source: https://www.emergentmind.com/topics/confidence-based-skipping-conflayers