Attention Skipping in Transformers

• Attention Skipping is a collection of techniques that modify attention computations in transformers, including residualization, token skipping, and block omission.
• These methods boost efficiency and training stability by leveraging cross-layer redundancy and reducing computational load in models such as ViTs, LLMs, and multimodal systems.
• Attention skipping strategies range from static layer removal for inference acceleration to dynamic token-level adaptations that improve self-attention conditioning.

Attention skipping denotes several distinct but related techniques for reducing, bypassing, reusing, or structurally regularizing attention computations in transformers and transformer-derived architectures. In recent work, the phrase has referred both to the identity residual path around self-attention inside a Vision Transformer block and to explicit omission or approximation of attention across layers, tokens, modalities, or time steps during inference. The resulting literature spans ViTs, decoder-only LLMs, multimodal LLMs, and Conformer-based ASR, and it treats “skip” as anything from $y = X + O$ rather than $y = O$ to static layer removal, current-layer token selection, V→V masking, or CTC-driven frame bypass (Ji et al., 4 May 2025, Venkataramanan et al., 2023, Wu et al., 19 Jan 2026, Ma et al., 7 Jun 2026, Shukor et al., 2024, Liu et al., 2024, Eyceoz et al., 3 Jun 2025, Zhu et al., 2024).

1. Terminological scope

The phrase is not unitary. In the ViT-conditioning literature, “skip attention” means the additive identity residual path around the self-attention operator itself, and it is explicitly distinguished from stochastic depth, skipped tokens, and sparse attention. In efficiency-oriented work, by contrast, attention skipping usually means that some attention computations are omitted, approximated, reused, or selectively masked at inference time.

Usage of the term	Mechanism	Representative paper
Residualized attention	Use $y = X + O$ instead of $y = O$ inside a self-attention block	"Always Skip Attention" (Ji et al., 4 May 2025)
Cross-layer reuse	Reuse $Z_{l-1}^{\mathrm{MSA}}$ through a lightweight $\Phi$ instead of recomputing MSA	"Skip-Attention: Improving Vision Transformers by Paying Less Attention" (Venkataramanan et al., 2023)
Token skipping in LLMs	Compute attention only for selected active tokens or queries	"Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference" (Wu et al., 19 Jan 2026)
Fixed layer/block omission	Skip entire intermediate decoder blocks according to a target speedup ratio	"Accelerating Inference in LLMs with a Unified Layer Skipping Strategy" (Liu et al., 2024)
Post-training block removal	Remove selected attention sub-blocks and compensate with learned scalars	"Hopscotch: Discovering and Skipping Redundancies in LLMs" (Eyceoz et al., 3 Jun 2025)
Multimodal visual-branch skipping	Bypass saturated V→V attention while keeping FFNs and cross-modal paths	"Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs" (Ma et al., 7 Jun 2026)
Static multimodal interval skipping	Skip entire blocks, FFNs, or SA layers with fixed schedules	"Skipping Computations in Multimodal LLMs" (Shukor et al., 2024)
Sequence-shortening before later attention	Use intermediate CTC to bypass or drop frames before later Conformer blocks	"Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognition" (Zhu et al., 2024)

A persistent source of confusion is that some papers use “skip” to preserve an identity path, while others use it to eliminate computation. The first interpretation is about conditioning and trainability; the second is about efficiency, latency, and memory.

2. Residual self-attention as a conditioning requirement in ViTs

"Always Skip Attention" formalizes the strongest architectural reading of the term: a self-attention block takes token matrix $X \in \mathbb{R}^{n \times d}$, forms

$Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$

$$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$

and then outputs either $y = O$ without a residual path or $y = O$0 with one. Its central empirical claim is that self-attention catastrophically fails to train when the skip path is removed, whereas removing the analogous FFN skip causes only modest degradation in ViTs, and CNNs such as ConvMixer can train competitively even without residuals (Ji et al., 4 May 2025).

The paper attributes this asymmetry to ill-conditioning intrinsic to self-attention. In the full softmax case, the Jacobian $y = O$1 contains multiple pathways in which a perturbation $y = O$2 is multiplied by learnable matrices and by data-dependent factors $y = O$3, $y = O$4, and $y = O$5. In the linear-attention surrogate,

$y = O$6

the directional derivative explicitly exhibits three factors of $y = O$7 multiplying $y = O$8 in aggregate. Under i.i.d. zero-mean initialization for $y = O$9, $y = X + O$0, and $y = X + O$1, Proposition 1 states

$y = X + O$2

with $y = X + O$3 statistically close to unity. The stated implication is that self-attention without skip becomes ill-conditioned even when the input tokens are only moderately conditioned.

The residual path acts as a spectral regularizer. If $y = X + O$4 is the Jacobian of the attention output, then the residualized block has

$y = X + O$5

Under the additional assumption that $y = X + O$6 is positive semidefinite, Proposition 2 states that

$y = X + O$7

The paper interprets this as a shift of the singular or eigen spectrum away from zero, improving gradient flow and preventing near-rank collapse.

The empirical evidence is correspondingly sharp. In ViT-Tiny, the SAB output embedding measured without the skip has condition number around $y = X + O$8, while the residualized SAB, residualized FFN, and FFN without residual are around $y = X + O$9. On CIFAR-10, removing the SAB skip drops top-1 accuracy by 22%; on Tiny-ImageNet, training diverges after about 30 epochs. Removing the FFN skip yields only about 2% accuracy loss. ConvMixer-Tiny, by contrast, trains competitively with or without residuals across datasets. The paper also reports that output embeddings collapse toward rank-1 when the SAB skip is removed, implying near-infinite condition numbers.

A second contribution is Token Graying, which preconditions input tokens because $y = O$0 scales with $y = O$1. SVD Token Graying computes $y = O$2, then reconstructs

$y = O$3

with $y = O$4, thereby amplifying non-maximal singular values relative to $y = O$5. DCT Token Graying applies a 2D DCT, normalizes magnitudes, raises them to the power $y = O$6, restores the sign, and applies the inverse DCT. SVDTG has complexity $y = O$7 and makes ViT-Base training about $y = O$8 slower in the reported implementation; DCTTG has complexity $y = O$9 and remains near baseline training time.

The accuracy gains are small but consistent. On ViT-Base/ImageNet-1k, DCTTG improves top-1 from 81.0% to 81.3% and top-5 from 95.3% to 95.4%, with $Z_{l-1}^{\mathrm{MSA}}$0 around 0.95 giving the best trade-off. SVDTG improves conditioning more strongly, with $Z_{l-1}^{\mathrm{MSA}}$1 and $Z_{l-1}^{\mathrm{MSA}}$2 dropping from 6.72/6.74 to about 5.71/5.73 at $Z_{l-1}^{\mathrm{MSA}}$3, and reaches 81.4% top-1. DCTTG also yields small gains across ViT-S, Swin-S, CaiT-S, and PVT V2 b3, and improves MAE finetuning from 83.0% to 83.2%.

The paper’s practical prescription is unusually categorical: always wrap self-attention with $Z_{l-1}^{\mathrm{MSA}}$4. It further recommends PreNorm, near-isotropic zero-mean initialization for $Z_{l-1}^{\mathrm{MSA}}$5, $Z_{l-1}^{\mathrm{MSA}}$6, and $Z_{l-1}^{\mathrm{MSA}}$7, conservative learning rates when residual topology is altered, DCTTG with $Z_{l-1}^{\mathrm{MSA}}$8 for low-overhead gains, and SVDTG with $Z_{l-1}^{\mathrm{MSA}}$9 when maximum conditioning improvement is worth the cost. Its main theoretical limitation is that the proofs use linear attention and treat output-embedding conditioning as a proxy for Jacobian conditioning; a complete softmax-attention Jacobian analysis remains open.

3. Cross-layer reuse in vision transformers

"Skip-Attention: Improving Vision Transformers by Paying Less Attention" uses the phrase in a different sense: not residualization, but cross-layer reuse of already computed self-attention outputs. The paper argues that attention maps and MSA outputs are highly correlated across adjacent ViT layers. For CLS-token attention, cosine similarities between adjacent layers often reach 0.97, and CKA shows high correlation for $\Phi$0 particularly between layers 3 to 10 in ViT-T/16 and for $\Phi$1 particularly between layers 2 to 8. This motivates replacing full MSA in selected layers with a learned approximation of the previous layer’s MSA output (Venkataramanan et al., 2023).

The mechanism is SkipAt. At a skipped layer $\Phi$2, the model reuses the previous MSA output and computes

$\Phi$3

then applies the residual and MLP as usual:

$\Phi$4

The default $\Phi$5 separates CLS and patch embeddings, applies $\Phi$6 with GeLU, reshapes patches to a $\Phi$7 grid, applies a depth-wise convolution with kernel $\Phi$8 and GeLU, flattens back, applies $\Phi$9, then adds Efficient Channel Attention before concatenating CLS again. Identity reuse is allowed but degrades accuracy because it lacks inter-token interaction.

The default static schedule skips MSA in layers 3–8 for ViT-T/S/B. The total number of layers does not change; rather, some MSA blocks are removed from the computational graph. This directly targets the $X \in \mathbb{R}^{n \times d}$0 component of attention. Standard MSA has cost $X \in \mathbb{R}^{n \times d}$1, while $X \in \mathbb{R}^{n \times d}$2 has cost $X \in \mathbb{R}^{n \times d}$3, so the savings increase with sequence length.

The reported results cover both classification and dense prediction. On ImageNet-1K at $X \in \mathbb{R}^{n \times d}$4, ViT-T/16 improves from 72.8% top-1, 1.2 GFLOPs, and 5.8k img/s to 72.9%, 1.1 GFLOPs, and 6.9k img/s; ViT-S/16 from 79.8%, 4.6 GFLOPs, and 3.2k to 80.2%, 4.0 GFLOPs, and 3.8k; ViT-B/16 from 81.8%, 17.6 GFLOPs, and 1.2k to 82.2%, 15.2 GFLOPs, and 1.5k. In DINO self-supervision with ViT-S/16, SkipAt reaches 74.1% versus 73.6% after 100 pretraining epochs and reduces training time from 131 to 96 GPU-hours. On ADE20K, UperNet with ViT-B improves from 45.6 mIoU, 787 GFLOPs, and 11.1 fps to 46.3 mIoU, 633 GFLOPs, and 15.5 fps. The method is also reported on SIDD image denoising, DAVIS video denoising, Pascal VOC probing, and Samsung Galaxy S22 on-device latency.

Its scope is therefore narrower than the residual-conditioning argument of "Always Skip Attention": SkipAt assumes standard residual and normalization structure remain intact and uses skipping to exploit cross-layer redundancy rather than to repair ill-conditioning. The main limitations stated in the paper are that identity reuse harms accuracy, $X \in \mathbb{R}^{n \times d}$5 design and skipping schedules matter, and the schedules are static rather than dynamically selected.

4. Token- and block-level skipping in LLMs

In decoder-only LLMs, attention skipping is primarily an inference-time acceleration strategy. Three representative directions differ in granularity: token-level selective execution, fixed sample-independent layer skipping, and post-training removal of task-redundant attention blocks.

Self-predictive token skipping. SPTS is a training-free framework for long-context inference that skips computation inside residual MHA and FFN blocks on a subset of tokens only. For residual token skipping, a block $X \in \mathbb{R}^{n \times d}$6 is evaluated only on active tokens $X \in \mathbb{R}^{n \times d}$7, while skipped tokens pass through the residual shortcut unchanged. PAP computes keys for all tokens, computes the query only for the last token, forms per-token attention scores to that last token averaged over heads, and selects the top-$X \in \mathbb{R}^{n \times d}$8 tokens for active attention. This reduces dense attention from $X \in \mathbb{R}^{n \times d}$9 to $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$0, and during prefilling it compresses the KV cache by storing K/V only for active tokens. LTP then builds an offline low-rank FFN proxy and combines its transformation magnitude with PAP’s attention score, while MSDP prunes candidates stagewise to reduce redundancy interference. On LLaMA-3.1-8B-Instruct with 32K tokens, SPTS achieves up to $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$1 TTFT and $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$2 end-to-end generation speedups; component-wise ablation at 32K gives $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$3 for PAP, $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$4 for PAP+LTP, and $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$5 for PAP+LTP+MSDP (Wu et al., 19 Jan 2026).

Unified layer skipping. Unified Layer Skipping removes entire intermediate decoder blocks according to a user-specified target speedup ratio $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$6, rather than input-dependent gating. With $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$7 total layers, it sets $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$8, retains the first and last layers, and keeps an evenly spaced set of intermediate layers. During response decoding, retained layers execute the full pre-LN block and skipped layers are identity passes $Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$9. Because the retained-layer set is fixed across samples, tokens, and time steps, the method is batch-compatible and KV-cache compatible. On A100-40G SXM, it increases tokens/sec by about 80%–290% versus the full model at batch size 1 and by about 30%–70% versus input-aware dynamic methods, while improving accuracy over top-only or bottom-only contiguous skipping. For LLaMA-13B, the reported unified-skipping results are BLEU/COMET/ROUGE-L of 26.93/78.74/28.65 at $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$0, 24.82/75.43/26.37 at $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$1, 24.24/75.05/26.21 at $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$2, 21.55/72.64/23.29 at $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$3, and 15.67/68.27/16.98 at $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$4 (Liu et al., 2024).

Task-adaptive post-training block removal. Hopscotch skips entire attention sub-blocks chosen by a greedy iterative removal criterion and compensates with four trainable scalars per transformer block: $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$5, $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$6, $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$7, and $$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$8. Its modified block equations are

$$A = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right),\qquad O = AV,$$9

$y = O$0

Setting $y = O$1 disables attention in layer $y = O$2. The method trains only these scalars, keeps base weights frozen, and uses the base model’s own greedy generations as targets. For Llama-3.1-8B-Instruct and Qwen2.5-7B, the paper reports less than a 2% drop in performance even after skipping four attention blocks. On Llama-3.1-8B-Instruct, skipping 4 attention blocks yields GSM8K FE 81.05 from 83.10, ARC 62.42 from 64.33, GPQA 29.53 from 29.07, Social IQA 49.49 from 49.85, and PAWS-X 61.90 from 63.26, while reducing total forward time by about 8.24%; skipping 7 blocks yields about 14.42% time reduction (Eyceoz et al., 3 Jun 2025).

Taken together, these methods define three distinct operating points. SPTS preserves the block graph but changes which tokens are processed. Unified Layer Skipping preserves a fixed subnetwork determined only by $y = O$3. Hopscotch preserves the original weights but removes selected attention blocks after a task-specific adaptation stage.

5. Multimodal variants

Two multimodal lines of work treat attention skipping as selective omission inside an autoregressive MLLM, but they differ sharply in granularity. "Skipping Computations in Multimodal LLMs" evaluates static, interval-based skipping schedules over entire blocks, FFN layers, or SA layers. A transformer block is written as

$y = O$4

With starting layer $y = O$5 and interval $y = O$6, every $y = O$7-th layer from $y = O$8 onward is skipped according to a fixed, task- and input-agnostic schedule. The paper studies T-only skipping during autoregressive decoding and P+T skipping over both prompt and text, finding that T-only generally performs better. It further reports that SA-only skipping is the most harmful variant, whereas Skip Block is best and Skip FFN is reasonably safe. For QA tasks, up to 33% of blocks can be skipped while retaining at least 90% of original accuracy; captioning typically tolerates only 15–25%. Starting later, such as $y = O$9, is more robust. During training, skipping computations can recover about 97% of original performance even when skipping half of the blocks or removing 70% of the weights, and training with smaller LLMs can match models 2 or 3 times larger on several tasks (Shukor et al., 2024).

"Look Less, Reason More" proposes V-Skip, a training-free inference paradigm based on Visual Attention Saturation. Its claim is that visual-to-visual self-attention in MLLMs rapidly stabilizes in deeper blocks, whereas FFNs remain necessary for projecting visual features into the evolving textual semantic space. The method therefore bypasses the V→V branch only, while preserving T→T, T→V, V→T, and the FFN. In a pre-norm block,

$y = O$00

V-Skip replaces the V→V component with a passthrough or, equivalently, masks the V→V score submatrix by setting it to $y = O$01 before softmax. Skip configurations are selected by few-shot calibration under a compute budget, typically over deeper blocks where saturation indicators such as $y = O$02 and $y = O$03 are small. Across diverse MLLMs, the paper reports 94.16%–100.31% performance retention while reducing compute and latency, and it emphasizes that preserving FFNs and cross-modal interactions is critical (Ma et al., 7 Jun 2026).

These two papers jointly illustrate a multimodal asymmetry. Coarse SA skipping can work, but it is comparatively fragile; finer structured omission of only the saturated intra-visual branch is more targeted. A plausible implication is that multimodal models are especially sensitive to which interaction channel is bypassed, not merely to how much computation is removed.

6. Speech recognition and recurring design questions

In speech recognition, attention skipping appears as dynamic sequence shortening before later attention layers. Skipformer splits a Conformer encoder into $y = O$04 and $y = O$05, uses an intermediate CTC head after $y = O$06, and partitions frames into crucial, skipping, and ignoring groups based on blank posterior $y = O$07 with threshold $y = O$08. In the default Mode 2, crucial frames are the non-blank set $y = O$09, skipping frames are the nearest right blank neighbors $y = O$10, and the remaining blank frames are ignored. Only crucial frames enter $y = O$11; skipping frames bypass it and are later recovered in original temporal order. This replaces the $y = O$12 attention cost in the second encoder stack with $y = O$13, where $y = O$14. The paper reports average sequence-length reduction to the crucial path of about $y = O$15 on Aishell-1 and $y = O$16 on Librispeech. On Aishell-1, Skipformer improves CER to 4.48% greedy and 4.27% rescoring, versus 5.12% and 4.64% for the Conformer baseline, while increasing Inv-RTF from 25.8 to 33.0 on CPU and from 49.6 to 81.7 on GPU for greedy decoding. On Librispeech with rescoring, Skipformer† reaches 3.07/7.99 WER on clean/other and is reported as 47%/56% faster than SqueezeFormer on CPU/GPU (Zhu et al., 2024).

Across the broader literature, skipping is rarely placed in the earliest representation-building stages. SkipAt skips ViT MSA mainly in layers 3–8, SPTS starts skipping from deeper layers, V-Skip targets deeper blocks after visual attention saturates, static MLLM schedules recommend avoiding early layers with $y = O$17, Unified Layer Skipping always preserves the first and last decoder layers, and Skipformer inserts its split after $y = O$18 rather than before the encoder (Venkataramanan et al., 2023, Wu et al., 19 Jan 2026, Ma et al., 7 Jun 2026, Shukor et al., 2024, Liu et al., 2024). This suggests a recurring division of labor in which early attention establishes structure and later attention is more redundant or more amenable to residual passthrough, selective execution, or structural masking.

The open problems are correspondingly technical. A complete softmax-attention Jacobian analysis remains open in the conditioning literature; SkipAt does not explore dynamic schedules; SPTS notes that highly cross-referential or globally entangled contexts can make aggressive token skipping harmful; V-Skip requires calibration because task-dependent reasoning depth varies; and static multimodal skipping warns that SA skipping is the most harmful option and that actual device speedups depend on implementation and hardware support (Ji et al., 4 May 2025, Wu et al., 19 Jan 2026, Ma et al., 7 Jun 2026, Shukor et al., 2024). Attention skipping is therefore best understood not as a single recipe, but as a family of interventions whose effectiveness depends on whether attention is being regularized, approximated, selectively executed, or removed outright.