---
title: Delta Attention Selective Halting (DASH)
url: https://www.emergentmind.com/topics/delta-attention-selective-halting-dash
type: topic
---

# Delta Attention Selective Halting (DASH)

Delta Attention Selective Halting (DASH) is a training-free, inference-time method for accelerating the prefill stage of long-context Transformers by halting tokens whose self-attention updates have already stabilized. Its defining thesis is that **stability implies redundancy**: if a token’s representation is no longer changing meaningfully through the attention branch, then continuing to run full Transformer computation on that token in deeper layers is often wasteful. DASH operates during prefilling rather than decoding, applies a single-shot token-selection policy at a chosen depth, and is designed to remain compatible with hardware-efficient dense kernels such as FlashAttention [2604.18103].

## 1. Problem setting and motivation

DASH addresses the prefill bottleneck in long-context LLMs and LMMs. In autoregressive Transformers, inference comprises a **prefill** phase, which processes the entire prompt once and initializes the KV cache, and a **decoding** phase, which generates output tokens autoregressively using the cached prompt states. The paper focuses on prefill because for long contexts it becomes the dominant latency bottleneck, especially for long-document QA and summarization, retrieval-augmented prompts, and multimodal prompts with many visual tokens [2604.18103].

For an input sequence of length \(T\), prefilling runs attention and FFN over all prompt tokens across all layers. Because self-attention scales at least quadratically in sequence length in the standard formulation, the forward-pass cost rises rapidly as \(T\) grows. The paper argues that although quantization and FlashAttention improve memory efficiency and kernel execution, the arithmetic burden of long-context prefilling remains substantial. As prompts reach tens or hundreds of thousands of tokens, forward-pass FLOPs become the primary bottleneck, which in turn increases time-to-first-token [2604.18103].

The method is motivated by a critique of prior token-pruning approaches. Many earlier acceleration methods estimate token importance using accumulated attention weights or other heuristics derived from full attention matrices. The paper identifies two deployment problems with such strategies: they require access to the attention matrix, and they can break compatibility with hardware-efficient kernels whose efficiency depends on fused dense execution without explicitly materializing the full \(T \times T\) attention matrix. DASH therefore reframes the problem from identifying which tokens are important to identifying which tokens have already finished their job [2604.18103].

The core empirical intuition is that token-wise attention-branch updates across layers exhibit a highly skewed or heavy-tailed distribution: most tokens have update magnitudes near zero, while only a sparse subset continues to change substantially in deeper layers. The authors interpret this as evidence that many tokens approach **semantic fixed points**, i.e., depth regions where further self-attention updates become very small. This suggests that stabilized tokens can be halted with limited effect on downstream behavior [2604.18103].

## 2. Halting rule and algorithmic structure

DASH is a **single-shot token halting policy** applied during prefill. The model runs normally up to a chosen start layer \(l_s\), computes a per-token delta-attention score at that layer, retains the top-scoring tokens as an active set, halts the rest for all deeper layers, and then continues execution only on the compacted active sequence [2604.18103].

In this paper, **delta attention** does not denote a difference between two attention matrices. It denotes the magnitude of the self-attention residual update for each token at a given layer. If \(\mathbf{U}^{(l)}\) is the self-attention output before residual addition, then the token-wise score is

\[
\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.
\]

This score is interpreted as a proxy for the marginal benefit of further updating token \(t\) at depth \(l\). Tokens with larger \(\Delta_t^{(l)}\) are still receiving substantial contextual refinement; tokens with smaller values are treated as more stable and therefore more redundant [2604.18103].

At the selected start layer \(l_s\), DASH applies a ranking-based TopK rule rather than an absolute threshold. Let \(S=\{1,\dots,T\}\) denote all token indices and let \(\rho \in [0,1)\) denote the pruning ratio. Then

\[
K = \lfloor (1-\rho)T \rfloor, \qquad
S^\star = \mathrm{TopK}(S, K, \Delta^{(l_s)}).
\]

Tokens in \(S^\star\) remain active, while tokens in \(S \setminus S^\star\) are halted. Halted tokens keep their last updated hidden states and are no longer processed by self-attention or FFN in deeper layers. The decision granularity is therefore **token-wise**, but the schedule is **layer-triggered** at a single selected layer and then held fixed for the remainder of the network [2604.18103].

The halting mechanism is not adaptive in the sense of repeated per-layer or per-head stop decisions. The paper explicitly characterizes it as a **single-shot schedule**. It is also training-free and inference-only: the principal tunables are the start layer \(l_s\) and the pruning ratio \(\rho\). A lightweight perplexity-derived proxy is mentioned for selecting \(l_s\) without exhaustive sweeps, and the best start layer is reported as typically lying in \([0.3L, 0.5L]\), with \(0.4L\) often best or near-best [2604.18103].

A practical implementation detail appears in the appendix FLOPs analysis: DASH can always keep a prefix and suffix of the sequence. In the reported Qwen2.5-7B theoretical setting, the implementation keeps the first \(64\) tokens and last \(32\) tokens, so \(n_{\text{fix}}=96\). This is not the main formulation, but it is part of the implementation analysis [2604.18103].

## 3. Mathematical formulation and computational interpretation

The paper formulates DASH within the standard Transformer residual architecture. Let \(\mathbf{H}^{(l)} \in \mathbb{R}^{T\times d}\) denote the token representations entering block \(l\). The attention output before residual addition is

\[
\mathbf{U}^{(l)}=\mathrm{Attn}\!\left(\mathrm{LN}(\mathbf{H}^{(l)})\right).
\]

The post-attention residual state is

\[
\widetilde{\mathbf{H}}^{(l)}=\mathbf{H}^{(l)}+\mathbf{U}^{(l)},
\]

and the block output is

\[
\mathbf{H}^{(l+1)} = \widetilde{\mathbf{H}}^{(l)} + \mathrm{FFN}\!\left(\mathrm{LN}(\widetilde{\mathbf{H}}^{(l)})\right).
\]

DASH monitors the norm of \(\mathbf{U}^{(l)}_t\), not the full blockwise change. This distinction is methodologically central: the authors later compare attention-branch delta against block-level delta and report that attention-branch delta is consistently better across both text and vision-language benchmarks [2604.18103].

The computational motivation is explicit in the appendix FLOPs model. Using the per-layer proxy

\[
A(n) = 4nd^2 + 2n^2d + 2ndm,
\]

the baseline full-prefill cost is

\[
C_{\text{full}}(n) = L\,A(n).
\]

If the first \(l_s\) layers operate on full length \(n\) and the remaining \(L-l_s\) layers operate on compacted length \(\hat n\), then DASH has cost

\[
C_{\text{ours}}(n) = l_s A(n) + (L-l_s)A(\hat n).
\]

The resulting FLOPs reduction and speedup are written as

\[
r_{\text{FLOPs}}(n) = 1 - \frac{C_{\text{ours}}(n)}{C_{\text{full}}(n)},
\qquad
s_{\text{FLOPs}}(n) = \frac{C_{\text{full}}(n)}{C_{\text{ours}}(n)}.
\]

In the fixed-prefix/suffix implementation analysis, the effective kept length is

\[
\hat n = n_{\text{fix}} + (1-c)(n-n_{\text{fix}})
      = (1-c)n + c\,n_{\text{fix}},
\]

where \(c\) is the compression fraction over eligible tokens and \(n_{\text{fix}} = n_{\text{first}} + n_{\text{last}}\). These equations formalize the paper’s claim that the dominant savings arise because later layers run as ordinary dense kernels on a shorter sequence, rather than because of irregular fine-grained sparse execution [2604.18103].

A plausible implication is that DASH trades semantic selectivity for systems regularity: it uses a semantically motivated halting signal, but after selection it reverts to standard dense computation on a compacted tensor rather than maintaining dynamic sparsity throughout the stack.

## 4. Hardware compatibility and systems characteristics

A major systems claim of DASH is compatibility with FlashAttention and related dense-kernel implementations. The paper argues that prior pruning methods often require explicit attention matrices to compute token scores, which undermines the IO-aware fused execution model of FlashAttention. DASH avoids this by scoring tokens using the norm of the attention output vector,

\[
\Delta_t^{(l)}=\|\mathbf{U}^{(l)}_t\|_2,
\]

which can be computed from the attention output activations without materializing the full attention matrix [2604.18103].

This compatibility determines the execution strategy. DASH does not alter the kernel structure at every layer with complex dynamic masks. Instead, it runs dense attention normally before \(l_s\), performs one selection step at \(l_s\), compacts the sequence once, and then runs all deeper layers on the compacted dense sequence. The effective sequence length after halting becomes \(\hat T = \lfloor (1-\rho)T \rfloor\) in the main formulation. Halted tokens skip both self-attention and FFN, while their hidden states remain frozen [2604.18103].

The paper emphasizes four systems-level advantages: FlashAttention compatibility, no need to materialize attention matrices, preservation of dense-kernel execution after compaction, and practical end-to-end latency gains rather than only theoretical FLOPs savings. It also notes a secondary effect: halting prompt tokens shortens the KV cache and can reduce per-token decoding cost as well, even though the method is framed primarily as a prefill accelerator [2604.18103].

The implementation is not entirely parameter-free. It requires choosing the start layer \(l_s\) and pruning ratio \(\rho\), and real efficiency gains depend on sequence compaction and downstream execution quality. The paper reports text experiments on A100 40GB and vision-language experiments on RTX A6000 48GB. Code is stated to be released at `https://github.com/verach3n/DASH.git` [2604.18103].

## 5. Empirical results across text and vision-language settings

The text backbone used in the reported experiments is **Qwen2.5-7B-Instruct-1M**, and the vision-language backbone is **Qwen2-VL-7B**. Text evaluation is conducted on **LongBench-E** and **LooGLE**; VL evaluation spans six benchmarks including GQA, MME, POPE, MMStar, OCRBench, and ChartQA. Reported metrics include benchmark score, average score, average decline ratio (ADR) for VL, prefill speedup, generation speedup, end-to-end latency, throughput, and peak GPU memory [2604.18103].

On text benchmarks, the baseline Qwen2.5-7B-Instruct-1M scores **48.87** on LongBench-E and **22.69** on LooGLE. DASH scores **46.76** on LongBench-E and **19.94** on LooGLE. Compared compressed baselines on LongBench-E are FastV **43.99**, LLMLingua2 **44.16**, D\(^3\) **45.00**, and SnapKV(pr.) **46.15**; on LooGLE they are D\(^3\) **19.49**, LLMLingua2 **19.56**, FastV **19.78**, and SnapKV(pr.) **19.87**. The paper therefore identifies DASH as the best compressed method by average score on both suites, while remaining below the uncompressed backbone [2604.18103].

Under a 40% pruning ratio, the FlashAttention compatibility experiment reports the following. With eager execution, DASH reaches **46.78** on LongBench-E with \(1.52\times\) speedup and **19.90** on LooGLE with \(1.34\times\) speedup. With FlashAttention, DASH reaches **46.76** on LongBench-E with \(1.74\times\) speedup and **19.94** on LooGLE with \(1.71\times\) speedup. The paper highlights that accuracy is essentially unchanged between eager and FlashAttention variants, while FlashAttention yields substantially better latency [2604.18103].

Theoretical FLOPs speedup improves with context length. For Qwen2.5-7B with \(L=28\), \(l_s=\lfloor 0.4L\rfloor=11\), \(\rho=0.667\), and \(T=16{,}384\), the theoretical prefill FLOPs speedup is **\(1.83\times\)**. The appendix reports **\(1.76\times\)** at 8,192 tokens, **\(1.83\times\)** at 16,384, **\(1.92\times\)** at 32,768, **\(2.00\times\)** at 65,536, and **\(2.07\times\)** at 131,072, indicating that the benefit increases with longer context [2604.18103].

On LongBench-E task families, DASH often achieves prefill speedups around **\(1.72\times\)**, **\(1.74\times\)**, **\(1.82\times\)**, **\(1.86\times\)**, and up to **\(1.89\times\)**. End-to-end speedups are lower but still substantial, commonly around **\(1.28\times\)** to **\(1.56\times\)**. Peak memory changes are described as modest, with relative entries around **\(1.02\times\)**–**\(1.04\times\)**, and the paper does not present large memory savings as the main contribution [2604.18103].

Vision-language results are reported through ADR under aggressive token reduction. At 75% token reduction, DASH achieves **85.2** ADR, compared with FastV **80.0**, VisionZip **83.0**, PruMerge+ **79.5**, and DART **81.4**. At 88.89% reduction, DASH records **74.7**, exceeding FastV **65.9**, VisionZip **69.8**, PruMerge+ **68.4**, and DART **69.1**. At 93.75% reduction, DASH reaches **66.7**, versus FastV **58.7**, VisionZip **59.7**, PruMerge+ **58.9**, and DART **61.4**. The paper therefore characterizes DASH as degrading more gracefully under aggressive compression [2604.18103].

An additional VL efficiency table on MMBench-EN reports that with ViT retain 35% and LLM retain 35%, DASH has latency **70.6%** of vanilla, prefill latency **74.8%**, **13.1 TFLOPs**, **35.6 MB** KV cache, and accuracy **77.5**. With retain 20% and 20%, it has latency **61.6%**, prefill latency **62.5%**, **12.6 TFLOPs**, **20.2 MB** KV cache, and accuracy **73.9** [2604.18103].

## 6. Ablations, limitations, and relation to adjacent work

The ablations clarify which components of DASH are load-bearing. A directionality study under the same 40% pruning ratio reports the following LongBench-E/LooGLE scores: Vanilla **48.87 / 22.69**, Random **33.65 / 10.16**, **High \(\Delta_{\mathrm{attn}}\) halting** **25.45 / 10.22**, and **Low \(\Delta_{\mathrm{attn}}\) halting (DASH)** **46.76 / 19.94**. This shows that the method depends specifically on halting **low-\(\Delta\)** tokens; reversing the rule is catastrophic [2604.18103].

A single-shot versus multi-shot comparison on Qwen2.5-7B finds only small gains from repeated selection: 1-shot **46.76 / 19.94**, 2-shot **46.92 / 19.88**, 3-shot **47.14 / 20.21**, and 4-shot **46.87 / 20.03**. The authors therefore retain the single-shot design on efficiency grounds, arguing that it captures most of the benefit with lower overhead [2604.18103].

Start-layer selection is sensitive. For 20% token reduction on LongBench-E, the average score is **34.65** at \(0.2L\), **43.27** at \(0.3L\), **46.25** at \(0.4L\), **42.33** at \(0.5L\), and **30.11** at \(0.6L\). The paper concludes that halting too early or too late harms performance, and that good start layers typically lie in \([0.3L, 0.5L]\), with \(0.4L\) often near-optimal [2604.18103].

To validate the scoring signal, the appendix measures correlation between \(\Delta_{\mathrm{attn}}\) ranking and full-attention importance. It reports Spearman **0.74**, IoU **82.4%** at layer 10; Spearman **0.81**, IoU **86.7%** at layer 16; and Spearman **0.88**, IoU **91.2%** at layer 24. Agreement strengthens in deeper layers, which supports the use of attention-output magnitude as a proxy for future importance [2604.18103].

The paper also notes a modality difference: visual tokens saturate earlier, whereas textual tokens stabilize later. This helps explain why multimodal prompts can admit more aggressive token reduction. The heavy-tailed distribution of \(\Delta_{\mathrm{attn}}\) values is repeatedly emphasized as the structural reason TopK halting works effectively [2604.18103].

Several limitations are stated explicitly. The work does not exhaustively explore all model scales, architectures, deployment configurations, or system-level optimizations. Its effectiveness may weaken on tasks requiring fine-grained local detail, especially OCR-heavy or chart reasoning; in cases where many tokens remain active deep into the model; under poor start-layer choice; or on model families not covered in the study. The method assumes that diminishing attention residual updates are aligned with diminishing future importance, an assumption that is empirically supported in the studied models but not presented as universal [2604.18103].

A persistent source of confusion is the acronym **DASH** itself. In contemporary arXiv usage, the same acronym is also used for **Drift Aware advantage SHaping**, a training-time RL post-training method for reducing overthinking in reasoning traces; that work is explicitly **not** about Delta Attention Selective Halting [2607.00482]. Likewise, **DashAttention** denotes **Differentiable and Adaptive Sparse Hierarchical Attention**, a query-dependent sparse routing method rather than a halting mechanism [2605.18753]. Related selective-computation work includes **DELTA**, a training-free sparse attention mechanism for decode-time acceleration that preserves the full KV cache and selectively restricts attention reads in later layers, but does not implement halting [2510.09883]. In recurrent-memory settings, **Erase-then-Delta Attention** introduces address-selective erase and write decoupling, which is adjacent to selective memory suppression but not an explicit halting policy [2606.26560]. This suggests that DASH, in the literal sense of **Delta Attention Selective Halting**, is most precisely identified with the prefill-acceleration method that halts low-\(\Delta_{\mathrm{attn}}\) tokens once their attention updates have stabilized [2604.18103].

Source: https://www.emergentmind.com/topics/delta-attention-selective-halting-dash