Delta Attention Selective Halting (DASH)

• Delta Attention Selective Halting (DASH) is a training-free method that accelerates the prefill stage of long-context Transformers by halting tokens whose attention updates have stabilized.
• It computes a per-token delta score at a chosen layer, applying a single-shot TopK selection to retain only tokens that continue to evolve meaningfully.
• DASH remains compatible with hardware-efficient dense kernels like FlashAttention while achieving significant FLOPs reduction and latency improvements.

Delta Attention Selective Halting (DASH) is a training-free, inference-time method for accelerating the prefill stage of long-context Transformers by halting tokens whose self-attention updates have already stabilized. Its defining thesis is that stability implies redundancy: if a token’s representation is no longer changing meaningfully through the attention branch, then continuing to run full Transformer computation on that token in deeper layers is often wasteful. DASH operates during prefilling rather than decoding, applies a single-shot token-selection policy at a chosen depth, and is designed to remain compatible with hardware-efficient dense kernels such as FlashAttention (Chen et al., 20 Apr 2026).

1. Problem setting and motivation

DASH addresses the prefill bottleneck in long-context LLMs and LMMs. In autoregressive Transformers, inference comprises a prefill phase, which processes the entire prompt once and initializes the KV cache, and a decoding phase, which generates output tokens autoregressively using the cached prompt states. The paper focuses on prefill because for long contexts it becomes the dominant latency bottleneck, especially for long-document QA and summarization, retrieval-augmented prompts, and multimodal prompts with many visual tokens (Chen et al., 20 Apr 2026).

For an input sequence of length $T$, prefilling runs attention and FFN over all prompt tokens across all layers. Because self-attention scales at least quadratically in sequence length in the standard formulation, the forward-pass cost rises rapidly as $T$ grows. The paper argues that although quantization and FlashAttention improve memory efficiency and kernel execution, the arithmetic burden of long-context prefilling remains substantial. As prompts reach tens or hundreds of thousands of tokens, forward-pass FLOPs become the primary bottleneck, which in turn increases time-to-first-token (Chen et al., 20 Apr 2026).

The method is motivated by a critique of prior token-pruning approaches. Many earlier acceleration methods estimate token importance using accumulated attention weights or other heuristics derived from full attention matrices. The paper identifies two deployment problems with such strategies: they require access to the attention matrix, and they can break compatibility with hardware-efficient kernels whose efficiency depends on fused dense execution without explicitly materializing the full $T \times T$ attention matrix. DASH therefore reframes the problem from identifying which tokens are important to identifying which tokens have already finished their job (Chen et al., 20 Apr 2026).

The core empirical intuition is that token-wise attention-branch updates across layers exhibit a highly skewed or heavy-tailed distribution: most tokens have update magnitudes near zero, while only a sparse subset continues to change substantially in deeper layers. The authors interpret this as evidence that many tokens approach semantic fixed points, i.e., depth regions where further self-attention updates become very small. This suggests that stabilized tokens can be halted with limited effect on downstream behavior (Chen et al., 20 Apr 2026).

2. Halting rule and algorithmic structure

DASH is a single-shot token halting policy applied during prefill. The model runs normally up to a chosen start layer $l_s$, computes a per-token delta-attention score at that layer, retains the top-scoring tokens as an active set, halts the rest for all deeper layers, and then continues execution only on the compacted active sequence (Chen et al., 20 Apr 2026).

In this paper, delta attention does not denote a difference between two attention matrices. It denotes the magnitude of the self-attention residual update for each token at a given layer. If $\mathbf{U}^{(l)}$ is the self-attention output before residual addition, then the token-wise score is

$\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$

This score is interpreted as a proxy for the marginal benefit of further updating token $t$ at depth $l$. Tokens with larger $\Delta_t^{(l)}$ are still receiving substantial contextual refinement; tokens with smaller values are treated as more stable and therefore more redundant (Chen et al., 20 Apr 2026).

At the selected start layer $l_s$, DASH applies a ranking-based TopK rule rather than an absolute threshold. Let $T$0 denote all token indices and let $T$1 denote the pruning ratio. Then

$T$2

Tokens in $T$3 remain active, while tokens in $T$4 are halted. Halted tokens keep their last updated hidden states and are no longer processed by self-attention or FFN in deeper layers. The decision granularity is therefore token-wise, but the schedule is layer-triggered at a single selected layer and then held fixed for the remainder of the network (Chen et al., 20 Apr 2026).

The halting mechanism is not adaptive in the sense of repeated per-layer or per-head stop decisions. The paper explicitly characterizes it as a single-shot schedule. It is also training-free and inference-only: the principal tunables are the start layer $T$5 and the pruning ratio $T$6. A lightweight perplexity-derived proxy is mentioned for selecting $T$7 without exhaustive sweeps, and the best start layer is reported as typically lying in $T$8, with $T$9 often best or near-best (Chen et al., 20 Apr 2026).

A practical implementation detail appears in the appendix FLOPs analysis: DASH can always keep a prefix and suffix of the sequence. In the reported Qwen2.5-7B theoretical setting, the implementation keeps the first $T \times T$0 tokens and last $T \times T$1 tokens, so $T \times T$2. This is not the main formulation, but it is part of the implementation analysis (Chen et al., 20 Apr 2026).

3. Mathematical formulation and computational interpretation

The paper formulates DASH within the standard Transformer residual architecture. Let $T \times T$3 denote the token representations entering block $T \times T$4. The attention output before residual addition is

$T \times T$5

The post-attention residual state is

$T \times T$6

and the block output is

$T \times T$7

DASH monitors the norm of $T \times T$8, not the full blockwise change. This distinction is methodologically central: the authors later compare attention-branch delta against block-level delta and report that attention-branch delta is consistently better across both text and vision-language benchmarks (Chen et al., 20 Apr 2026).

The computational motivation is explicit in the appendix FLOPs model. Using the per-layer proxy

$T \times T$9

the baseline full-prefill cost is

$l_s$0

If the first $l_s$1 layers operate on full length $l_s$2 and the remaining $l_s$3 layers operate on compacted length $l_s$4, then DASH has cost

$l_s$5

The resulting FLOPs reduction and speedup are written as

$l_s$6

In the fixed-prefix/suffix implementation analysis, the effective kept length is

$l_s$7

where $l_s$8 is the compression fraction over eligible tokens and $l_s$9. These equations formalize the paper’s claim that the dominant savings arise because later layers run as ordinary dense kernels on a shorter sequence, rather than because of irregular fine-grained sparse execution (Chen et al., 20 Apr 2026).

A plausible implication is that DASH trades semantic selectivity for systems regularity: it uses a semantically motivated halting signal, but after selection it reverts to standard dense computation on a compacted tensor rather than maintaining dynamic sparsity throughout the stack.

4. Hardware compatibility and systems characteristics

A major systems claim of DASH is compatibility with FlashAttention and related dense-kernel implementations. The paper argues that prior pruning methods often require explicit attention matrices to compute token scores, which undermines the IO-aware fused execution model of FlashAttention. DASH avoids this by scoring tokens using the norm of the attention output vector,

$\mathbf{U}^{(l)}$0

which can be computed from the attention output activations without materializing the full attention matrix (Chen et al., 20 Apr 2026).

This compatibility determines the execution strategy. DASH does not alter the kernel structure at every layer with complex dynamic masks. Instead, it runs dense attention normally before $\mathbf{U}^{(l)}$1, performs one selection step at $\mathbf{U}^{(l)}$2, compacts the sequence once, and then runs all deeper layers on the compacted dense sequence. The effective sequence length after halting becomes $\mathbf{U}^{(l)}$3 in the main formulation. Halted tokens skip both self-attention and FFN, while their hidden states remain frozen (Chen et al., 20 Apr 2026).

The paper emphasizes four systems-level advantages: FlashAttention compatibility, no need to materialize attention matrices, preservation of dense-kernel execution after compaction, and practical end-to-end latency gains rather than only theoretical FLOPs savings. It also notes a secondary effect: halting prompt tokens shortens the KV cache and can reduce per-token decoding cost as well, even though the method is framed primarily as a prefill accelerator (Chen et al., 20 Apr 2026).

The implementation is not entirely parameter-free. It requires choosing the start layer $\mathbf{U}^{(l)}$4 and pruning ratio $\mathbf{U}^{(l)}$5, and real efficiency gains depend on sequence compaction and downstream execution quality. The paper reports text experiments on A100 40GB and vision-language experiments on RTX A6000 48GB. Code is stated to be released at https://github.com/verach3n/DASH.git (Chen et al., 20 Apr 2026).

5. Empirical results across text and vision-language settings

The text backbone used in the reported experiments is Qwen2.5-7B-Instruct-1M, and the vision-language backbone is Qwen2-VL-7B. Text evaluation is conducted on LongBench-E and LooGLE; VL evaluation spans six benchmarks including GQA, MME, POPE, MMStar, OCRBench, and ChartQA. Reported metrics include benchmark score, average score, average decline ratio (ADR) for VL, prefill speedup, generation speedup, end-to-end latency, throughput, and peak GPU memory (Chen et al., 20 Apr 2026).

On text benchmarks, the baseline Qwen2.5-7B-Instruct-1M scores 48.87 on LongBench-E and 22.69 on LooGLE. DASH scores 46.76 on LongBench-E and 19.94 on LooGLE. Compared compressed baselines on LongBench-E are FastV 43.99, LLMLingua2 44.16, D$\mathbf{U}^{(l)}$6 45.00, and SnapKV(pr.) 46.15; on LooGLE they are D$\mathbf{U}^{(l)}$7 19.49, LLMLingua2 19.56, FastV 19.78, and SnapKV(pr.) 19.87. The paper therefore identifies DASH as the best compressed method by average score on both suites, while remaining below the uncompressed backbone (Chen et al., 20 Apr 2026).

Under a 40% pruning ratio, the FlashAttention compatibility experiment reports the following. With eager execution, DASH reaches 46.78 on LongBench-E with $\mathbf{U}^{(l)}$8 speedup and 19.90 on LooGLE with $\mathbf{U}^{(l)}$9 speedup. With FlashAttention, DASH reaches 46.76 on LongBench-E with $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$0 speedup and 19.94 on LooGLE with $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$1 speedup. The paper highlights that accuracy is essentially unchanged between eager and FlashAttention variants, while FlashAttention yields substantially better latency (Chen et al., 20 Apr 2026).

Theoretical FLOPs speedup improves with context length. For Qwen2.5-7B with $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$2, $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$3, $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$4, and $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$5, the theoretical prefill FLOPs speedup is $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$6. The appendix reports $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$7 at 8,192 tokens, $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$8 at 16,384, $\Delta_t^{(l)} = \|\mathbf{U}^{(l)}_t\|_2.$9 at 32,768, $t$0 at 65,536, and $t$1 at 131,072, indicating that the benefit increases with longer context (Chen et al., 20 Apr 2026).

On LongBench-E task families, DASH often achieves prefill speedups around $t$2, $t$3, $t$4, $t$5, and up to $t$6. End-to-end speedups are lower but still substantial, commonly around $t$7 to $t$8. Peak memory changes are described as modest, with relative entries around $t$9–$l$0, and the paper does not present large memory savings as the main contribution (Chen et al., 20 Apr 2026).

Vision-language results are reported through ADR under aggressive token reduction. At 75% token reduction, DASH achieves 85.2 ADR, compared with FastV 80.0, VisionZip 83.0, PruMerge+ 79.5, and DART 81.4. At 88.89% reduction, DASH records 74.7, exceeding FastV 65.9, VisionZip 69.8, PruMerge+ 68.4, and DART 69.1. At 93.75% reduction, DASH reaches 66.7, versus FastV 58.7, VisionZip 59.7, PruMerge+ 58.9, and DART 61.4. The paper therefore characterizes DASH as degrading more gracefully under aggressive compression (Chen et al., 20 Apr 2026).

An additional VL efficiency table on MMBench-EN reports that with ViT retain 35% and LLM retain 35%, DASH has latency 70.6% of vanilla, prefill latency 74.8%, 13.1 TFLOPs, 35.6 MB KV cache, and accuracy 77.5. With retain 20% and 20%, it has latency 61.6%, prefill latency 62.5%, 12.6 TFLOPs, 20.2 MB KV cache, and accuracy 73.9 (Chen et al., 20 Apr 2026).

6. Ablations, limitations, and relation to adjacent work

The ablations clarify which components of DASH are load-bearing. A directionality study under the same 40% pruning ratio reports the following LongBench-E/LooGLE scores: Vanilla 48.87 / 22.69, Random 33.65 / 10.16, High $l$1 halting 25.45 / 10.22, and Low $l$2 halting (DASH) 46.76 / 19.94. This shows that the method depends specifically on halting low-$l$3 tokens; reversing the rule is catastrophic (Chen et al., 20 Apr 2026).

A single-shot versus multi-shot comparison on Qwen2.5-7B finds only small gains from repeated selection: 1-shot 46.76 / 19.94, 2-shot 46.92 / 19.88, 3-shot 47.14 / 20.21, and 4-shot 46.87 / 20.03. The authors therefore retain the single-shot design on efficiency grounds, arguing that it captures most of the benefit with lower overhead (Chen et al., 20 Apr 2026).

Start-layer selection is sensitive. For 20% token reduction on LongBench-E, the average score is 34.65 at $l$4, 43.27 at $l$5, 46.25 at $l$6, 42.33 at $l$7, and 30.11 at $l$8. The paper concludes that halting too early or too late harms performance, and that good start layers typically lie in $l$9, with $\Delta_t^{(l)}$0 often near-optimal (Chen et al., 20 Apr 2026).

To validate the scoring signal, the appendix measures correlation between $\Delta_t^{(l)}$1 ranking and full-attention importance. It reports Spearman 0.74, IoU 82.4% at layer 10; Spearman 0.81, IoU 86.7% at layer 16; and Spearman 0.88, IoU 91.2% at layer 24. Agreement strengthens in deeper layers, which supports the use of attention-output magnitude as a proxy for future importance (Chen et al., 20 Apr 2026).

The paper also notes a modality difference: visual tokens saturate earlier, whereas textual tokens stabilize later. This helps explain why multimodal prompts can admit more aggressive token reduction. The heavy-tailed distribution of $\Delta_t^{(l)}$2 values is repeatedly emphasized as the structural reason TopK halting works effectively (Chen et al., 20 Apr 2026).

Several limitations are stated explicitly. The work does not exhaustively explore all model scales, architectures, deployment configurations, or system-level optimizations. Its effectiveness may weaken on tasks requiring fine-grained local detail, especially OCR-heavy or chart reasoning; in cases where many tokens remain active deep into the model; under poor start-layer choice; or on model families not covered in the study. The method assumes that diminishing attention residual updates are aligned with diminishing future importance, an assumption that is empirically supported in the studied models but not presented as universal (Chen et al., 20 Apr 2026).

A persistent source of confusion is the acronym DASH itself. In contemporary arXiv usage, the same acronym is also used for Drift Aware advantage SHaping, a training-time RL post-training method for reducing overthinking in reasoning traces; that work is explicitly not about Delta Attention Selective Halting (Lee et al., 1 Jul 2026). Likewise, DashAttention denotes Differentiable and Adaptive Sparse Hierarchical Attention, a query-dependent sparse routing method rather than a halting mechanism (Huang et al., 18 May 2026). Related selective-computation work includes DELTA, a training-free sparse attention mechanism for decode-time acceleration that preserves the full KV cache and selectively restricts attention reads in later layers, but does not implement halting (Zarch et al., 10 Oct 2025). In recurrent-memory settings, Erase-then-Delta Attention introduces address-selective erase and write decoupling, which is adjacent to selective memory suppression but not an explicit halting policy (Li et al., 25 Jun 2026). This suggests that DASH, in the literal sense of Delta Attention Selective Halting, is most precisely identified with the prefill-acceleration method that halts low-$\Delta_t^{(l)}$3 tokens once their attention updates have stabilized (Chen et al., 20 Apr 2026).