---
title: Adaptive Sparse Attention Module (ASAM)
url: https://www.emergentmind.com/topics/adaptive-sparse-attention-module-asam
type: topic
---

# Adaptive Sparse Attention Module (ASAM)

Searching arXiv for the cited papers and closely related ASAM terminology.
{"query":"id:2502.12216 OR title:\"Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs\"","max_results":5}
{"query":"\"Adaptive Sparse Attention Module\" arXiv long-context attention ASAM","max_results":10}
{"query":"2502.12082 AdaSplash adaptive sparse flash attention arXiv","max_results":5}
Adaptive Sparse Attention Module (ASAM) denotes a class of sparse-attention mechanisms that replace dense, fixed-pattern, or fixed-budget attention with input-dependent selection rules. In the specific formulation described in "Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs," ASAM is a decode-phase mechanism for long-context LLM inference that selects tokens by cumulative attention mass rather than by a predetermined token count, and approximates that selection through clustering-based sorting and distribution fitting [2502.12216]. More broadly, the same label, or closely related “ASAM-like” formulations, has been used for distinct modules in long-context reasoning, long-video MLLMs, video diffusion, and efficient entmax-based transformers, with the common objective of making sparsity adaptive to heads, layers, queries, contexts, or motion regimes rather than statically imposed [2510.01585].

## 1. Definition and problem setting

In Tactic, ASAM is introduced for the decode phase of long-context LLM inference, where repeatedly attending over a very large KV cache becomes the dominant systems bottleneck. The motivating observation is that in long contexts, loading K/V vectors can take over 50% of decode time, while prior sparse-attention or KV-compression methods typically enforce a fixed token budget per head or layer [2502.12216]. Tactic argues that this assumption is mismatched to actual attention behavior because sparsity varies across heads, across layers, and across query tokens and contexts. Some heads are described as “retrieval” heads with broader attention, whereas others are “streaming” heads with very peaky attention; similarly, earlier and later layers exhibit different sparsity levels, and the same head can become denser or sparser depending on the token being generated.

Within that framing, ASAM replaces the fixed-budget objective with a target cumulative attention score. This shifts the question from “how many tokens should be kept?” to “how much of the attention probability mass should be preserved?” The module is explicitly described as sparsity-adaptive and calibration-free, and its adaptation is performed online from the current query’s attention pattern rather than from manual layer/head budget assignment or calibration data [2502.12216].

A related but broader use of the name appears in ReSSFormer, where ASAM is one of three architectural components and combines token-level sparsity, activation sparsity through sparsemax or entmax, and expert-level sparsity through sparse routing. There the emphasis is not KV-cache loading during autoregressive decode, but replacing dense self-attention with focused, sparse computation for long-context reasoning [2510.01585]. This suggests that “ASAM” has become a general label for adaptive sparse attention subsystems, but the Tactic formulation is specifically an online, query-aware, attention-mass-controlled selection mechanism for long-context LLM decoding.

## 2. Attention-mass criterion and approximation guarantee

For one decode step with query \(q\) and previous tokens \(1,\dots,n\), Tactic writes full attention as
\[
o = \sum_{i=1}^{n}s_i v_i,\qquad s_i=\frac{\exp\left(\frac{qk_i^\top}{\sqrt{d}}\right)}{\sum_{j=1}^{n}\exp\left(\frac{qk_j^\top}{\sqrt{d}}\right)}.
\]
Given a selected token set \(I\), the cumulative attention score is defined as
\[
p(I)=\sum_{i\in I}s_i =\frac{\sum_{i\in I}\exp\left(\frac{qk_i^\top}{\sqrt{d}}\right)} {\sum_{j=1}^{n}\exp\left(\frac{qk_j^\top}{\sqrt{d}}\right)}.
\]
ASAM then selects the smallest set of tokens whose cumulative attention mass reaches a target fraction \(P\), i.e. the minimal prefix \(I\) in descending attention order such that
\[
p(I)\ge P.
\]
The target \(P\) is a fraction of total attention probability mass, not a fraction of tokens. If attention is concentrated, only a few tokens are required; if it is spread out, more tokens are selected. In that sense, the token count is a consequence of the attention distribution rather than an externally fixed prior [2502.12216].

The paper also states an approximation bound under the assumption that value vectors have similar norms across tokens:
\[
\epsilon(I)=\|o-\tilde{o}(I)\| \leq 2(1-p(I))\max_i\|v_i\|.
\]
Accordingly, when \(p(I)\) is close to 1, the sparse output is provably close to full attention. This is the main formal argument for preferring cumulative attention mass over fixed token budgets: the criterion is directly tied to approximation quality rather than indirectly controlled through a budget heuristic [2502.12216].

A conceptually related criterion appears in OmniSparse, where KV budget selection is also expressed through retained attention mass, though in a different architecture and with a different allocation rule. There, the budget \(b\) is chosen from the flattest head as the minimal number of keys whose cumulative accumulated attention reaches \(pN\), and the paper cites the theoretical error upper bound \((1-p)\cdot \|\mathbf{V}\|\) [2511.12201]. The shared theme is that adaptive sparse attention can be parameterized by retained mass or recall instead of by a fixed top-\(k\).

## 3. Tactic ASAM pipeline: clustering-based sorting and distribution fitting

The core difficulty in Tactic is that identifying the minimal set satisfying \(p(I)\ge P\) would ordinarily require both the exact ranking of token importance and the cumulative sum of exact attention scores. ASAM approximates both with a two-stage pipeline: clustering-based sorting followed by distribution fitting [2502.12216].

During prefill, for each head in each layer, Tactic runs K-means clustering on key vectors \(K\), grouping tokens by key-vector similarity rather than position. The K-means objective is
\[
\min_{\{\mathcal{C}_m\},\{\mu_m\} \sum_{m}\sum_{k_i\in \mathcal{C}_m}\|k_i-\mu_m\|^2,
\]
where \(\mu_m\) is a cluster centroid. The motivation is that positional grouping is unreliable because consecutive tokens can have very different key vectors while far-apart tokens can be semantically similar. At decode time, the query \(Q\) is scored against each cluster centroid using \(Q\cdot \mu_m\), clusters are sorted by that score, and tokens are unpacked in cluster order to form a partially sorted list. The paper reports an average cluster size of about 32 and states that clustering overhead stays below 6% of prefill time [2502.12216].

Clustering only approximates the order; it does not recover the absolute attention values needed for cumulative-mass stopping. Tactic therefore fits a simple long-tail curve after sorting:
\[
y = \frac{a}{x}+b,
\]
where \(x\) is token rank and \(y\) estimates attention score magnitude. The parameters \(a,b\) are fit using two sampled windows in the middle of the sorted sequence, for example around 10% and 60% of tokens, by computing mean exponential attention values \(\mu_1,\mu_2\). The first 1–2% of tokens are treated as outliers: their exact \(\exp(qk^\top/\sqrt{d})\) values are computed directly, and the fitted curve is used for the remainder [2502.12216].

The appendix procedure summarized in the paper is: fit \(y=a/x+b\); define
\[
w_i =
\begin{cases}
\exp(x_i\cdot Q/\sqrt{d}), & i\le N \\
a/i+b, & i>N
\end{cases}
\]
and choose the smallest \(k\) such that
\[
\sum_{i=1}^{k} w_i \ge P\cdot \sum_{i=1}^{n} w_i.
\]
The returned \(k\) is the token budget for that head. In operational terms, ASAM estimates the entire descending attention curve, accumulates estimated mass, and stops at the prescribed attention fraction \(P\) rather than at a fixed top-\(k\) [2502.12216].

## 4. Adaptation across heads, layers, contexts, and grouped-query attention

The adaptivity of Tactic’s ASAM is explicitly multi-level. Token selection is performed per head using that head’s query and keys, so different heads can produce different budgets and selected sets. Clustering is also done for each head in each layer during prefill, which captures layer-specific key structure. Because the query token changes at every decode step, the ranking and estimated cumulative mass are re-evaluated online, so the selected tokens can also change with context and generation state [2502.12216].

This per-head, per-layer, per-step adaptivity is what the paper contrasts with static allocation methods such as PyramidKV or Ada-KV. The central claim is not merely that ASAM is sparse, but that the sparsity level is allowed to vary with the actual attention distribution instead of being prescribed by calibration tables or manual budget schedules [2502.12216].

For Grouped-Query Attention, where multiple query heads share one KV head, Tactic computes selected tokens per query head and then takes the union of selected tokens for the shared KV head. The paper notes that finding the exact minimum union satisfying all heads is NP-hard, so the union is used as a practical approximation. It further reports that taking the union across GQA heads can yield up to 1.65× attention speedup versus processing query heads separately [2502.12216].

The importance of head heterogeneity appears across several later adaptive sparse-attention papers. OmniSparse determines a shared KV budget by analyzing the flattest head via kurtosis, then lets each head select its own top-\(b\) keys under that shared budget [2511.12201]. HASTE assigns per-head top-\(p\) thresholds offline through Error-guided Budgeted Calibration and decides per-head mask reuse online through query-key drift, arguing that mask stability and threshold-response curves vary substantially across heads [2605.14513]. AdaSpa likewise states that DiT sparse patterns vary with inputs, layers, and heads, motivating head-adaptive hierarchical block sparse attention [2502.21079]. Taken together, these formulations support a broader interpretation of adaptive sparse attention as an answer to head- and layer-level heterogeneity rather than a single universal sparsity pattern.

## 5. Empirical behavior in Tactic

Tactic reports that ASAM consistently outperforms fixed-budget sparse methods including Quest, PyramidKV, and Ada-KV / Ada-SnapKV. On LongBench, it achieves the best scores across HotpotQA, TriviaQA, MultiFieldQA, Qasper, NarrativeQA, and Musique, and at both 70% and 90% cumulative-attention thresholds it is better than the baselines, often much better at the same approximate token budget. On RULER, it again outperforms baselines across context lengths 16K–96K, and at 90% threshold its accuracy is described as close to full attention and clearly better than other sparse methods [2502.12216].

The paper also evaluates attention approximation quality by comparing the global optimal token selection order, the cluster optimal order, and Tactic. It states that Tactic gets very close to the cluster-optimal token count and reaches the desired cumulative attention threshold with high success rates. The reported success rates are 92% at 50% threshold, 89% at 60%, 86% at 70%, 84% at 80%, and 86% at 90% [2502.12216].

For output-distribution fidelity, the paper reports that on PG19, Tactic has the lowest KL-divergence from full attention among the compared methods, indicating that the sparse output distribution is closest to full attention. For efficiency, it reports up to 7.29× decode attention speedup and up to 1.58× end-to-end inference speedup. These numbers are paired with the claim that clustering overhead is small relative to prefill and that distribution fitting and cluster sorting are lightweight in decode [2502.12216].

The broader sparse-attention literature represented in the supplied papers shows analogous but task-specific efficiency claims. Adamas reports up to 4.4× self-attention and 1.5× end-to-end speedups on 32K-length sequences while matching full attention at a 64-token budget and being near-lossless at 128 [2510.18413]. OmniSparse reports comparable performance to full attention together with up to 2.7× speedup during prefill and 2.4× memory reduction during decoding [2511.12201]. These are not directly comparable experimental settings, but they reinforce the recurring empirical pattern that adaptive selection is used to preserve output quality under aggressive sparsity.

## 6. Related ASAM formulations and conceptual boundaries

The label “Adaptive Sparse Attention Module” is not confined to Tactic. In ReSSFormer, ASAM is a module inside a larger recurrent architecture and combines hard top-\(k\) key selection, sparse activation mappings \(\phi \in \{\text{sparsemax}, \text{entmax}_\alpha\}\), and expert-level sparsity with \(E=8\) experts and \(e=2\) active experts per head. There the stated computational transition is \(O(n^2)\to O(nk)\), and ablation shows that removing ASAM changes Wikitext PPL from 17.4 to 18.6 and HotpotQA EM from 71.2 to 66.1 [2510.01585]. This use of ASAM is architectural and end-to-end rather than an inference-only KV-selection mechanism.

OmniSparse presents an “ASAM-like adaptive sparse attention” in long-video MLLMs that operates in both training and inference. Its three mechanisms are query selection via lazy-active classification, KV selection with head-level dynamic budget allocation based on the flattest head, and KV cache slimming during decoding by skipping visual KV fetches for lazy heads. The paper explicitly frames its contribution as bridging the training–inference gap seen in inference-only sparsification [2511.12201]. By contrast, Tactic is explicitly post-training and calibration-free, with a lightweight prefill-time clustering step that is not treated as training or calibration in the usual sense [2502.12216].

Other related mechanisms occupy additional points in the design space. AdaSplash makes \(\alpha\)-entmax attention practical by a hybrid Halley-bisection solver and custom Triton kernels, using adaptive sparsity from the probability map itself rather than from explicit top-\(k\) token budgeting [2502.12082]. Adamas is a training-free, token-level sparse attention method that applies the Hadamard transform, bucketization, 2-bit compression, Manhattan-distance estimation, and top-\(k\) token retrieval, and its adaptivity is query-dependent rather than learned by a gating network [2510.18413]. AdaSpa, DynamicRad, and HASTE focus on video diffusion, where sparse patterns are described respectively as dynamic blockified patterns with online precise search, content-adaptive radial candidate sets with dual operating modes, and head-wise adaptive top-\(p\) control through Temporal Mask Reuse and Error-guided Budgeted Calibration [2502.21079; 2604.20470; 2605.14513].

A common misconception is to treat all such modules as interchangeable under the single term ASAM. The supplied papers do not support that simplification. Tactic’s ASAM is centered on cumulative attention mass and clustering-plus-curve-fitting for long-context LLM decode [2502.12216]; ReSSFormer’s ASAM is a sparse-attention submodule integrated with recurrent reasoning and structure induction [2510.01585]; OmniSparse’s ASAM-like framework is training-aware and multi-dimensional [2511.12201]. A plausible implication is that “ASAM” is best understood as a family resemblance term for adaptive sparse-attention designs rather than as a single canonical algorithm.

## 7. Strengths, limitations, and preferred operating regimes

For Tactic, the reported strengths are explicit: it is adaptive, calibration-free, theoretically grounded through the cumulative-attention approximation bound, efficient because clustering and curve fitting are cheap enough for decoding, accurate relative to fixed-budget methods, and compatible with GQA through the union strategy [2502.12216]. The method is positioned for settings where long-context decode latency is a concern, some approximation is acceptable, and accuracy matters more than aggressive compression.

The limitations stated in the paper are equally specific. ASAM remains approximate because it relies on clustering and curve fitting rather than exact full attention. Prefill includes clustering overhead, though it is reported as small. The union strategy for GQA is a practical approximation rather than the exact global optimum. The method is also best suited to cases where attention score distributions are reasonably smooth so that curve fitting works well [2502.12216].

Across the related literature, analogous trade-offs recur in different forms. AdaSplash notes that adaptive but unstructured sparsity depends on the amount of learned sparsity and that the forward pass can be slower than FlashAttention-2 before sparsity is high enough to skip blocks [2502.12082]. HASTE emphasizes that even training-free top-\(p\) sparse attention can suffer from non-negligible mask-prediction cost and from suboptimal shared thresholds across heterogeneous heads, which motivates its plug-in head-wise control layer [2605.14513]. DynamicRad distinguishes a speed-optimized static-ratio mode from a quality-first dynamic-threshold mode, making the speed–quality trade-off an explicit part of the module definition [2604.20470].

Within the Tactic formulation, the preferred regime is therefore clear from the supplied description: long contexts, decode-time KV loading that dominates latency, accuracy-sensitive applications, and a requirement for a dynamic sparse method that does not depend on calibration. The examples listed are long-document QA, long-context reasoning, summarization or generation over 32K–512K contexts, and serving scenarios where full attention is too slow but fixed budgets are too brittle [2502.12216].

Source: https://www.emergentmind.com/topics/adaptive-sparse-attention-module-asam