Query-Aware Native Sparse Attention

• The paper highlights that query-aware native sparse attention selects top-scoring keys based on query similarity, leading to efficient long-context processing.
• It details a three-branch architecture combining compression, selection, and sliding window mechanisms to balance global and local context.
• Empirical results show that native training with Top-k sparse attention can outperform full dense attention on long-context benchmarks.

Query-aware native sparse attention is a family of attention mechanisms in which the active attention support is determined by the current query rather than by a fixed sparsity pattern, while the sparse operator itself is part of the model’s native training and inference regime. In the formulation emphasized by "A Preliminary Study on the Promises and Challenges of Native Top-$k$ Sparse Attention" (Xiu et al., 3 Dec 2025), this means that each query attends only to the keys with highest query–key similarity, so the retained context is query-dependent by construction. More broadly, the literature on Native Sparse Attention (NSA) frames this design as a hardware-aligned, natively trainable alternative to dense full attention: it preserves direct access to past key–value states, but reduces effective attention work through query-conditioned compression, selection, and local windows (Yuan et al., 16 Feb 2025). Subsequent work extends the idea to hybrid linear-attention systems, tabular learning, video-LLMs, offload-aware decoding, and layerwise local–global alternation, while also clarifying which neighboring methods are only loosely related because they are content-aware rather than query-aware in the strict query–key sense (He et al., 23 Oct 2025, Eslamian et al., 12 Mar 2025, Song et al., 2 Oct 2025, Huang et al., 15 Oct 2025, Hu et al., 2 Nov 2025, Shi et al., 4 Aug 2025).

1. Definition and conceptual boundaries

In the strict sense used by the Top-$k$ sparse-attention study, query-aware native sparse attention selects a query-specific retained key set $\mathcal{K}_{top}\subset\mathcal{K}$ of size $\mathcal{W}$, where only the top-scoring keys under query–key similarity are used for that decoding step. The retained fraction is summarized by the Top-$k$ ratio

$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$

A smaller $\rho$ corresponds to sparser and cheaper computation, while $\rho=1$ recovers full attention (Xiu et al., 3 Dec 2025).

The adjective native has a specific technical meaning in this literature. It denotes that the model is trained with the same sparse attention operator that is intended to be used at inference time, rather than applying sparsity only as a post hoc approximation to a dense model. The original NSA paper makes this distinction explicit by contrasting native training with inference-only sparsification and by presenting sparse routing as part of the model architecture during pretraining (Yuan et al., 16 Feb 2025). In this sense, query-aware native sparse attention is not merely an execution-time acceleration trick; it is a learned inductive bias.

The broader NSA family generalizes exact Top-$k$ retrieval into a hierarchical sparse remapping of keys and values. Instead of attending densely over the full prefix, a query interacts with a much smaller set assembled from compression, selection, and sliding-window branches, combined by learned gates. This architecture remains query-aware because the selection path is driven by query-dependent scores, even though some branches, such as the local window, are not themselves adaptive (Yuan et al., 16 Feb 2025).

The boundary of the concept matters. Some recent efficient-attention methods are adjacent but not strictly instances of query-aware native sparse attention. Dynamic Mask Attention, for example, learns sparse masks from value representations and then executes sparse attention efficiently, but its mask is not derived from the query–key interaction; it is therefore better characterized as content-aware native sparse attention with query-dependent execution rather than classic query-aware sparse retrieval (Shi et al., 4 Aug 2025). Sparse Query Attention reduces the number of query heads and thereby sparsifies computation across heads, but it does not sparsify token-to-token connectivity over sequence positions (Filipek, 2 Oct 2025). By contrast, methods such as NSA, native Top-$k$ decoding, and laNSA preserve the defining property that the current query determines which historical tokens or blocks are attended (Yuan et al., 16 Feb 2025, Xiu et al., 3 Dec 2025, He et al., 23 Oct 2025).

2. Core mechanisms and mathematical structure

The canonical NSA formulation begins from dense causal attention

$k$0

and replaces the full key–value history by a query-dependent remapped set

$k$1

The sparse output is then computed as

$k$2

where the branches correspond to compression, selection, and sliding window, and $k$3 are learned gates produced by an MLP plus sigmoid from input features (Yuan et al., 16 Feb 2025).

The compression branch supplies coarse global context by aggregating contiguous tokens into block-level representations. In NSA, keys and values are grouped into overlapping blocks, compressed by an MLP with intra-block positional encoding, and exposed to the query as a shorter sequence of summaries. The selection branch then uses the compressed attention scores to rank fine-grained blocks and retain only the top-$k$4 blocks at full resolution. When compression and selection blocks differ, coarse scores are aggregated into selection-block scores before ranking. The local window branch preserves recent context through

$k$5

This hierarchy gives NSA its characteristic combination of global scan, precise retrieval, and local continuity (Yuan et al., 16 Feb 2025).

The native Top-$k$6 study isolates the query-aware component in its most explicit form. For a decoding query and candidate key set $k$7, the method computes scores to all keys, retains the top $k$8, and discards the rest for that step. The retained keys are exactly those with highest similarity to the current query; the discarded keys contribute nothing to the attention output. The mechanism is therefore sparse over sequence positions, exact with respect to the selected set, and query-dependent at every step (Xiu et al., 3 Dec 2025).

Hybrid systems implement related blockwise query-aware selection. In laNSA, the sequence is partitioned into blocks $k$9, each block is compressed into a summary $\mathcal{K}_{top}\subset\mathcal{K}$0, block relevance is scored by

$\mathcal{K}_{top}\subset\mathcal{K}$1

and the sparse support is the union of tokens in the top-$\mathcal{K}_{top}\subset\mathcal{K}$2 blocks,

$\mathcal{K}_{top}\subset\mathcal{K}$3

The paper uses this blockwise scheme to motivate NSA as a stronger three-branch mechanism that blends selective sparse attention, compressive attention, and sliding-window attention under query-dependent gating (He et al., 23 Oct 2025).

A similar three-branch structure appears in TabNSA, where each feature column is treated as a token and the query influences block compression scores, block selection scores, and the final selected set $\mathcal{K}_{top}\subset\mathcal{K}$4. The sparse output

$\mathcal{K}_{top}\subset\mathcal{K}$5

has the same mixture form, but the domain is heterogeneous tabular data rather than long-text decoding (Eslamian et al., 12 Mar 2025). VideoNSA retains the same native sparse structure for vision tokens while using dense GQA for text tokens, with frame-aligned blocks and a learned gate that routes computation across compression, selection, and sliding-window branches (Song et al., 2 Oct 2025).

3. Native training, decoding regimes, and approximation fidelity

A central result of the Top-$\mathcal{K}_{top}\subset\mathcal{K}$6 study is that exact Top-$\mathcal{K}_{top}\subset\mathcal{K}$7 decoding is often competitive with dense full attention even at low retained ratios. The experiments use Llama-3-8B-ProLong-512k-Instruct and Qwen3-32B across HELMET, MATH500, GSM8K, LongBench v2, and AIME24, and report that on HELMET and LongBench v2, Top-$\mathcal{K}_{top}\subset\mathcal{K}$8 decoding remains highly competitive and can even surpass full attention at low retained ratios (Xiu et al., 3 Dec 2025). The paper’s interpretation is that keeping only the highest-similarity keys can filter distracting context rather than merely approximating dense attention.

The same study argues that training–inference consistency is decisive. To test native Top-$\mathcal{K}_{top}\subset\mathcal{K}$9 training, the authors take Llama-3-8B-ProLong-512k-Base, perform supervised fine-tuning with a Top-$\mathcal{W}$0 attention kernel, precompute exact Top-$\mathcal{W}$1 indices and scores, pass them into the kernel, and update masks accordingly. This produces variable-length Top-$\mathcal{W}$2 attention training at a 1% ratio. The reported result is that the Top-$\mathcal{W}$3 SFT model outperforms the original full-attention Instruct model on downstream tasks, including the 8K HELMET variant (Xiu et al., 3 Dec 2025). The implication drawn in that work is not merely that sparse decoding can be tolerated, but that a model can adapt to and exploit the sparse regime when it is exposed to the same operator during training.

The original NSA paper makes the same point at pretraining scale. NSA is described as natively trainable because sparsity is in the main path from the outset: there is no discrete clustering or hash bottleneck, selection scores are reused from the compression branch, branches are separated to reduce gradient interference, and learned gates combine them smoothly. In the reported 27B-total, 3B-active DeepSeekMoE setup, both Full Attention and NSA are trained to full convergence, with NSA using compression block size $\mathcal{W}$4, stride $\mathcal{W}$5, selection block size $\mathcal{W}$6, selected block count $\mathcal{W}$7, and sliding window $\mathcal{W}$8 (Yuan et al., 16 Feb 2025).

Approximate retrieval introduces a second axis of analysis: fidelity of the sparse selector itself. The Top-$\mathcal{W}$9 study defines retrieval precision as

$k$0

where $k$1 is the set retrieved by an approximate method. The reported observation is a positive correlation between precision and downstream performance, with performance improving as $k$2 increases until saturation (Xiu et al., 3 Dec 2025). In DeepSeek-V3.2-Exp, the Lightning Indexer achieves average precision around 60% on HELMET 128K, yet end-to-end performance remains strong; the study attributes this tolerance partly to model scale and notes that the indexer shares KV tokens across heads, making it closer to an MQA-style approximation of exact Top-$k$3 (Xiu et al., 3 Dec 2025).

Offload-aware sparse attention extends the same training-consistency principle to the memory system. NOSA begins from the observation that trainable sparse attention such as InfLLM-V2 still leaves the full KV cache on GPU. It decomposes selection into a query-aware component $k$4 scored by $k$5 and a query-agnostic component $k$6 scored by a learned function of hidden or value states. The selected set is

$k$7

By construction, this yields a guaranteed overlap lower bound across adjacent decoding steps,

$k$8

thereby increasing cache reuse and reducing CPU–GPU transfer volume without changing the trained sparse computation at inference (Huang et al., 15 Oct 2025). This is still query-aware, but only in part: retrieval power is preserved by the query-aware branch, while offloadability is supplied by the stable query-agnostic backbone.

4. Empirical behavior and theoretical interpretation

The empirical case for query-aware native sparse attention is unusually strong because several papers report that sparse models can equal or exceed dense baselines on long-context tasks rather than merely trading quality for efficiency. In the original NSA work, LongBench average scores are reported as 0.469 for NSA, 0.437 for Full Attention, and 0.423 for Exact-Top, with notable per-task gains of +0.087 on HPQ, +0.051 on 2Wiki, +0.069 on LCC, and +0.075 on PassR-en over Full Attention. On 64k NIAH, NSA achieves perfect retrieval accuracy across all positions (Yuan et al., 16 Feb 2025). The same paper also reports average general-benchmark scores of 0.456 for NSA versus 0.443 for Full Attention, although individual tasks vary.

The preliminary native Top-$k$9 study sharpens the decoding-side claim by showing that exact Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$0 can rival or outperform full attention on HELMET and LongBench v2 and that native Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$1 SFT further improves performance (Xiu et al., 3 Dec 2025). This suggests that query-aware sparsification can function as both an efficiency mechanism and, in some regimes, a useful denoising prior.

The same study offers an entropy-based explanation for when this happens. Comparing Llama-3-8B-ProLong-512k-Instruct and Llama-3-8B-ProLong-512k-TopK-SFT on the 8K HELMET benchmark across all-head versus retrieval-head scopes, mean/min/max/median aggregation, prefilling versus decoding, and token counts of 1, 5, or 10, it reports clear entropy reduction in the Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$2 SFT model (Xiu et al., 3 Dec 2025). The proposed interpretation is that low-entropy states are better adapted to Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$3 decoding because the useful attention mass is more concentrated, allowing a small retrieved subset to preserve most of the signal. This is framed as a conceptual link rather than a full derivation.

Retrieval-oriented hybrid models reinforce the same pattern from a different starting point. In laNSA, the motivation is linear-attention forgetfulness: Gated DeltaNet compresses history into a fixed-size recurrent state, which harms retrieval-intensive tasks. Interleaving GDN with query-aware native sparse attention restores direct access to historical KVs while remaining cheaper than full attention. On EVAPORATE average at 0.4B, the reported scores are 19.47 for GDN, 22.81 for GDN+SWA, 25.26 for laLTE, 25.58 for laNSA, 27.19 for GDN+Attn., and 27.01 for Transf.++. On RULER average at 0.4B, the corresponding scores are 51.7, 54.7, 63.3, 74.8, 80.4, and 75.1 (He et al., 23 Oct 2025). At 1.4B, laNSA remains strong on EVAPORATE average at 33.80, though its RULER average of 77.2 trails both GDN and Transf.++ in the reported table (He et al., 23 Oct 2025). These results show that query-aware sparse retrieval is especially valuable in retrieval-heavy settings, but does not universally dominate every architecture-scale combination.

Complementary evidence comes from memory-augmented backbones rather than new sparse selectors. RAT+ with exponentially decaying memory improves the accuracy of existing query-aware sparse inference methods—Quest, MoBA, and SnapKV—under the same sparse budgets on eight RULER needle-in-a-haystack tasks. The paper argues for two mechanisms: improved critical-token selection and improved information propagation into the selected states. Supporting experiments include higher head-level hit rates and large gains even under random selection, such as Quest MK-1 improving from 53.4 to 84.9 and Quest MV from 39.6 to 74.4 at $$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$4 budget on OLMo2-7B (Wei et al., 27 May 2026). Although this work does not evaluate NSA directly, it suggests that the quality of sparse retrieval depends not only on the selector but also on the information content of the underlying KV states.

5. Architectural variants and domain-specific adaptations

The literature has diversified the basic query-aware native sparse-attention idea along at least four axes: hybridization with other sequence models, layerwise organization, adaptation to new modalities, and memory/offloading constraints.

Hybridization with linear attention is exemplified by laNSA. Here, NSA is interleaved with Gated DeltaNet in a 1:1 ratio, placing query-aware sparse retrieval between pure recurrent compression and full Transformer attention. The intended role is precise: GDN supplies efficient sequence modeling, while NSA reopens access to exact past information when the query requires it (He et al., 23 Oct 2025). This places query-aware sparse attention as a corrective to forgetfulness rather than as a wholesale replacement for the backbone.

Layerwise organization is the focus of Alternating Sparse Attention. ASA starts from NSA’s three-branch decomposition but argues that using the same local–global mixture in every layer is suboptimal. It alternates sliding-window layers with compressed/selective layers, while replacing GQA-based branches with latent-attention variants: Multi-head Latent Attention for the sliding-window branch and Grouped-head Latent Attention for the compressed/selective branches. The paper reports that this reduces KV-cache storage by 50% versus NSA and improves common-sense reasoning and long-text understanding on models from 340M to 1.3B trained on 15B and 100B tokens (Hu et al., 2 Nov 2025). Its ablations further claim that sliding-window attention mainly drives common-sense reasoning, selective attention is essential for in-context retrieval, and compressed attention mostly acts as cheap global routing (Hu et al., 2 Nov 2025).

TabNSA transfers NSA to heterogeneous tabular learning by treating each feature as a token and learning query-dependent sparse interactions across feature blocks. The framework combines a TabMixer backbone with an NSA module and reports first-place ranking across eight binary classification datasets, including scores of 0.874 on Credit-g, 0.967 on Adult, 0.979 on Insurance-Co, and 0.967 on 1995-Income. On ForestCoverType, it reports ACC 92.89 and F1 82.66, while noting more mixed regression behavior, including RMSE 0.0273 on Topo-2-1 and 14.2931 on CPU-Predict (Eslamian et al., 12 Mar 2025). In this setting, query-aware native sparse attention is reinterpreted as dynamic focus on relevant feature subsets per instance.

VideoNSA adapts NSA to multimodal long-video understanding by retaining dense GQA for text and applying native sparse attention only to vision tokens. It is trained end-to-end on a 216K filtered video instruction set derived from LLaVA-Video-178K, uses frame-aligned blocks, and studies budget allocation between global sparse blocks and local windows. The reported findings include reliable scaling to 128K tokens, strong performance on LongVideoBench at 60.2, MLVU at 51.8, TimeScope at 83.7, LongTimeScope at 44.4, Tomato at 26.5, and VSIBench at 36.0, together with an overall sink ratio of about 0.3% (Song et al., 2 Oct 2025). The paper also reports that VideoNSA can achieve strong results with only 3.6% of the full attention budget (Song et al., 2 Oct 2025). In this modality, query-aware sparse attention becomes a mechanism for preserving temporally sparse but semantically critical video evidence without resorting to lossy token compression.

A more domain-specific reinterpretation appears in dynamic-query charged-particle reconstruction. The geometry-aware dynamic-query decoder initializes decoder queries from selected encoder-level measurement representations and replaces dense cross-attention with Local Strided Cross-Attention, a geometry-induced sparse support executed with block sparsity. The DQ architecture increases reconstruction efficiency from 94.1% to 98.1% and reduces the fake rate by more than a factor of two relative to the fixed-query baseline, while DQ+LSCA reduces tracking latency from $$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$5 ms to $$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$6 ms and inference memory from 2.2 GB to 0.2 GB in Pix1.0 (Duckett et al., 16 Jun 2026). This is not NSA in the original language-model sense, but it illustrates that query-aware sparsity can emerge from input-conditioned query construction plus structured sparse attention support.

6. Limitations, misconceptions, and open directions

A persistent misconception is that any efficient sparse attention mechanism is automatically query-aware. The cited literature does not support that conflation. Native Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$7, NSA, laNSA, and related block-selection methods are query-aware because the current query determines which keys or blocks are retained (Xiu et al., 3 Dec 2025, Yuan et al., 16 Feb 2025, He et al., 23 Oct 2025). Dynamic Mask Attention is adaptive, trainable, and native, but its sparse mask is generated from value-side content rather than query–key similarity, so it is not query-aware in the standard sense (Shi et al., 4 Aug 2025). Sparse Query Attention reduces query-head count and therefore FLOPs, but it leaves full tokenwise connectivity intact and is best understood as head sparsification within exact attention (Filipek, 2 Oct 2025).

A second misconception is that sparse attention quality is determined solely by the sparsity pattern. Several papers suggest a more conditional picture. The Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$8 study reports that approximation fidelity correlates positively with downstream performance (Xiu et al., 3 Dec 2025). RAT+ shows that upstream memory augmentation can markedly improve sparse-inference performance even when the selector is unchanged (Wei et al., 27 May 2026). NOSA shows that systems constraints such as KV offloading can be addressed only if the selection pattern is trained to exhibit sufficient temporal locality (Huang et al., 15 Oct 2025). These findings indicate that selector design, backbone state quality, and hardware realization are interdependent.

Practical limitations remain explicit across the literature. Exact Top-$$\rho = \frac{|\mathcal{K}_{top}|}{|\mathcal{K}|} \in (0,1].$$9 attention is computationally expensive, and standard tiled kernels such as FlashAttention do not naturally expose all attention scores needed for exact Top-$\rho$0 selection (Xiu et al., 3 Dec 2025). NSA still requires retaining the full past KV history and is therefore less memory-frugal than linear attention, even if its effective attention work is much lower (He et al., 23 Oct 2025). NOSA improves offloading compatibility, but only by sacrificing part of the budget to query-agnostic selection and by engineering for CPU–GPU transfer locality (Huang et al., 15 Oct 2025). VideoNSA’s performance depends strongly on global–local budget allocation, and LSCA in tracking trades some fidelity for speed relative to learned mask-gated dense cross-attention (Song et al., 2 Oct 2025, Duckett et al., 16 Jun 2026).

The open technical direction suggested by these papers is not a single unified replacement for dense attention, but a design space in which sparse attention is treated as a native model property. This suggests that future progress is likely to come from better alignment of training and inference operators, higher-fidelity approximate retrieval, improved sparse kernels, stronger memory-augmented backbones, and more explicit allocation of local versus global roles across depth. What the current literature already establishes is narrower but significant: when sparsity is query-dependent, learned natively, and implemented in a hardware-conscious way, it can preserve or improve long-context behavior while materially reducing the effective cost of attention (Yuan et al., 16 Feb 2025, Xiu et al., 3 Dec 2025, Hu et al., 2 Nov 2025).