Sparse Query Attention (SQA) Explained

• SQA is a family of sparse attention mechanisms that drive efficiency by reducing query heads or selectively routing keys/values.
• It utilizes methods like query-head reduction and adaptive sparse masking to lower FLOPs, achieving speedups up to 3× in compute-bound scenarios.
• Variants preserve dense query representations while sparsifying key-value access, thereby maintaining accuracy with optimized computational cost.

In the literature surveyed here, Sparse Query Attention (SQA) is used for a family of sparse attention mechanisms in which the query is the driver of sparsity. In one narrow and explicit formulation, SQA reduces the number of query heads and thereby reduces the floating-point computation of attention by a factor proportional to the reduction in query heads (Filipek, 2 Oct 2025). In broader usage, SQA-like methods keep the query side fully present but make the selection of keys or values sparse and adaptive, for example by learning a query-conditioned sparse block mask, selecting representative queries that determine a reduced KV set, or using query magnitudes or query clusters to choose sparse dimensions, tokens, or blocks (Gao et al., 10 Jun 2025). This suggests that SQA is best understood as a technical family centered on query-conditioned sparsity rather than as a single universally fixed operator.

1. Definitions and terminology

The most explicit use of the name appears in “Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction” (Filipek, 2 Oct 2025). There, the defining operation is architectural: reduce the number of query heads, optionally also reduce the number of key/value heads, and keep the rest of the Transformer block unchanged. The paper frames this as complementary to Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), which mainly optimize memory bandwidth and KV-cache size rather than the FLOPs of dense attention-score computation (Filipek, 2 Oct 2025).

Other papers use the term more broadly or describe their method as “very close in spirit” to SQA. In “SeerAttention-R” (Gao et al., 10 Jun 2025), the query remains dense, but each decoding step predicts which KV blocks are most relevant and computes attention only on those blocks. “SpotAttention” (Ahmad et al., 22 Jun 2026) similarly attaches a lightweight selector to a frozen pretrained transformer and learns a query-dependent distribution over past blocks. “STS” (Xu et al., 15 May 2026) constructs a token-and-head-wise sparsity mask for the target model from a draft model’s attention patterns. “QUOKA” (Jones et al., 9 Feb 2026) selects a small set of representative queries and then uses those queries to subselect the most relevant KV pairs for chunked prefill.

Use of “SQA”	Mechanism	Representative papers
Query-head reduction	Reduce $H_q$ relative to $H$	(Filipek, 2 Oct 2025)
Query-conditioned sparse KV routing	Select sparse blocks/tokens for each query	(Gao et al., 10 Jun 2025, Ahmad et al., 22 Jun 2026, Xu et al., 15 May 2026, Song et al., 30 Mar 2026)
Query or feature sparsification	Select sparse queries or sparse query dimensions/features	(Klimek et al., 2021, S et al., 14 Sep 2025, Xie et al., 17 Mar 2026)

A recurrent caveat across this literature is that “sparse query” does not necessarily mean dropping query tokens. Several papers explicitly state that the query remains dense while the attention pattern becomes sparse over selected keys, values, or blocks (Ahmad et al., 22 Jun 2026).

2. Canonical SQA as query-head reduction

The direct SQA formulation begins from standard scaled dot-product attention,

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$

and standard multi-head attention,

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$

with

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$

The SQA paper then defines $H$ as the total number of heads in a comparable MHA baseline, $H_q$ as the number of query heads, and $H_{kv}$ as the number of key/value heads, with $1 \le H_{kv} \le H_q < H$. Queries, keys, and values are projected to

$$Q \in \mathbb{R}^{N \times H_q \times d_{\text{head}}}, \quad K,V \in \mathbb{R}^{N \times H_{kv} \times d_{\text{head}}},$$

after which $H$0 and $H$1 are repeated by factor $H$2 so that each of the $H$3 query heads can attend to matching key/value heads. The final attention output is

$H$4

Because score computation and value aggregation are both performed per query head, the attention compute scales as

$H$5

rather than

$H$6

giving a theoretical speed-up of

$H$7

This is the central mathematical claim of the paper (Filipek, 2 Oct 2025).

The paper defines a family of variants. Standard SQA allows independent choices of $H$8 and $H$9. Symmetric SQA (sSQA) sets $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$0, yielding a clean theoretical 2x compute reduction relative to MHA. Extreme SQA (xSQA) uses a more aggressive regime, typically $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$1, to explore the lower bound of query capacity before quality degrades more strongly (Filipek, 2 Oct 2025).

This formulation is positioned against MHA, MQA, and GQA. MHA has full expressivity and the highest compute and memory cost. MQA shares one K and one V head across all query heads and greatly reduces KV-cache size, but it does not reduce the number of query heads and therefore does not reduce the FLOPs of dense attention-score computation in compute-bound settings. GQA shares K/V heads across groups of query heads and is presented as a compromise between MHA and MQA, but again mainly reduces memory bandwidth rather than attention FLOPs (Filipek, 2 Oct 2025).

The long-sequence benchmark in the SQA paper makes this distinction concrete. On a single NVIDIA A100 40GB GPU, at sequence length 200,000, the reported time per forward step is 1.4116 s for SQA, 1.4824 s for sSQA, 0.8194 s for xSQA, 2.8596 s for GQA, and 2.8734 s for MHA. The same paper reports “up to 3x” throughput improvements in computation-bound scenarios such as model pre-training, fine-tuning, and encoder-based tasks (Filipek, 2 Oct 2025).

3. Query-conditioned sparse KV routing

A large portion of the recent literature treats SQA not as query-head reduction, but as query-conditioned sparse routing over keys and values. The common pattern is to keep the current query and replace dense attention over the full KV set with sparse attention over a selected subset.

“SeerAttention-R” (Gao et al., 10 Jun 2025) is a sparse attention framework specifically tailored for long decoding of reasoning models. It removes query pooling to accommodate autoregressive decoding, keeps a block-sparse learned gating module, and uses a design aligned with GQA to share sparsity decisions within query groups. The query side is still fully present, but the selection of which KV blocks to attend to is sparse and adaptive. The self-distilled Attention Gate is trained while the original pretrained model is kept frozen, and inference converts soft gate scores into binary block masks by either token budget / Top-k or thresholding. The paper reports that, with a 4K token budget, SeerAttention-R is nearly lossless on AIME benchmarks and that its TileLang sparse decoding kernel achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity (Gao et al., 10 Jun 2025).

“SpotAttention” (Ahmad et al., 22 Jun 2026) attaches a lightweight selector to every full-attention layer of a frozen pretrained backbone and trains only the selector by forward KL distillation against the backbone’s dense attention. The selector estimates an attention distribution over past blocks/keys for each query and layer, and the resulting distribution is used for either static top-$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$2 selection or a dynamic dual top-$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$3 rule with sink and recency blocks reserved. The budget therefore varies per query and per layer. On Qwen3-8B at $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$4K, the paper reports 3.9x faster than FlashAttention and 1.8x faster than Twilight, while matching dense accuracy up to 128K tokens and generalizing from 16K training length to 128K evaluation length (Ahmad et al., 22 Jun 2026).

“STS” (Xu et al., 15 May 2026) is training-free and integrates into speculative decoding. The small draft model produces both candidate next tokens and attention weights; STS repurposes those attention weights to build a token-and-head-wise sparsity mask for the large target model. The target model then performs sparse attention using that mask. The paper reports 2.67× speedup at about 90% sparsity on NarrativeQA with negligible accuracy degradation compared to dense attention, and >98% recall on passkey retrieval even with a budget of 256 tokens out of 10K (Xu et al., 15 May 2026).

“CSAttention” (Song et al., 30 Mar 2026) is a query-centric sparse attention method for reusable long prompts. During offline prefill it clusters queries in subspaces, precomputes centroid-to-key scores, and stores bounded-capacity Top-$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$5 tables. During decode it matches the current query to one centroid per subspace, unions the retrieved lists, reduces by key, selects Top-$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$6, and runs ordinary attention only on the selected keys. The paper reports near-identical accuracy to full attention under 95% sparsity and long-context settings 32K-128K, with up to 4.6x inference speedup over the most accurate baseline at 128K (Song et al., 30 Mar 2026).

“QUOKA” (Jones et al., 9 Feb 2026) is explicitly query-oriented. Under chunked prefill it first retains a small set of representative queries, chosen because low cosine similarity with respect to the mean query is associated with stronger interaction with more keys, and then subselects the keys most aligned with those queries. The paper reports a 3x reduction in time-to-first-token, 5x speedup in attention on Nvidia GPUs, up to nearly a 7x speedup on Intel Xeon CPUs, and 88% fewer key-value pairs per attention evaluation while achieving near-baseline accuracy (Jones et al., 9 Feb 2026).

Across these methods, the core commonality is query-conditioned sparse selection. The differences lie in where the sparsity is represented: learned block masks, draft-guided token masks, query-centric lookup tables, or representative-query scoring.

4. Query selection, feature sparsity, and dimension sparsity

Another branch of the literature makes the query itself sparse, or uses query statistics to sparsify other axes of attention.

“Query Selector” (Klimek et al., 2021) is a deterministic sparse-attention mechanism for long-term time-series forecasting. For input length $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$7 and sparsity factor $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$8, it keeps

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,$$9

queries, scores queries by similarity to a deterministic summary key $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$0, computes full attention only for the selected queries, and fills the remaining positions with the mean of the value vectors. The paper presents this as a concrete version of sparse query attention because sparsity is applied to the query dimension rather than to keys or values, and reports state-of-the-art results on ETT, Helpdesk, and BPI’12 (Klimek et al., 2021).

“AQUA” (S et al., 14 Sep 2025) makes the query drive sparsity over feature dimensions. After an offline SVD-based calibration step that produces an orthogonal projection matrix $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$1, the online phase projects query and key vectors into the calibrated basis,

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$2

forms the query magnitude vector

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$3

and selects the sparse dimension set by

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$4

The paper states that with $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$5, AQUA achieves a 25% reduction in attention dot-product computation with a statistically insignificant impact on performance on Llama-3.1-8B-Instruct, and also introduces AQUA-Memory as a direct KV-cache memory reduction variant (S et al., 14 Sep 2025). This is SQA-like in the sense that the query’s own magnitudes determine which dimensions are kept.

“Scaling Attention via Feature Sparsity” (Xie et al., 17 Mar 2026) moves further along the feature axis. Sparse Feature Attention (SFA) applies row-wise Top-$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$6 to both queries and keys,

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$7

so each token becomes a $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$8-sparse code in the original ambient dimension $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,$$9. The score between query token $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$0 and key token $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$1 then depends only on overlapping active coordinates: $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$2 Under the balance assumption used in the paper, attention cost drops from $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$3 to

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$4

The paper reports speedups of up to 2.5×, FLOPs and KV-cache reductions of nearly 50%, and about 41% KV-cache reduction in inference settings. It also emphasizes that feature sparsity is complementary to token-sparse methods rather than a replacement for them (Xie et al., 17 Mar 2026).

A useful boundary case is “Steered Generation via Gradient-Based Optimization on Sparse Query Features” (Bhattacharyya et al., 21 May 2026). That work applies Sparse Autoencoders to attention query activations and performs gradient-based optimization in sparse latent space, but explicitly states that it is not proposing SQA as a new attention architecture. Its relevance lies instead in providing a conceptual rationale for treating query activations as a high-fidelity intervention point (Bhattacharyya et al., 21 May 2026).

5. Historical and domain-specific realizations

Several papers instantiate SQA-like ideas in domain-specific forms that predate, parallel, or extend the more recent long-context LLM literature.

“Long Document Ranking with Query-Directed Sparse Transformer” (Jiang et al., 2020) replaces fully connected self-attention with a sparse adjacency matrix

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$5

combining local contextualization, sentence-level global attention via $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$6 markers, query-directed global attention, and $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$7 global attention. The query-directed component makes query tokens global so they can gather information from the whole document. The paper reports NDCG@10 = 0.667 on TREC-19 DL and states that its sparse attention with TVM implementation is twice more efficient than the fully-connected self-attention (Jiang et al., 2020).

In dense vision, “Sparse Spatial Attention Network for Semantic Segmentation” (Liu et al., 2021) introduces a sparse non-local block that samples only K sampled key/value elements for each query position, rather than forming a dense $$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$8 affinity matrix. The paper gives dense complexity

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).$$9

and sparse complexity

$H$0

with $H$1. On Cityscapes validation, the reported inference-time comparison is 42 ms and 73.75% for the standard NL block versus 39 ms and 74.51% for SNL. The paper also reports 81.8% mIoU on Cityscapes test, 54.9% mIoU on PASCAL Context val, and 45.50% mIoU on ADE20K val (Liu et al., 2021). Although the paper uses non-local and spatial-attention terminology rather than transformer terminology, it can be read as an early vision-oriented form of sparse query attention.

In scientific reconstruction, “Better Queries, Cheaper Attention” (Duckett et al., 16 Jun 2026) replaces a fixed learned query bank with event-conditioned dynamic queries initialized from selected encoder-level measurement embeddings. The number of dynamic queries varies by event and has a strong linear correlation with the number of reconstructable particles, with $H$2, mean dynamic queries $H$3, and max dynamic queries 2000 in the nominal pixel configuration. Local Strided Cross-Attention (LSCA) then restricts each query to a local diagonal band of hits, implemented with PyTorch FlexAttention as a BlockMask. The headline efficiency results include nearly 50% lower end-to-end latency and more than 10× reduction in peak allocated inference memory relative to the fixed-query baseline (Duckett et al., 16 Jun 2026).

In video generation, “AdaCluster” (Tan et al., 20 Apr 2026) presents a training-free adaptive clustering framework for video diffusion transformers. Queries are clustered after normalization by an angle-similarity-preserving method, keys by a euclidean-similarity-preserving method, and critical clusters are selected with a Quest-style upper-bound scoring method reformulated as TensorQuest. The final attention is computed only for a sparse subset of selected key/value entries, guided by query-aware cluster selection. On one A40 GPU, the paper reports speedups of 1.67× on CogVideoX-2B, 1.85× on Wan-2.1-1.3B, 1.68× on HunyuanVideo, and up to 4.31× at 176.4K tokens with negligible quality degradation (Tan et al., 20 Apr 2026).

These cases show that SQA-like mechanisms are not confined to decoder-only language modeling. The same basic idea—make the query determine a sparse support over attention targets—appears in document ranking, semantic segmentation, scientific tracking, and video diffusion.

6. Empirical trade-offs, misconceptions, and adjacent directions

A central misconception is that SQA is interchangeable with MQA or GQA. The direct SQA paper argues the opposite: MQA and GQA mainly optimize the memory bandwidth bottleneck that dominates autoregressive inference latency, whereas SQA reduces the number of query heads and directly decreases the FLOPs of attention score computation. The paper therefore positions SQA as especially useful for compute-bound settings such as pre-training, fine-tuning, encoder-only models, and prompt processing for long-context decoders, while noting that it is less directly beneficial in pure token-by-token decoding unless $H$4 is chosen carefully (Filipek, 2 Oct 2025).

A second misconception is that “sparse query” always means pruning queries themselves. Many of the strongest recent methods do not do that. SeerAttention-R explicitly says that it is not a pure query-sparsification method in the sense of pruning query tokens themselves; STS says it does not prune queries themselves but instead sparsifies the attention pattern per query head; SpotAttention and CSAttention both estimate or retrieve sparse key sets for each query while keeping the query side dense (Gao et al., 10 Jun 2025). A plausible implication is that the phrase SQA has become overloaded: it may refer to query-head reduction, sparse query token selection, or dense queries with sparse query-conditioned KV routing.

A third misconception is that SQA is always training-free. Training-free and plug-in methods are prominent—STS, QUOKA, CSAttention, AQUA, and AdaCluster all emphasize training-free usage—but other work argues that sparse attention should be trained to be intrinsically sparse. “SSA” (Shen et al., 25 Nov 2025) identifies a “gradient update deficiency” in native sparse-attention training, proposes a dual-stream full/sparse training framework with bidirectional alignment at every layer, and reports that SSA achieves the highest attention sparsity among the compared methods while remaining strong under both sparse and full attention inference. The same paper also argues that native sparse-attention training improves long-context extrapolation by mitigating sink-area over-allocation (Shen et al., 25 Nov 2025).

Related work also shows that SQA-like sparse inference can benefit from upstream architectural changes. “Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity” (Wei et al., 27 May 2026) studies Quest, MoBA, and SnapKV under RAT+, a recurrence-augmented attention backbone with exponentially decaying memory. The paper reports that RAT+ consistently improves all three methods over standard attention across eight needle-in-a-haystack tasks, including average gains of +34.11 points at $H$5 and +40.03 points at $H$6 for SnapKV. Its practical lesson is explicit: query-aware sparse attention works better when the backbone already carries exponentially decaying memory (Wei et al., 27 May 2026).

Taken together, these results define the present technical profile of SQA. The family includes architectural query-head reduction, query-conditioned routing over blocks or tokens, deterministic query subselection, query-magnitude dimension selection, and feature-level sparsity. Some variants mainly target prefill or training FLOPs, some mainly target decode-time KV access, and some are tied to reusable prompts or to structured geometry. The unifying principle is that the query, or a query-derived proxy, determines a sparse support over the computation that dense attention would otherwise perform exhaustively.