---
title: Sparse Query Attention (SQA) Explained
url: https://www.emergentmind.com/topics/sparse-query-attention-sqa
type: topic
---

# Sparse Query Attention (SQA) Explained

In the literature surveyed here, Sparse Query Attention (SQA) is used for a family of sparse attention mechanisms in which the query is the driver of sparsity. In one narrow and explicit formulation, SQA reduces the number of query heads and thereby reduces the floating-point computation of attention by a factor proportional to the reduction in query heads [2510.01817]. In broader usage, SQA-like methods keep the query side fully present but make the selection of keys or values sparse and adaptive, for example by learning a query-conditioned sparse block mask, selecting representative queries that determine a reduced KV set, or using query magnitudes or query clusters to choose sparse dimensions, tokens, or blocks [2506.08889]. This suggests that SQA is best understood as a technical family centered on query-conditioned sparsity rather than as a single universally fixed operator.

## 1. Definitions and terminology

The most explicit use of the name appears in “Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction” [2510.01817]. There, the defining operation is architectural: reduce the number of query heads, optionally also reduce the number of key/value heads, and keep the rest of the Transformer block unchanged. The paper frames this as complementary to Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), which mainly optimize memory bandwidth and KV-cache size rather than the FLOPs of dense attention-score computation [2510.01817].

Other papers use the term more broadly or describe their method as “very close in spirit” to SQA. In “SeerAttention-R” [2506.08889], the query remains dense, but each decoding step predicts which KV blocks are most relevant and computes attention only on those blocks. “SpotAttention” [2606.22874] similarly attaches a lightweight selector to a frozen pretrained transformer and learns a query-dependent distribution over past blocks. “STS” [2605.15508] constructs a token-and-head-wise sparsity mask for the target model from a draft model’s attention patterns. “QUOKA” [2602.08722] selects a small set of representative queries and then uses those queries to subselect the most relevant KV pairs for chunked prefill.

| Use of “SQA” | Mechanism | Representative papers |
|---|---|---|
| **Query-head reduction** | Reduce \(H_q\) relative to \(H\) | [2510.01817] |
| **Query-conditioned sparse KV routing** | Select sparse blocks/tokens for each query | [2506.08889], [2606.22874], [2605.15508], [2604.08584] |
| **Query or feature sparsification** | Select sparse queries or sparse query dimensions/features | [2107.08687], [2509.11155], [2603.22300] |

A recurrent caveat across this literature is that “sparse query” does not necessarily mean dropping query tokens. Several papers explicitly state that the query remains dense while the attention pattern becomes sparse over selected keys, values, or blocks [2606.22874].

## 2. Canonical SQA as query-head reduction

The direct SQA formulation begins from standard scaled dot-product attention,
\[
\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V,
\]
and standard multi-head attention,
\[
\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O,
\]
with
\[
\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V).
\]
The SQA paper then defines \(H\) as the total number of heads in a comparable MHA baseline, \(H_q\) as the number of query heads, and \(H_{kv}\) as the number of key/value heads, with \(1 \le H_{kv} \le H_q < H\). Queries, keys, and values are projected to
\[
Q \in \mathbb{R}^{N \times H_q \times d_{\text{head}}}, \quad
K,V \in \mathbb{R}^{N \times H_{kv} \times d_{\text{head}}},
\]
after which \(K\) and \(V\) are repeated by factor \(G = H_q/H_{kv}\) so that each of the \(H_q\) query heads can attend to matching key/value heads. The final attention output is
\[
\text{SQA}(X) = \text{Concat}(\text{head}_1, \dots, \text{head}_{H_q})W^O.
\]
Because score computation and value aggregation are both performed per query head, the attention compute scales as
\[
O(H_q \cdot N^2 \cdot d_{\text{head}}),
\]
rather than
\[
O(H \cdot N^2 \cdot d_{\text{head}}),
\]
giving a theoretical speed-up of
\[
\frac{H}{H_q}.
\]
This is the central mathematical claim of the paper [2510.01817].

The paper defines a family of variants. **Standard SQA** allows independent choices of \(H_q\) and \(H_{kv}\). **Symmetric SQA (sSQA)** sets \(H_q = H_{kv} = H/2\), yielding a clean theoretical 2x compute reduction relative to MHA. **Extreme SQA (xSQA)** uses a more aggressive regime, typically \(H_q \le H/4\), to explore the lower bound of query capacity before quality degrades more strongly [2510.01817].

This formulation is positioned against MHA, MQA, and GQA. MHA has full expressivity and the highest compute and memory cost. MQA shares one K and one V head across all query heads and greatly reduces KV-cache size, but it does not reduce the number of query heads and therefore does not reduce the FLOPs of dense attention-score computation in compute-bound settings. GQA shares K/V heads across groups of query heads and is presented as a compromise between MHA and MQA, but again mainly reduces memory bandwidth rather than attention FLOPs [2510.01817].

The long-sequence benchmark in the SQA paper makes this distinction concrete. On a single NVIDIA A100 40GB GPU, at sequence length **200,000**, the reported time per forward step is **1.4116 s** for SQA, **1.4824 s** for sSQA, **0.8194 s** for xSQA, **2.8596 s** for GQA, and **2.8734 s** for MHA. The same paper reports “up to 3x” throughput improvements in computation-bound scenarios such as model pre-training, fine-tuning, and encoder-based tasks [2510.01817].

## 3. Query-conditioned sparse KV routing

A large portion of the recent literature treats SQA not as query-head reduction, but as query-conditioned sparse routing over keys and values. The common pattern is to keep the current query and replace dense attention over the full KV set with sparse attention over a selected subset.

“SeerAttention-R” [2506.08889] is a sparse attention framework specifically tailored for long decoding of reasoning models. It removes query pooling to accommodate autoregressive decoding, keeps a block-sparse learned gating module, and uses a design aligned with GQA to share sparsity decisions within query groups. The query side is still fully present, but the selection of which KV blocks to attend to is sparse and adaptive. The self-distilled Attention Gate is trained while the original pretrained model is kept frozen, and inference converts soft gate scores into binary block masks by either token budget / Top-k or thresholding. The paper reports that, with a **4K token budget**, SeerAttention-R is nearly lossless on AIME benchmarks and that its TileLang sparse decoding kernel achieves near-theoretical speedups of up to **9x** over FlashAttention-3 on H100 GPU at **90% sparsity** [2506.08889].

“SpotAttention” [2606.22874] attaches a lightweight selector to every full-attention layer of a frozen pretrained backbone and trains only the selector by forward KL distillation against the backbone’s dense attention. The selector estimates an attention distribution over past blocks/keys for each query and layer, and the resulting distribution is used for either static top-\(K\) selection or a dynamic dual top-\(p\) rule with sink and recency blocks reserved. The budget therefore varies per query and per layer. On Qwen3-8B at \(L=128\)K, the paper reports **3.9x faster than FlashAttention** and **1.8x faster than Twilight**, while matching dense accuracy up to **128K tokens** and generalizing from **16K training length** to **128K evaluation length** [2606.22874].

“STS” [2605.15508] is training-free and integrates into speculative decoding. The small draft model produces both candidate next tokens and attention weights; STS repurposes those attention weights to build a token-and-head-wise sparsity mask for the large target model. The target model then performs sparse attention using that mask. The paper reports **2.67× speedup** at about **90% sparsity** on **NarrativeQA** with negligible accuracy degradation compared to dense attention, and **>98% recall** on passkey retrieval even with a budget of **256 tokens out of 10K** [2605.15508].

“CSAttention” [2604.08584] is a query-centric sparse attention method for reusable long prompts. During offline prefill it clusters queries in subspaces, precomputes centroid-to-key scores, and stores bounded-capacity Top-\(L\) tables. During decode it matches the current query to one centroid per subspace, unions the retrieved lists, reduces by key, selects Top-\(K\), and runs ordinary attention only on the selected keys. The paper reports near-identical accuracy to full attention under **95% sparsity** and long-context settings **32K-128K**, with up to **4.6x inference speedup** over the most accurate baseline at **128K** [2604.08584].

“QUOKA” [2602.08722] is explicitly query-oriented. Under chunked prefill it first retains a small set of representative queries, chosen because low cosine similarity with respect to the mean query is associated with stronger interaction with more keys, and then subselects the keys most aligned with those queries. The paper reports a **3x reduction in time-to-first-token**, **5x speedup in attention on Nvidia GPUs**, up to nearly a **7x speedup on Intel Xeon CPUs**, and **88% fewer key-value pairs per attention evaluation** while achieving near-baseline accuracy [2602.08722].

Across these methods, the core commonality is query-conditioned sparse selection. The differences lie in where the sparsity is represented: learned block masks, draft-guided token masks, query-centric lookup tables, or representative-query scoring.

## 4. Query selection, feature sparsity, and dimension sparsity

Another branch of the literature makes the query itself sparse, or uses query statistics to sparsify other axes of attention.

“Query Selector” [2107.08687] is a deterministic sparse-attention mechanism for long-term time-series forecasting. For input length \(L\) and sparsity factor \(0<f<1\), it keeps
\[
\ell = \lfloor (1-f)L \rfloor
\]
queries, scores queries by similarity to a deterministic summary key \(\hat K\), computes full attention only for the selected queries, and fills the remaining positions with the mean of the value vectors. The paper presents this as a concrete version of sparse query attention because sparsity is applied to the query dimension rather than to keys or values, and reports state-of-the-art results on **ETT**, **Helpdesk**, and **BPI’12** [2107.08687].

“AQUA” [2509.11155] makes the query drive sparsity over feature dimensions. After an offline SVD-based calibration step that produces an orthogonal projection matrix \(P\), the online phase projects query and key vectors into the calibrated basis,
\[
\hat q_i = q_i P, \qquad \hat k_i = k_i P,
\]
forms the query magnitude vector
\[
v_{\text{mag}} = |\hat q_i|,
\]
and selects the sparse dimension set by
\[
I_{\text{top}k} = \arg\text{TopK}(v_{\text{mag}}).
\]
The paper states that with \(k_{\text{ratio}} = 0.75\), AQUA achieves a **25% reduction in attention dot-product computation** with a statistically insignificant impact on performance on Llama-3.1-8B-Instruct, and also introduces AQUA-Memory as a direct KV-cache memory reduction variant [2509.11155]. This is SQA-like in the sense that the query’s own magnitudes determine which dimensions are kept.

“Scaling Attention via Feature Sparsity” [2603.22300] moves further along the feature axis. Sparse Feature Attention (SFA) applies row-wise Top-\(k\) to both queries and keys,
\[
Q = \mathrm{Topk}_k(Q), \qquad K = \mathrm{Topk}_k(K),
\]
so each token becomes a \(k\)-sparse code in the original ambient dimension \(d\). The score between query token \(i\) and key token \(j\) then depends only on overlapping active coordinates:
\[
S_{ij} = \sum_{u\in S_i\cap S_j} q_{i,u}k_{j,u}.
\]
Under the balance assumption used in the paper, attention cost drops from \(\Theta(n^2 d)\) to
\[
\Theta\!\left(\frac{n^2 k^2}{d}\right).
\]
The paper reports speedups of up to **2.5×**, FLOPs and KV-cache reductions of nearly **50%**, and about **41%** KV-cache reduction in inference settings. It also emphasizes that feature sparsity is complementary to token-sparse methods rather than a replacement for them [2603.22300].

A useful boundary case is “Steered Generation via Gradient-Based Optimization on Sparse Query Features” [2605.23040]. That work applies Sparse Autoencoders to attention query activations and performs gradient-based optimization in sparse latent space, but explicitly states that it is **not proposing SQA as a new attention architecture**. Its relevance lies instead in providing a conceptual rationale for treating query activations as a high-fidelity intervention point [2605.23040].

## 5. Historical and domain-specific realizations

Several papers instantiate SQA-like ideas in domain-specific forms that predate, parallel, or extend the more recent long-context LLM literature.

“Long Document Ranking with Query-Directed Sparse Transformer” [2010.12683] replaces fully connected self-attention with a sparse adjacency matrix
\[
A_{\text{QDS}} = A_{\text{local}} \cup A_{\text{sent}} \cup A_{\text{query}} \cup A_{\text{[CLS]}},
\]
combining local contextualization, sentence-level global attention via \([SOS]\) markers, query-directed global attention, and \([CLS]\) global attention. The query-directed component makes query tokens global so they can gather information from the whole document. The paper reports **NDCG@10 = 0.667** on TREC-19 DL and states that its sparse attention with TVM implementation is twice more efficient than the fully-connected self-attention [2010.12683].

In dense vision, “Sparse Spatial Attention Network for Semantic Segmentation” [2109.01915] introduces a sparse non-local block that samples only **K sampled key/value elements** for each query position, rather than forming a dense \(N \times N\) affinity matrix. The paper gives dense complexity
\[
\mathcal{O}(N^2 C)
\]
and sparse complexity
\[
\mathcal{O}(N K C),
\]
with \(K \ll N\). On Cityscapes validation, the reported inference-time comparison is **42 ms** and **73.75%** for the standard NL block versus **39 ms** and **74.51%** for SNL. The paper also reports **81.8% mIoU** on Cityscapes test, **54.9% mIoU** on PASCAL Context val, and **45.50% mIoU** on ADE20K val [2109.01915]. Although the paper uses non-local and spatial-attention terminology rather than transformer terminology, it can be read as an early vision-oriented form of sparse query attention.

In scientific reconstruction, “Better Queries, Cheaper Attention” [2606.17631] replaces a fixed learned query bank with event-conditioned dynamic queries initialized from selected encoder-level measurement embeddings. The number of dynamic queries varies by event and has a strong linear correlation with the number of reconstructable particles, with \(R^2 = 0.98\), mean dynamic queries \(1400 \pm 200\), and max dynamic queries **2000** in the nominal pixel configuration. Local Strided Cross-Attention (LSCA) then restricts each query to a local diagonal band of hits, implemented with PyTorch FlexAttention as a BlockMask. The headline efficiency results include nearly **50% lower end-to-end latency** and more than **10× reduction in peak allocated inference memory** relative to the fixed-query baseline [2606.17631].

In video generation, “AdaCluster” [2604.18348] presents a training-free adaptive clustering framework for video diffusion transformers. Queries are clustered after normalization by an angle-similarity-preserving method, keys by a euclidean-similarity-preserving method, and critical clusters are selected with a Quest-style upper-bound scoring method reformulated as TensorQuest. The final attention is computed only for a sparse subset of selected key/value entries, guided by query-aware cluster selection. On one A40 GPU, the paper reports speedups of **1.67×** on CogVideoX-2B, **1.85×** on Wan-2.1-1.3B, **1.68×** on HunyuanVideo, and up to **4.31×** at **176.4K tokens** with negligible quality degradation [2604.18348].

These cases show that SQA-like mechanisms are not confined to decoder-only language modeling. The same basic idea—make the query determine a sparse support over attention targets—appears in document ranking, semantic segmentation, scientific tracking, and video diffusion.

## 6. Empirical trade-offs, misconceptions, and adjacent directions

A central misconception is that SQA is interchangeable with MQA or GQA. The direct SQA paper argues the opposite: MQA and GQA mainly optimize the memory bandwidth bottleneck that dominates autoregressive inference latency, whereas SQA reduces the number of query heads and directly decreases the FLOPs of attention score computation. The paper therefore positions SQA as especially useful for compute-bound settings such as pre-training, fine-tuning, encoder-only models, and prompt processing for long-context decoders, while noting that it is less directly beneficial in pure token-by-token decoding unless \(H_{kv}\) is chosen carefully [2510.01817].

A second misconception is that “sparse query” always means pruning queries themselves. Many of the strongest recent methods do not do that. SeerAttention-R explicitly says that it is **not a pure query-sparsification method in the sense of pruning query tokens themselves**; STS says it **does not** prune queries themselves but instead sparsifies the attention pattern per query head; SpotAttention and CSAttention both estimate or retrieve sparse key sets for each query while keeping the query side dense [2506.08889]. A plausible implication is that the phrase SQA has become overloaded: it may refer to query-head reduction, sparse query token selection, or dense queries with sparse query-conditioned KV routing.

A third misconception is that SQA is always training-free. Training-free and plug-in methods are prominent—STS, QUOKA, CSAttention, AQUA, and AdaCluster all emphasize training-free usage—but other work argues that sparse attention should be trained to be intrinsically sparse. “SSA” [2511.20102] identifies a “gradient update deficiency” in native sparse-attention training, proposes a dual-stream full/sparse training framework with bidirectional alignment at every layer, and reports that SSA achieves the highest attention sparsity among the compared methods while remaining strong under both sparse and full attention inference. The same paper also argues that native sparse-attention training improves long-context extrapolation by mitigating sink-area over-allocation [2511.20102].

Related work also shows that SQA-like sparse inference can benefit from upstream architectural changes. “Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity” [2605.28640] studies Quest, MoBA, and SnapKV under RAT+, a recurrence-augmented attention backbone with exponentially decaying memory. The paper reports that RAT+ consistently improves all three methods over standard attention across eight needle-in-a-haystack tasks, including average gains of **+34.11 points at \(1/4\)** and **+40.03 points at \(1/8\)** for SnapKV. Its practical lesson is explicit: query-aware sparse attention works better when the backbone already carries exponentially decaying memory [2605.28640].

Taken together, these results define the present technical profile of SQA. The family includes architectural query-head reduction, query-conditioned routing over blocks or tokens, deterministic query subselection, query-magnitude dimension selection, and feature-level sparsity. Some variants mainly target prefill or training FLOPs, some mainly target decode-time KV access, and some are tied to reusable prompts or to structured geometry. The unifying principle is that the query, or a query-derived proxy, determines a sparse support over the computation that dense attention would otherwise perform exhaustively.

Source: https://www.emergentmind.com/topics/sparse-query-attention-sqa