SparDA: Sparse Decoupled Attention

Updated 4 July 2026

SparDA is a decoupled sparse attention mechanism that augments Transformers with a fourth 'Forecast' projection for efficient long-context inference.
It predicts future KV cache blocks one layer ahead, overlapping CPU-GPU prefetch with computation to mitigate quadratic selection complexity.
By optimizing sparse selection in GQA settings with reduced FLOPs, SparDA achieves up to 1.7× faster decoding while adding less than 0.5% extra parameters.

SparDA, short for Sparse Decoupled Attention, is a decoupled sparse attention architecture for efficient long-context LLM inference that addresses two bottlenecks left unresolved by prior sparse attention systems: KV cache capacity still grows with sequence length, and the sparse selection step itself can retain $O(T^2)$ complexity and dominate attention cost at long contexts (Fu et al., 3 Jun 2026). Its central mechanism is a fourth per-layer projection, the Forecast, added alongside Query, Key, and Value. The Forecast predicts the KV blocks needed by the next layer, enabling one-layer-ahead selection that overlaps CPU-to-GPU prefetch with current-layer execution, while also reducing selection overhead in GQA settings by using one Forecast head per GQA group (Fu et al., 3 Jun 2026).

1. Motivation and problem setting

SparDA is situated in the regime of long-context Transformer decoding, where the KV cache grows linearly with the decoded sequence length $T$ . For very long contexts, described as $T \gg 10^4$ , even block-sparse attention cannot prevent the cache itself from ballooning on GPU, because sparse attention reduces which entries are attended to, not the storage cost of the cache as such (Fu et al., 3 Jun 2026).

The second difficulty is sparse selection complexity. In block-sparse attention methods such as InfLLM-V2, NSA, and MoBA, the per-token attention cost is reduced to $O(|B|\cdot B_{block}) \ll O(T)$ , but their top- $k$ block-selection step still requires scanning scores over all compressed key blocks. Since the compressed key grid has size $N_b \approx T/B_{block}$ , computing scores for each of $T$ queries against $N_b$ blocks yields $O(T \cdot N_b)=O(T^2)$ in the worst case. As $T$ grows, selection becomes the new bottleneck (Fu et al., 3 Jun 2026).

A third limitation arises when KV entries are offloaded to host memory. Offloading is a natural remedy to GPU KV-cache blow-up, but PCIe host-to-device bandwidth is much lower than on-GPU HBM bandwidth. If each layer stalls waiting for CPU-to-GPU DMA, decoding latency degrades sharply (Fu et al., 3 Jun 2026). The paper places SparDA in relation to InfiniGen, which introduced lookahead prefetch using the previous layer’s hidden state to predict what the next layer would need; SparDA instead treats the prediction of future KV-block access as a trainable signal rather than a training-free heuristic (Fu et al., 3 Jun 2026).

This suggests that SparDA is best understood not merely as a sparse attention variant, but as a reorganization of the memory-access and selection pathway in sparse long-context inference.

2. Architectural formulation

In each Transformer layer $T$ 0, SparDA replaces the usual three-projection map with a four-projection map:

$T$ 1

Here $T$ 2 is the Forecast projection (Fu et al., 3 Jun 2026).

The role of Forecast is distinct from the role of Query. Forecast is used to select the top- $T$ 3 blocks that layer $T$ 4 will attend, while $T$ 5 is used as usual for the sparse attention computation in layer $T$ 6 (Fu et al., 3 Jun 2026). This is the basis for the paper’s use of the term decoupled: the signal used for indexing future blocks is separated from the signal used for current-layer attention.

The one-layer-ahead selection rule is defined using compressed keys $T$ 7, obtained by mean-pooling over each block:

$T$ 8

where $T$ 9 is the set of fixed “initial” blocks, $T \gg 10^4$ 0 denotes sliding-window blocks around each query, and $T \gg 10^4$ 1 returns the indices of the top- $T \gg 10^4$ 2 largest entries per query in a score matrix $T \gg 10^4$ 3 (Fu et al., 3 Jun 2026). The attention computation in layer $T \gg 10^4$ 4 is then

$T \gg 10^4$ 5

A central design consequence is that Forecast is not used for dot-product attention itself. Because of this, SparDA can shrink the selector in GQA architectures to one head per KV head, or equivalently one head per GQA group, rather than keeping the original multi-head selector. The paper describes this as reducing the selection FLOPs by a factor of $T \gg 10^4$ 6 compared to the original selector (Fu et al., 3 Jun 2026).

3. Decoupling, GQA grouping, and complexity

The decoupling argument is most explicit in the GQA setting. In block-sparse attention backbones such as InfLLM-V2, each GQA group $T \gg 10^4$ 7 contains $T \gg 10^4$ 8 query heads that share one KV head. In the original selector, one must score each of the $T \gg 10^4$ 9 query heads, producing $O(|B|\cdot B_{block}) \ll O(T)$ 0 scores per group and incurring expensive $O(|B|\cdot B_{block}) \ll O(T)$ 1 work (Fu et al., 3 Jun 2026).

SparDA changes this by introducing a Forecast projection with $O(|B|\cdot B_{block}) \ll O(T)$ 2 output heads, one per KV head. Since $O(|B|\cdot B_{block}) \ll O(T)$ 3 and $O(|B|\cdot B_{block}) \ll O(T)$ 4, the selector becomes substantially smaller (Fu et al., 3 Jun 2026). The paper attributes two benefits to this change. First, the raw selection FLOPs shrink from the original $O(|B|\cdot B_{block}) \ll O(T)$ 5 form to $O(|B|\cdot B_{block}) \ll O(T)$ 6. Second, because selection is performed one layer ahead, the latency contribution can be overlapped with current-layer computation, yielding an end-to-end latency overhead that approaches $O(|B|\cdot B_{block}) \ll O(T)$ 7 rather than remaining exposed as a quadratic bottleneck (Fu et al., 3 Jun 2026).

The stated complexity comparison is summarized below.

Component	Original sparse selection	SparDA Forecast indexer
Score computation	$O(\|B\|\cdot B_{block}) \ll O(T)$ 8	$O(\|B\|\cdot B_{block}) \ll O(T)$ 9
Relation to context length	$k$ 0 with $k$ 1	Selection FLOPs shrink by $k$ 2
Runtime effect	Selection becomes bottleneck at long context	One layer ahead allows overlap with compute

A plausible implication is that SparDA does not eliminate the need for sparse compression or block structure; rather, it shifts the scaling-critical portion of sparse inference from synchronous query-time selection to an anticipatory indexing stage.

4. Runtime system and overlap of prefetch with execution

SparDA’s runtime exploits the fact that $k$ 3 can be predicted during the execution of layer $k$ 4. The selected KV blocks for the next layer are then prefetched from CPU to GPU asynchronously in parallel with the current layer’s attention and FFN computation (Fu et al., 3 Jun 2026).

The decode-step algorithm given in the paper is:

$k$ 5
Append $k$ 6 to CPU KV cache; update compressed keys $k$ 7
Compute $k$ 8
Issue asynchronous DMA( $k$ 9) on prefetch stream
Wait for DMA( $N_b \approx T/B_{block}$ 0) launched in layer $N_b \approx T/B_{block}$ 1
$N_b \approx T/B_{block}$ 2 (Fu et al., 3 Jun 2026)

The implementation uses a persistent Triton UVA kernel on a dedicated CUDA stream, with a small set of CTAs continuously grabbing new block-copy tasks so as to avoid thousands of small launches (Fu et al., 3 Jun 2026). The paper also notes adaptive CTA allocation per batch size as an implementation detail (Fu et al., 3 Jun 2026).

This execution model is designed to hide host-to-device transfers behind GPU compute. Because the selection and DMA launch for $N_b \approx T/B_{block}$ 3 happen while the kernels for layer $N_b \approx T/B_{block}$ 4 are running, the CPU-to-GPU transfer latency is described as being largely hidden (Fu et al., 3 Jun 2026). In that sense, SparDA is as much a scheduling mechanism as it is an architectural change.

5. Training objective and parameterization

SparDA is integrated into existing sparse-pretrained models by training only the Forecast projections while keeping the backbone frozen. On an 8B model, the Forecast projections add 33.5 M parameters (0.41% of 8 B), and the abstract summarizes the increase as $N_b \approx T/B_{block}$ 5 parameters (Fu et al., 3 Jun 2026).

The target signal for Forecast is derived from the original selector’s attention distribution. Let $N_b \approx T/B_{block}$ 6 denote the ground-truth per-group, per-query importance scores, obtained by summing over the $N_b \approx T/B_{block}$ 7 query heads and softmaxing:

$N_b \approx T/B_{block}$ 8

The predicted score from the previous layer’s Forecast is

$N_b \approx T/B_{block}$ 9

If $T$ 0 is the set of $T$ 1 selected blocks per query, the paper forms two $T$ 2-dimensional distributions by keeping the $T$ 3 in-set scores individually and summing the remaining $T$ 4 scores into a single “rest” bucket, followed by renormalization. The training loss is then

$T$ 5

All backbone weights remain frozen; only the Forecast submatrix is updated (Fu et al., 3 Jun 2026).

The training configuration reported in the paper is 2 K AdamW steps, constant LR $T$ 6, BF16, batch size 32, on ProLong-64 K data (Fu et al., 3 Jun 2026). This narrow adaptation scope is significant because the method is explicitly presented as a retrofit for existing sparse-pretrained models rather than as a new end-to-end pretraining recipe.

6. Experimental evaluation and reported results

The reported experiments use two sparse-pretrained 8B models:

MiniCPM4.1-8B (InfLLM-V2 backbone, prefilled at 32 K) up to 64 K
NOSA-8B (InfLLM-V2 + query-agnostic head, prefilled at 16 K) up to 32 K (Fu et al., 3 Jun 2026)

Benchmarks are HELMET, LongBench, RULER (32 K–128 K sweep), and a long-reasoning suite (MATH-500, AIME 2024/25). Hardware includes NVIDIA H100 (80 GB HBM3, PCIe Gen5×16) and NVIDIA A100 (80 GB HBM2e, PCIe Gen4×16) (Fu et al., 3 Jun 2026).

The selected H100 results reported in the paper are:

Setting	Baselines	SparDA
MiniCPM4.1-8B prefill at 128 K, batch size 4	Dense: 8.09 K; Sparse: 13.66 K; InfiniGen: 13.64 K	17.09 K (1.25× vs. Sparse)
NOSA-8B prefill at 128 K, batch size 4	Dense: 8.12 K; Sparse: 9.81 K; InfiniGen: 9.75 K	11.33 K (1.16× vs. Sparse)
MiniCPM4.1-8B decode at 128 K	447.9 tok/s at $T$ 7	705.3 tok/s, peak speedup 1.69× at $T$ 8
NOSA-8B decode at 128 K	529.2 tok/s at $T$ 9	735.0 tok/s, peak speedup 1.40× at $N_b$ 0

The paper further states that, against the non-offloaded sparse baseline (which OOMs past $N_b$ 1), SparDA runs larger batches → up to 5.28× higher throughput (Fu et al., 3 Jun 2026). The abstract rounds the peak gains as up to $N_b$ 2 prefill speedup, $N_b$ 3 decode speedup over the sparse-attention offload baseline, and up to $N_b$ 4 higher decode throughput than the non-offload sparse baseline (Fu et al., 3 Jun 2026).

Accuracy is reported as an average over HELMET/LongBench/RULER/Reasoning:

MiniCPM4.1-8B: Sparse 61.4 → SparDA 61.7 (+0.3)
NOSA-8B: Sparse 49.4 → SparDA 51.7 (+2.3) (Fu et al., 3 Jun 2026)

The paper characterizes these outcomes by stating that SparDA matches or slightly improves accuracy, whereas InfiniGen suffers up to 6 points loss on some benchmarks (Fu et al., 3 Jun 2026). This suggests that the Forecast projection is not merely a systems optimization; it can also preserve, and in the reported cases slightly improve, the effective quality of sparse selection.

The paper’s conclusion frames SparDA as showing that sparse attention’s memory-access pattern can be “lifted” out of the attention critical path and treated as a schedulable, trainable signal (Fu et al., 3 Jun 2026). In the paper’s own decomposition, the method achieves two effects simultaneously: it hides CPU-to-GPU latency behind GPU compute by predicting next-layer block needs one layer early, and it slashes selection FLOPs by shrinking the indexer via grouping (Fu et al., 3 Jun 2026).

The stated limitations are narrow but important. SparDA builds on a given sparse backbone, so its accuracy ceiling is that of the underlying sparse attention method (Fu et al., 3 Jun 2026). The paper identifies as a natural next step the application of decoupled, one-layer-ahead Forecasting to token-level sparse attention (e.g. DSA) and to larger, more heavily compressed sparse models (e.g. DeepSeek-V4) (Fu et al., 3 Jun 2026). A plausible implication is that the general idea of decoupled forecasting is architecture-agnostic within the broader family of sparse attention systems, even though the current instantiation is explicitly block-sparse and GQA-aware.

A recurrent source of confusion is the similarity of names between SparDA and SPARD. They denote different systems: SparDA is “Sparse Decoupled Attention for Efficient Long-Context LLM Inference” (Fu et al., 3 Jun 2026), whereas SPARD is a defense framework for harmful fine-tuning attacks (Chen et al., 27 May 2026). The overlap is lexical rather than conceptual.

SparDA’s source code is released at https://github.com/NVlabs/SparDA (Fu et al., 3 Jun 2026).