Shared-KV Gated Sliding-Window Attention

• Shared-KV Gated Sliding-Window Attention is a sparse attention design that preserves linear-time efficiency while augmenting sliding-window models with learnable KV sharing and gating.
• It introduces two gating paradigms—token eviction and logit-decay gating—that respectively enable selective retention of distant tokens and stabilize memory dynamics.
• Empirical evaluations show enhanced retrieval performance and throughput on long-context benchmarks, making it a competitive alternative for efficient sequence modeling.

Searching arXiv for the cited works to ground the article in the relevant papers. Found the primary paper on hybrid sparse attention and learnable token eviction. Searching for related Shared-KV gated sliding-window formulations. Shared-KV Gated Sliding-Window Attention denotes a family of sparse-attention constructions that preserve the linear-time, linear-memory profile of sliding-window attention while improving memory access or memory dynamics through KV sharing and gating. In the cited literature, this family appears in at least two technically distinct forms: a hybrid sparse-attention design in which a sliding window is augmented by a learned cache of retained distant KV-pairs selected by a contextualized gating module (He et al., 23 Oct 2025), and a FlashAttention-compatible sliding-window mechanism in which a per-token gate is accumulated into a decay bias on the attention logits, yielding a learnable contraction of the induced associative-memory recurrence (Liu et al., 8 Dec 2025). A related systems-oriented precursor is MixAttention, which combines sliding-window attention with KV-cache sharing across layers to reduce inference memory and improve throughput, and whose summary explicitly treats gating only as an optional extension rather than as part of the base mechanism (Rajput et al., 2024).

1. Problem setting and lineage

The motivating problem is the tension between efficient sequence modeling and direct token retrieval. Pure linear-attention models compress the entire input sequence into a fixed-size recurrent state and therefore operate in constant time and space per token, but their finite memory induces forgetfulness that harms retrieval-intensive tasks (He et al., 23 Oct 2025). Full Softmax attention restores direct access to all past KV-pairs, but incurs $O(N)$ cost per token and $O(N^2)$ cost overall. Sliding-Window Attention (SWA) occupies an intermediate regime: each query attends only to the most recent $w$ tokens, sometimes augmented with a small set of global tokens, so its per-token time and space are $O(w)$, constant when $w$ is fixed (He et al., 23 Oct 2025).

Within this design space, “Shared-KV” addresses a second systems bottleneck: the cost of storing and indexing keys and values. MixAttention formalizes one form of sharing by computing new $K,V$ only in selected “compute-KV” layers and reusing them in other layers, while also restricting attention to a sliding window of size $s$ (Rajput et al., 2024). The later gated variants keep the sliding restriction but add either selective retention of out-of-window KV-pairs or a learnable decay bias inside the window (He et al., 23 Oct 2025, Liu et al., 8 Dec 2025).

This suggests that Shared-KV Gated Sliding-Window Attention is best understood as a design family organized around three recurring objectives: retain SWA’s asymptotic efficiency, reduce KV-cache overhead through sharing or compaction, and recover some of the long-range utility that plain SWA loses once tokens exit the window.

2. Sliding-window attention as the base operator

In the hybrid sparse-attention formulation, let the sequence length be $N$ and let the per-head query, key, and value matrices satisfy $Q,K,V\in\mathbb{R}^{N\times d}$. For head $h$ and position $O(N^2)$0, the sliding-window index set is defined as

$O(N^2)$1

where $O(N^2)$2 is the size of a small attention sink of global tokens (He et al., 23 Oct 2025). The corresponding attention weights and head output are

$O(N^2)$3

In matrix form for a query tile, this is the usual Softmax–MatMul restricted to $O(N^2)$4 (He et al., 23 Oct 2025).

MixAttention adopts the same basic locality principle, but places it in a layer-sharing framework. Its sliding-window index set for token $O(N^2)$5 in layer $O(N^2)$6 is

$O(N^2)$7

and layers outside the compute-KV subset reuse keys and values from an earlier layer $O(N^2)$8 rather than recomputing them (Rajput et al., 2024). Optional gating is described there only as an extension: if used, it mixes local and shared keys and values through headwise scalars $O(N^2)$9 (Rajput et al., 2024).

The technical significance of SWA across these formulations is that it replaces the full past with a bounded local receptive field. All later gating mechanisms operate on top of this bounded substrate rather than replacing it.

3. Two gating paradigms

The first gating paradigm is learnable token eviction. In the hybrid model of (He et al., 23 Oct 2025), once a token leaves the sliding window, a per-token, per-head retention score $w$0 determines whether the KV-pair $w$1 is kept in an out-of-window cache. The module is explicitly fine-grained, contextualized, independent, and lightweight: it makes a separate decision for each token and head; it uses local context from both past and future adjacent tokens via a small CNN; it avoids global top-$w$2 selection by using a sigmoid-plus-threshold decision; and it uses only three small 1D convolutions per head, for approximately $w$3 extra parameters (He et al., 23 Oct 2025). Concretely, the module begins from the pre-RoPE concatenated vector

$w$4

then applies three grouped 1D dilated convolutions with kernel size $w$5 and dilation $w$6, halving channel width each layer, followed by Swish activations and dropout; a final grouped $w$7 convolution maps to a scalar logit $w$8, and

$w$9

Training uses a straight-through estimator, and sparsity is encouraged by a per-head $O(w)$0 penalty on $O(w)$1 with a learnable head-wise weight $O(w)$2 adjusted to target an average retained count $O(w)$3 (He et al., 23 Oct 2025).

The resulting sparsity pattern is described as “A+column”: local sliding-window tokens plus a learned sparse set of distant tokens retained per head (He et al., 23 Oct 2025). In effect, gating determines which old tokens remain directly retrievable after they have left the local window.

The second gating paradigm is logit-decay gating. GatedFWA computes a nonnegative gate $O(w)$4 for each layer, head, and position through a “softplus over scaled pre-activation” construction: $O(w)$5

$O(w)$6

It then accumulates a prefix

$O(w)$7

and defines the pairwise decay bias

$O(w)$8

which is added to the raw dot-product logits under the sliding mask (Liu et al., 8 Dec 2025). Under the exact-kernel associative-memory interpretation, this yields the recurrence

$O(w)$9

where

$w$0

The Jacobian with respect to the previous memory is $w$1, so $w$2 acts as a learnable contraction: when $w$3, memory and gradients pass; when $w$4, old history is suppressed (Liu et al., 8 Dec 2025).

These two gating paradigms solve different problems. Token eviction restores direct access to selected distant KV-pairs. Logit-decay gating stabilizes the memory dynamics of SWA itself. The shared label can therefore be misleading if the underlying gate semantics are not specified.

4. Shared-KV organization and cache semantics

The phrase “Shared-KV” is overloaded across the cited works. In the learnable-token-eviction design, all recent windowed KVs and all gated retained KVs are stored in a head-wise global cache that is physically contiguous. The cache consists of a circular buffer of size $w$5 for the window and a capped buffer of size $w$6 for retained tokens, together with $w$7 global sink tokens (He et al., 23 Oct 2025). Because post-RoPE $w$8 and $w$9 are stored, their relative order is declared unimportant, so entries can be compacted and reordered arbitrarily, which is what enables tiled FlashAttention kernels across both segments (He et al., 23 Oct 2025). In the same formulation, all heads share the same projection matrices for $K,V$0, while the gating CNN is implemented as grouped convolutions that are independent per head but share weights wherever possible (He et al., 23 Oct 2025).

In GatedFWA, Shared-KV refers primarily to headwise sharing of $K,V$1 computation and gate preprocessing. The implementation recipe explicitly recommends a single KV cache: compute $K,V$2 once per token, not per head, and broadcast to all heads, thereby halving the memory footprint and indexing logic (Liu et al., 8 Dec 2025). The gate can itself be shared or head-wise. The gate-preprocessing algorithm outputs $K,V$3 under the Shared-KV setting, with one gate per token broadcast to all heads, although one may instead compute $K,V$4 per head for greater expressivity (Liu et al., 8 Dec 2025).

In MixAttention, by contrast, sharing is inter-layer. Only layers in a subset $K,V$5 compute and store new $K,V$6; every other layer reuses the most recent $K,V$7 from an earlier compute-KV layer $K,V$8 (Rajput et al., 2024). This means that shared-KV can denote contiguous per-head storage, head-broadcasted KVs, or cache reuse across layers, depending on the paper.

A plausible implication is that any discussion of “Shared-KV Gated Sliding-Window Attention” should specify the sharing axis explicitly: across heads, across layers, or across local and retained memory segments.

5. Kernels, buffer management, and asymptotic properties

The systems implementations are designed around FlashAttention-style tiling and Triton or CUDA kernels. In the token-eviction architecture, prefilling maintains a sliding-window buffer and a capped KV cache per head. Attention is computed separately over the window and the retained cache, optionally combined as $K,V$9. When a KV-pair leaves the circular window buffer, its retention score is computed if necessary; if $s$0, it is inserted into the retained cache and the lowest-scoring entry is evicted if the cache is full (He et al., 23 Oct 2025). The implementation notes further specify lazy batching of CNN scoring every $s$1 steps to amortize cost, and fused Triton kernels for eviction and compaction (He et al., 23 Oct 2025).

GatedFWA organizes computation into two kernels: a one-pass tiled scan that computes the prefix vector $s$2 from the gate parameters, and a Flash-style tiled attention kernel that injects the bias $s$3 before the masked streaming Softmax (Liu et al., 8 Dec 2025). The implementation notes emphasize that only two extra vector loads per tile are required, the bias arithmetic is $s$4 additions, and the rest of the pipeline remains identical to FlashAttention, including online max/sum stabilization (Liu et al., 8 Dec 2025).

A compact comparison from the hybrid sparse-attention paper is reproduced below (He et al., 23 Oct 2025).

Method	Time/token	Space/token
Full Attn	$s$5	$s$6
Linear Attn	$s$7	$s$8
SWA	$s$9	$N$0
Pure LTE	$N$1	$N$2
Shared-KV Gated SWA	$N$3	$N$4
laNSA	$N$5	$N$6

Because $N$7 and $N$8 are fixed, the hybrid is described as retaining the constant-time, constant-space benefits of linear attention while still offering direct access to $N$9 distant tokens per head (He et al., 23 Oct 2025). GatedFWA gives a more kernel-level accounting: its compute is $Q,K,V\in\mathbb{R}^{N\times d}$0, its off-chip traffic is $Q,K,V\in\mathbb{R}^{N\times d}$1, and when $Q,K,V\in\mathbb{R}^{N\times d}$2 the additional scan cost is negligible, reported as less than $Q,K,V\in\mathbb{R}^{N\times d}$3 ms at $Q,K,V\in\mathbb{R}^{N\times d}$4K (Liu et al., 8 Dec 2025).

6. Empirical behavior and interpretation

The learnable-token-eviction variant is evaluated on retrieval-intensive benchmarks. On the RULER “needle-in-a-haystack” suite and EVAPORATE recall benchmarks, with contexts up to $Q,K,V\in\mathbb{R}^{N\times d}$5K tokens, Shared-KV Gated SWA under the name laLTE and layer-interleaving with Gated DeltaNet outperforms pure Gated DeltaNet, Gated DeltaNet plus SWA without gating, heuristic eviction via TOVA, and MLP-based gating (He et al., 23 Oct 2025). On RULER S-NIAH at $Q,K,V\in\mathbb{R}^{N\times d}$6 B parameters, the reported averages are $Q,K,V\in\mathbb{R}^{N\times d}$7 for GDN, $Q,K,V\in\mathbb{R}^{N\times d}$8 for GDN + SWA, $Q,K,V\in\mathbb{R}^{N\times d}$9 for laLTE, and $h$0 for the Full Attn hybrid. On EVAPORATE recall at $h$1 B parameters, the corresponding values are $h$2, $h$3, $h$4, and $h$5 (He et al., 23 Oct 2025). These results support the claim that a small learned sparse set of distant tokens can improve retrieval relative to plain SWA without abandoning fixed-size local computation.

GatedFWA is evaluated on language modeling with $h$6 M and $h$7 M-parameter models trained on OpenWebText/WikiText103 at context $h$8, window $h$9, and single-head SV redistribution with $O(N^2)$00 Shared-KV (Liu et al., 8 Dec 2025). The reported forward-plus-backward throughput on A100 is $O(N^2)$01 $O(N^2)$02s/token for FlashAttention, $O(N^2)$03 $O(N^2)$04s/token for SWA, and $O(N^2)$05 $O(N^2)$06s/token for GatedFWA, corresponding to approximately $O(N^2)$07 overhead versus SWA. Validation loss at $O(N^2)$08 M is $O(N^2)$09 for the LLaMA baseline at $O(N^2)$10, $O(N^2)$11 for $O(N^2)$12SWA, $O(N^2)$13 for $O(N^2)$14GatedFWA, and $O(N^2)$15 for $O(N^2)$16GatedFWA + NSA (Liu et al., 8 Dec 2025). On MQAR at $O(N^2)$17, GatedFWA achieves greater than $O(N^2)$18 recall where SWA and many SSMs drop below $O(N^2)$19 (Liu et al., 8 Dec 2025).

MixAttention provides complementary evidence that the broader combination of SWA and KV sharing can materially change the inference trade-off. At $O(N^2)$20K context on NVIDIA H100, the reported peak KV-cache and throughput are $O(N^2)$21 GB and $O(N^2)$22K/sec for Standard Attention, $O(N^2)$23 GB and $O(N^2)$24K/sec for Sliding Window only, $O(N^2)$25 GB and $O(N^2)$26K/sec for MixAttention (MA-Offset), and $O(N^2)$27 GB and $O(N^2)$28K/sec for MixAttention (MA-Pairs) (Rajput et al., 2024). On RULER at $O(N^2)$29K context, Standard Attention scores $O(N^2)$30 overall, Sliding only $O(N^2)$31, Mix-MA-Offset $O(N^2)$32, and Mix-MA-Pairs $O(N^2)$33 (Rajput et al., 2024). This suggests that shared-KV layouts are not merely memory optimizations; they also change which sparse-attention designs remain competitive under long-context deployment constraints.

A common misconception is that sliding-window attention fails only because its window is too small. The cited work points to two additional failure modes: irreversible loss of distant tokens once they leave the window, addressed by learnable token eviction (He et al., 23 Oct 2025), and unstable or weakly controlled memory dynamics under an associative-memory view, addressed by learnable decay gating (Liu et al., 8 Dec 2025). Another misconception is that “shared-KV” names a unique architectural primitive. In the cited literature, it instead denotes a cluster of storage, broadcasting, and reuse strategies that can be combined with different kinds of gates and with different objectives.