Windowed Sink Attention (WSA)

• Windowed Sink Attention (WSA) is a sparse attention mechanism that limits interactions to local windows while incorporating a small set of global sink tokens.
• It is applied in streaming language models, vocal source separation, and theoretical frameworks to reduce computational cost and stabilize long-sequence processing.
• WSA techniques offer control over attention allocation, mitigating oversmoothing and secondary sink issues across transformer layers.

Windowed Sink Attention (WSA) denotes a family of sparse attention constructions that combine local or windowed token interactions with a small set of globally visible “sink” tokens or sink-aware controls. Across the recent literature, the term does not name a single canonical mechanism. In streaming language modeling, WSA refers to maintaining a fixed-size sliding window together with persistent attention sinks in the KV cache; in vocal source separation, it denotes local temporal attention augmented by learned global sink tokens; in theoretical work, it is formalized as window masks plus sink connectivity, optionally with diagonal self-attention biases; and in work on secondary attention sinks, it becomes a layer-aware strategy for detecting and attenuating sink-like tokens inside windows (Xiao et al., 2023, Benetatos et al., 29 Oct 2025, Súkeník et al., 8 May 2026, Wong et al., 22 Dec 2025).

1. Terminological scope and core abstractions

All uses of WSA share two structural motifs: a locality constraint and a globally accessible bypass. The locality constraint restricts attention to a window or neighborhood, reducing the quadratic cost of dense attention. The global bypass is implemented either by retaining a small set of sink tokens, by appending learned sink vectors, or by explicitly detecting tokens that behave as sinks and modulating their attention.

A concise comparison is useful because the same label is used for distinct mechanisms.

Setting	WSA construction	Primary purpose
Streaming LLMs	persistent sink KV + recent-token window	stable infinite-length streaming
Vocal separation	local temporal window + learned sink tokens	reduce temporal attention cost
Theory of sinks	window mask + sink connectivity, optional diagonal bias	control switching and oversmoothing
Secondary-sink control	detect sink-like tokens per layer and reweight/cap them	mitigate middle-layer sink over-allocation

In the streaming formulation, the sink set $S$ and recent window $W$ define the effective keys and values as $K' = [K_S; K_W]$ and $V' = [V_S; V_W]$, after which standard masked attention is applied over $S \cup W$ rather than over the full past (Xiao et al., 2023). In the audio formulation, each query attends to a symmetric temporal neighborhood and to $S$ learned sinks, with output

$$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$

and an equivalent mask formulation

$$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$

This is explicitly presented as a replacement for full temporal self-attention in Mel-Band-Roformer (Benetatos et al., 29 Oct 2025).

The theoretical formulation generalizes the same pattern by defining a block-diagonal window mask $M_w$ and a sink connectivity mask $M_{\mathrm{sink}}$, yielding

$W$0

with an optional diagonal bias term $W$1 to encourage self-attention patterns rather than sink routing (Súkeník et al., 8 May 2026). The sink-aware control formulation in the secondary-sink study does not introduce a fixed global sink token; instead, it treats WSA as a layer-aware policy that suppresses, caps, or reweights tokens that exhibit sink-like alignment and norm signatures inside sliding windows (Wong et al., 22 Dec 2025).

2. Attention sinks, primary sinks, and secondary sinks

The empirical basis for WSA is the attention sink phenomenon: semantically uninformative tokens can receive disproportionately high attention weights across layers, creating vertical stripes in attention matrices. The canonical example is the beginning-of-sequence token, but later work distinguishes between “primary sinks” and “secondary sinks” (Wong et al., 22 Dec 2025).

Primary sinks emerge at the same layer as BOS, typically in early layers, persist throughout network depth, and draw large attention mass consistently across layers. In some models, multiple tokens such as BOS and chat-template tokens function as primary sinks and share the same emergence layer and lifetime. Secondary sinks differ in three ways: they arise primarily in middle layers, persist for a variable number of layers, and draw a smaller—but still significant—amount of attention mass. They are often semantically uninformative tokens such as whitespace, digits, or punctuation, and may occur at any position in the sequence (Wong et al., 22 Dec 2025).

The secondary-sink paper formalizes sink identification by combining attention, alignment, and norm criteria. The principal alignment metric is the cosine similarity between a token hidden state and the BOS hidden state at layer $W$2,

$W$3

with threshold $W$4. Sink tokens also exhibit at least an order-of-magnitude larger hidden-state $W$5 norm than average tokens while having lower key and value norms than average tokens. Sink score is defined per head and layer by

$W$6

and then averaged across heads (Wong et al., 22 Dec 2025).

A further organizing notion is the “sink level,” the attribute pair $W$7, where $W$8 is the first layer at which a token becomes a sink and lifetime is the number of subsequent layers for which the sink signature persists. Larger models exhibit more deterministic sink levels. The study identifies three sink levels in QwQ-32B and six levels in Qwen3-14B, while reporting concentrated creation layers such as $W$9 for Qwen2.5-14B and $K' = [K_S; K_W]$0 for Qwen2.5-32B (Wong et al., 22 Dec 2025).

3. Streaming LLMs: pinned sinks plus rolling windows

In StreamingLLM, WSA is the mechanism that combines a tiny persistent set of sink tokens with a sliding window of recent tokens. The central observation is that naive window attention collapses once the initial sink tokens are evicted from the KV cache, because a large portion of the softmax denominator disappears. Preserving those sink tokens stabilizes attention and enables efficient streaming inference over arbitrarily long sequences without fine-tuning (Xiao et al., 2023).

The algorithm maintains two KV partitions at every layer and head: a persistent sink cache $K' = [K_S; K_W]$1 and a rolling window $K' = [K_S; K_W]$2 of the last $K' = [K_S; K_W]$3 tokens. At each decoding step, attention is computed only over $K' = [K_S; K_W]$4, after which the new token is appended to the window and the oldest window token is evicted if necessary; sink tokens are never evicted. For RoPE, keys are cached before applying rotary embeddings, and rotary is re-applied each step using cache-local positions. For ALiBi, the linear bias is also computed in cache-local coordinates. This positional handling is part of the mechanism’s long-sequence generalization claim (Xiao et al., 2023).

The empirical results are explicit. On PG-19 with Llama-2-13B and cache 1024, window-only inference gives perplexity $K' = [K_S; K_W]$5, whereas retaining $K' = [K_S; K_W]$6 initial tokens plus $K' = [K_S; K_W]$7 recent tokens gives perplexity $K' = [K_S; K_W]$8; replacing the first four tokens with line breaks still gives $K' = [K_S; K_W]$9, indicating that absolute initial positions rather than semantics are responsible for the stabilizing effect. Across Llama-2, MPT, Falcon, and Pythia, keeping a small sink set—often $V' = [V_S; V_W]$0—is reported as sufficient, and a dedicated pretrained $V' = [V_S; V_W]$1 token can reduce this to $V' = [V_S; V_W]$2 (Xiao et al., 2023).

The same paper reports stable language modeling on continuous text exceeding $V' = [V_S; V_W]$3 million tokens, and up to $V' = [V_S; V_W]$4 per-token speedup relative to sliding-window recomputation on NVIDIA A6000 for Llama-2-7B and 13B. It also states that WSA matches one-shot accuracy in streaming ARC-E/C evaluations for instruction-tuned Llama-2 chat models, whereas dense attention runs out of memory and window-only inference collapses once it exceeds the cache (Xiao et al., 2023).

A common misconception is that this form of WSA expands the model’s accessible context. It does not. Content outside $V' = [V_S; V_W]$5 is not attendable, and tasks such as long-document QA or summarization may underperform unless important early content is retained in $V' = [V_S; V_W]$6, included in a larger window, or supplemented by retrieval or context-extension mechanisms (Xiao et al., 2023).

4. Local-window attention with learned sinks in vocal source separation

In vocal source separation, WSA is introduced as a replacement for full temporal self-attention in Mel-Band-Roformer and related RoFormer-based separators. The motivating empirical claim is that temporal attention in pretrained models is strongly local, with most attention mass concentrated within about $V' = [V_S; V_W]$7–$V' = [V_S; V_W]$8 frames, corresponding to $V' = [V_S; V_W]$9–$S \cup W$0 ms, while frequency attention remains global (Benetatos et al., 29 Oct 2025).

The construction is explicit. For an $S \cup W$1 s input with $S \cup W$2 ms hop, the time axis has length $S \cup W$3. Full temporal attention therefore forms $S \cup W$4 score matrices per head, per band, and per layer, with compute and memory scaling as $S \cup W$5 and $S \cup W$6. WSA restricts each query to a small local neighborhood plus a set of global sink tokens, giving complexity $S \cup W$7 and scores memory $S \cup W$8 (Benetatos et al., 29 Oct 2025).

The reported configuration uses $S \cup W$9 local keys per query and $S$0 sinks, so that with $S$1 the number of attention scores drops from $S$2 to $S$3, a reduction of approximately $S$4. The paper states that fine-tuning from the original checkpoint recovers $S$5 of the original SDR while reducing FLOPs by $S$6. The headline metrics after fine-tuning are SDR $S$7 dB versus $S$8 dB baseline, cSDR $S$9 dB versus $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$0 dB, Fullness $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$1 versus $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$2, and Bleedless $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$3 versus $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$4, with degradation concentrated in silent segments (Benetatos et al., 29 Oct 2025).

Integration is deliberately narrow: WSA replaces only the time-axis self-attention, while frequency attention is kept full because the number of mel bands is modest and its attention remains globally structured. Sinks are appended at the sequence level in each time transformer. The paper does not explicitly state whether sink embeddings are shared across layers, and notes that a standard implementation uses learnable token embeddings concatenated per layer and projected by that layer’s $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$5, $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$6, and $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$7 (Benetatos et al., 29 Oct 2025).

This formulation places WSA close to Longformer- and BigBird-style local/global hybrids, but the paper emphasizes a domain-specific rationale: the temporal locality in vocal separation is extremely tight, and the sink tokens preserve diffuse global context that the original full-attention model still exploited weakly (Benetatos et al., 29 Oct 2025).

5. Theoretical foundations: sink switching, diagonal patterns, and oversmoothing

The theoretical treatment of WSA frames sink tokens as one mechanism in a broader family of attention switches. A sink token can function as a hard switch: if many tokens attend exclusively to a sink and the sink value is zero, the attention branch contributes nothing to the residual stream. A diagonal pattern functions as a soft switch: it prohibits inter-token communication while still allowing each token to process its own content, effectively making the attention layer behave like an MLP on the residual stream (Súkeník et al., 8 May 2026).

The paper gives a geometric criterion for sink representability. Let $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$8 be the sink embedding and $$y_t = \sum_{i \in N(t)} \alpha_{t,i} V_i + \sum_{j=1}^{S} \beta_{t,j} V_{\mathrm{sinks},j},$$9 a subset of tokens. There exists a bilinear form $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$0 such that

$$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$1

if and only if the sets $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$2 and $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$3 lie in half-spaces, equivalently are linearly separable by a hyperplane through the origin. The paper also states a sufficient condition verified empirically in middle layers across multiple pretrained models: joint anti-alignment with the sink, $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$4 for all $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$5 (Súkeník et al., 8 May 2026).

The oversmoothing analysis introduces an interpolation between identity attention and uniform causal attention and studies the average cosine similarity after one attention step. The main qualitative result is that denser attention generally increases token similarity under conditions that are often satisfied empirically; across LLaMA3-8B, Gemma-7B, GPT2-XL, and Mistral-7B on C4, WikiText, and CodeParrot, the regime in which denser attention increases similarity dominates, while the regime in which the value-output map counteracts mixing is reported as rare (Súkeník et al., 8 May 2026). This motivates window masks and sink or diagonal switches as architectural controls on smoothing.

The same work also compares the representational cost of sinks and diagonal patterns under weight decay. In a simple orthogonal setting, realizing a sink with dominant logit margin $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$6 can be rank-$$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$7 with nuclear norm $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$8, whereas exact diagonal attention has nuclear norm $$A = \operatorname{softmax}\!\left(\frac{Q[K;K_{\mathrm{sinks}}]^\top}{\sqrt d} + M\right), \qquad Y = A[V;V_{\mathrm{sinks}}].$$9. The paper’s broader cost theorems for backcopy and clustered copy-paste tasks are then used to explain why sink heads are more prevalent than diagonal heads in pretrained transformers (Súkeník et al., 8 May 2026).

Within this framework, WSA is not merely a computational sparsity pattern. It is presented as a way to combine bounded communication radius, hard or soft switching, and low-rank routing favored by regularization. A plausible implication is that the effectiveness of WSA depends not only on sparsity but also on whether the sink geometry and value mapping induce true switching rather than indiscriminate global aggregation.

6. Secondary-sink-aware WSA: layer-aware control inside windows

The most specialized use of the term appears in the synthesis built around secondary attention sinks. Here WSA is a design pattern for windowed inference that detects sink-like tokens per layer and selectively suppresses, caps, or reweights them—especially in middle layers where secondary sinks emerge—while preserving useful routing by primary sinks (Wong et al., 22 Dec 2025).

The empirical basis is the formation mechanism of secondary sinks. At a critical middle layer $M_w$0, specific MLP modules map token representations to vectors aligned with the BOS sink direction of that layer. In DeepSeek-14B, an example cited is layer $M_w$1, where tokens behave like normal tokens up to layer $M_w$2 and are then converted by the MLP into sinks with BOS-like hidden-state norm scaling and lower key-value norms. PCA experiments indicate that multiple principal components of the MLP input space are selectively amplified and mapped to the same BOS-aligned direction, suggesting that a sink direction is encoded in the MLP weights (Wong et al., 22 Dec 2025).

The decisive control variable is the $M_w$3 norm of the creation-layer MLP output. The paper reports that this norm determines both the sink score and the lifetime of the resulting secondary sink. Sink-score grows approximately log-linearly with $M_w$4, and lifetime increases monotonically with the log-norm, with lifetimes ranging from as few as $M_w$5 layers up to approximately $M_w$6 layers. BOS sink-score itself shows a valley-shaped profile across depth, and secondary sinks emerge near the valley minimum, suggesting a compensatory routing role (Wong et al., 22 Dec 2025).

The proposed WSA policies are correspondingly layer-aware. Practical detection signals include alignment $M_w$7 with $M_w$8, hidden-state or MLP-output norms, and a local sink-score approximation in windowed settings,

$M_w$9

Suggested interventions include subtracting a penalty from logits targeting suspected sinks,

$M_{\mathrm{sink}}$0

with $M_{\mathrm{sink}}$1 defined from alignment and norm signals, or capping attention weights to suspected sinks and renormalizing (Wong et al., 22 Dec 2025).

This sink-aware WSA is explicitly framed as a trade-off. The reported benefits are reduced harmful over-allocation to secondary sinks in middle layers, preservation of the primary sink’s useful global routing, and mitigation of activation outliers and vertical attention stripes. The risks are equally explicit: sinks may support depth-wise compression, over-suppression may harm reasoning or stability, BOS or template tokens require bespoke treatment, and local approximations at window boundaries may mis-estimate global attention mass (Wong et al., 22 Dec 2025).

Across these strands of work, WSA therefore denotes a general design principle rather than a single algorithm: constrain most communication to a window, preserve or introduce a small sink-mediated global pathway, and treat sink behavior as either a stabilizing prior, a learned global hub, a theoretical switching mechanism, or a phenomenon to be selectively attenuated.