Reference Sliding Window Attention (R-SWA)

• The paper introduces R-SWA, which combines static reference tokens with a sliding window to maintain a constant key/value cache size, reducing memory growth during long-sequence decoding.
• R-SWA is defined by its dual-context approach, attending to a fixed set of reference tokens and the n most recent outputs, thus balancing long-term context with local dependency modeling.
• Empirical results show that R-SWA improves accuracy and throughput in applications such as OCR, ASR, and machine translation, offering stable performance for sequences exceeding conventional memory limits.

Reference Sliding Window Attention (R-SWA) is an efficient attention mechanism for Transformer architectures designed to address the scalability and memory limitations of classical full self-attention, particularly in long-sequence parsing and generation tasks. R-SWA maintains a bounded key/value (KV) cache by combining a set of fixed “reference” tokens—typically derived from static encodings such as images, speech features, or prompts—with a causal sliding window over the most recently generated tokens. This dual-context mechanism enables constant per-step compute and storage costs throughout autoregressive decoding, supporting high-throughput, long-context tasks such as OCR, ASR, and machine translation without significant degradation in accuracy or throughput, even for sequence lengths exceeding the memory limits of conventional attention (Yin et al., 22 Jun 2026).

1. Motivation and Underlying Principles

R-SWA addresses two central limitations of full attention in Transformer decoders: unbounded KV cache growth and variable stepwise latency in long-sequence generation. In standard Multi-Head Attention (MHA), at decoding step $t$, all $L_m + t$ past tokens must be stored and accessed, leading to $O(L_m + t)$ memory and rising per-step compute cost. In human working memory, however, transcription or copying tasks rely on access to static reference material and a limited local “window” of recent outputs.

R-SWA formalizes this intuition: for each new token generated, the decoder attends to (i) all $m$ reference tokens and (ii) a sliding window of $n$ most recent outputs, keeping the total attention span—and thus the KV cache—fixed at $m + n$ regardless of sequence length. This isolates static content from dynamic context, ensuring that visual or reference information does not degrade over time—a problem that arises in pure sliding window approaches without a persistent reference block (Yin et al., 22 Jun 2026).

2. Mathematical Definition and Attention Mechanism

At generation step $t$, let $L_m = m$ be the number of reference (prefix) tokens and $T$ be the total target sequence length. The model maintains:

• $P = \{1, \ldots, L_m\}$ (indices of all static reference tokens)
• $L_m + t$0 (indices of the most recent $L_m + t$1 generated tokens)

The attention index set for the current query is

$L_m + t$2

The per-step attention operation is

$L_m + t$3

where $L_m + t$4 is the query vector for the current position, and $L_m + t$5 are the key and value vectors for tokens in $L_m + t$6.

The critical property is that the total KV cache, i.e., the set of stored key/value vectors, has fixed size: $L_m + t$7 compared to

$L_m + t$8

for standard full MHA, resulting in a vanishing cache-ratio $L_m + t$9 as $O(L_m + t)$0 (Yin et al., 22 Jun 2026).

3. Algorithmic Implementation and Complexity

A pseudocode summary is as follows:

$m$5

Each token generation performs attention over $O(L_m + t)$1 tokens (constant), and the window buffers are managed queuewise, leading to a per-step cost of $O(L_m + t)$2 per head, independent of $O(L_m + t)$3. In contrast, standard attention incurs growing compute and memory cost—$O(L_m + t)$4 per head for step $O(L_m + t)$5 (Yin et al., 22 Jun 2026).

In existing frameworks, R-SWA is realized using two buffers per attention layer: a static buffer of length $O(L_m + t)$6 (never evicted) and a queue buffer of length $O(L_m + t)$7 (FIFO). Efficient kernel support is achieved by fusing reference and window KV into a single block, compatible with FlashAttention and similar memory-efficient implementations.

4. Integration and Task Generalization

R-SWA is primarily demonstrated in Unlimited OCR, where it replaces all attention layers in the decoder of a DeepSeek OCR-based architecture (DeepEncoder + LLM). The reference KV is derived from the visual encoder, whereas the sliding window encodes the most recent textual outputs. This decomposition directly generalizes to:

• ASR: reference = static acoustic embeddings; window = recent transcript tokens.
• Machine Translation: reference = encoded source sentence(s); window = previous target outputs.
• Any long-form parsing/generation: reference tokens represent immutable task- or input-specific context, while the window provides immediate history (Yin et al., 22 Jun 2026).

In all cases, the principal modification required is to maintain and concatenate two KV stores, without reengineering model structure or training objectives.

5. Empirical Performance and Benchmarks

R-SWA exhibits substantial practical and efficiency benefits. In Unlimited OCR, tested on OmniDocBench v1.5 and v1.6, the following are observed:

• Accuracy: R-SWA with $O(L_m + t)$8 yields 93.23% (v1.5) and 93.92% SOTA (v1.6), exceeding DeepSeek-OCR’s baseline of 89.17%.
• Edit Distance: Reduced from 0.049 (baseline) to 0.038.
• Table-TEDS (structural alignment): Improved by approximately 5.96%.
• Throughput: Average tokens-per-second (TPS) increases by 12.7% (4,951 → 5,580 TPS). For long-sequence decoding (up to 6,000 tokens), TPS for full attention falls by ~35%, while R-SWA remains nearly constant (~7,800 TPS).
• Long-sequence stability: For one-shot transcription of up to 40+ page documents, edit distance remains low (≈0.1069) and character-level distinctiveness high (~96.9% distinct-35).

Kernel latency experiments confirm that R-SWA eliminates stepwise decoding slowdowns (from increasing sequence length and memory allocation in full attention), enabling consistent real-time deployment (Yin et al., 22 Jun 2026).

6. Window Size Sensitivity and Practical Trade-Offs

The hyperparameter $O(L_m + t)$9 governs the length of history available in the window. Empirical studies indicate:

• $m$0: Local dependency modeling degrades, causing slight accuracy reduction.
• $m$1: Minimal additional accuracy gain; per-step computation rises.
• $m$2: Offers the best observed balance.

Consequently, the choice of $m$3 should reflect the domain’s dependency statistics and deployment constraints (Yin et al., 22 Jun 2026). The fixed reference block ensures that task-critical, static context is always directly accessible, while the sliding window captures necessary local dependencies. This architecture prevents both the loss of long-term static context—an issue with pure sliding windows—and unbounded memory bloat from full history accumulation.

7. Implementation Guidance and Broader Implications

Deployment of R-SWA involves dual-buffer management, queue logic for sliding windows, and modification of standard attention kernels to efficiently merge reference and dynamic KV inputs. Inference optimizations include pre-allocating GPU buffers of size $m$4 and avoiding reallocations at each step, which is critical for latency-sensitive or concurrent inference.

R-SWA is compatible with Megatron-LM, HuggingFace Transformers (with FlashAttention or SGLang inference), and expert-parallel regimes (e.g., DeepEP). In training, the encoder can be frozen with downstream LLM fine-tuning via AdamW and cosine annealing, supporting large context windows (up to 32K tokens on mainstream GPU clusters).

Broader adoption of R-SWA or analogous reference-aware windowed attention schemes is plausible in domains requiring efficient, scalable long-sequence generation—such as document-level translation, real-time streaming ASR, code synthesis, and beyond—where full self-attention remains a bottleneck (Yin et al., 22 Jun 2026).

A plausible implication is that any task partitionable into static context and dynamic output can benefit from R-SWA’s strict compute/memory bounds, supporting real-world one-shot long-form decoding scenarios without architectural non-locality or throughput collapse.