StreamLLM: Efficient Streaming for Transformers

Updated 5 July 2026

StreamLLM is a KV-cache management framework that retains a few initial attention sink tokens along with a rolling window of recent tokens for efficient long-context decoding.
It uses a static positional selection rule to maintain constant memory and near-constant per-token compute, achieving up to 22.2x speedup over sliding-window recomputation.
Extensions like SAGE-KV and SirLLM build on StreamLLM by adding content-aware retention mechanisms to further improve memory efficiency and long-context performance.

StreamLLM, usually written StreamingLLM, is a KV-cache management framework for decoder-only transformers that enables stable long-horizon decoding by retaining a small set of initial attention sink tokens together with a sliding window of recent tokens, rather than the full history. It was introduced to address the two dominant bottlenecks of streaming deployment—linear KV-cache growth and the failure of naive window attention beyond the training sequence length—and was shown to let Llama-2, MPT, Falcon, and Pythia perform stable and efficient language modeling with up to 4 million tokens and more, without any fine-tuning (Xiao et al., 2023).

1. Problem formulation and cache structure

StreamingLLM targets the standard autoregressive setting in which each generated token attends over a KV cache containing all prior tokens. In vanilla decoding, KV memory and per-token attention cost both grow linearly with sequence length, while straightforward window attention—keeping only the most recent $W$ tokens—fails once the total sequence length exceeds the cache size. The central empirical observation behind StreamingLLM is that this failure is not merely a loss of distant semantic context; it is a structural instability caused by evicting the very first tokens, which many heads continue to use as stable attention anchors (Xiao et al., 2023).

The framework therefore replaces full-history caching with a fixed-budget cache consisting of two regions:

$\text{StreamLLM cache} = \text{sink tokens} + \text{recent tokens}.$

The sink tokens are a small number of initial tokens, typically the first few positions of the sequence. The recent tokens are maintained as a rolling window. Everything in the middle is discarded. This gives constant memory and near-constant per-token compute after the cache fills, while preserving the initial positions that stabilize attention (Xiao et al., 2023).

This design is specifically a static positional selection rule. The retained tokens are determined by their location in the sequence rather than by content or query-dependent scoring. Later work repeatedly uses this property to position StreamingLLM as the canonical static KV-cache baseline.

2. Attention sinks and the mechanism of stability

The defining phenomenon is the attention sink: many transformer heads assign disproportionately large attention mass to the earliest positions, even when those tokens are not semantically important. In the paper’s analysis, higher layers in Llama-2 allocate strong attention to the first token or first few tokens across many queries, while lower layers remain more local. The same general pattern is reported for Llama-2, Falcon, MPT, and Pythia, and analogous sink-like behavior is also noted in BERT and ViTs (Xiao et al., 2023).

The mechanism is tied to the softmax normalization in self-attention:

$\text{Attn}(Q, K, V) = \text{softmax}\left( \frac{Q K^\top}{\sqrt{d_k}} \right) V.$

Because the attention weights must sum to one, the model learns to route surplus probability mass to a small set of globally visible early positions. These tokens function as denominators that stabilize the attention distribution. Removing them changes the normalization structure seen during training, so the model’s attention pattern becomes out-of-distribution even if the recent semantic evidence is still present (Xiao et al., 2023).

The evidence that sinks are primarily positional rather than semantic is explicit. Replacing the first four tokens with the newline token still preserves the effect, and reintroducing their KV states largely recovers perplexity. This indicates that what matters is not the meaning of those first tokens, but the continued existence of their positions as attention sinks (Xiao et al., 2023).

The same line of work also explored a stronger variant: adding a dedicated learnable sink token during pre-training. In small-scale Pythia-style experiments, this concentrates the sink behavior onto a single token and makes streaming deployment less dependent on multiple initial tokens, while leaving standard benchmark performance essentially unchanged (Xiao et al., 2023).

3. Inference procedure and positional handling

StreamingLLM is an inference-time modification of KV management rather than an architectural retraining scheme. The cache is partitioned into a fixed sink segment and a rolling segment for recent tokens. During generation, each new token attends only to these retained KV entries; the oldest non-sink token is evicted when the recent window is full (Xiao et al., 2023).

A critical implementation detail is positional encoding. StreamingLLM assumes relative-style encodings such as RoPE or ALiBi, and it reassigns positions relative to the current cache rather than preserving the original absolute indices. If the retained cache contains early sink tokens plus some recent tokens, those retained tokens are remapped to contiguous positions within the compact cache. For RoPE, the implementation stores pre-RoPE keys and reapplies the rotary transformation according to the compact cache positions; for ALiBi, the bias is likewise computed from distances inside the retained cache (Xiao et al., 2023).

This positional renumbering is essential. It allows the model to continue operating within the positional regime it was trained on, even though the global sequence has grown far beyond the original context window. In effect, StreamingLLM makes long-sequence decoding resemble a succession of fixed-size local decoding problems that share a permanent anchor at the start of the cache (Xiao et al., 2023).

Against the strongest accuracy baseline for fixed-budget decoding—sliding-window recomputation, which repeatedly recomputes the last window from scratch—StreamingLLM preserves the low-memory profile of windowed KV caching but avoids the quadratic recomputation cost. In the reported streaming settings, it outperforms the sliding-window recomputation baseline by up to 22.2x speedup (Xiao et al., 2023).

4. Empirical behavior and known limitations

The main empirical claim is stability over extremely long token streams. On concatenated PG-19 books, StreamingLLM keeps perplexity stable for model families including Llama-2, MPT, Falcon, and Pythia up to at least 4 million tokens, whereas dense attention deteriorates once the length exceeds the training window and naive window attention collapses once the initial tokens are evicted (Xiao et al., 2023).

Its effect is especially visible in streaming QA. On concatenated ARC-Easy and ARC-Challenge streams with Llama-2-70B-Chat, dense attention is memory-prohibitive, while a 1024-token sliding window collapses to 0.12% on ARC-Easy and 0.32% on ARC-Challenge. Under the same streaming setup, StreamingLLM reaches 91.37% and 80.20%, respectively, essentially matching one-shot non-streaming evaluation (Xiao et al., 2023).

These gains do not imply true unbounded semantic memory. StreamingLLM does not extend the model’s effective context beyond the retained sink-plus-window region. If a task requires evidence older than the rolling window and not recoverable from the sink positions, the method fails by design. In StreamEval-style retrieval settings, once the query–answer distance exceeds the cache size, accuracy drops to zero. On long-document benchmarks, the method can underperform simple truncation when the decisive information lies in the discarded middle of the sequence (Xiao et al., 2023).

The limitation is therefore structural: StreamingLLM is a highly effective streaming method, but not a general solution to long-range retrieval or document-level reasoning. Its strength lies in maintaining stable ongoing generation and dialogue when the immediately relevant information remains local, not in preserving arbitrary old facts.

Subsequent work consistently treats StreamingLLM as the reference static, positional KV-cache baseline and then attempts to keep its simplicity while repairing its information loss. SAGE-KV explicitly adopts the sink-plus-recent skeleton of StreamLLM but replaces the rule “drop everything in the middle” with a one-time, head-aware top- $k$ selection over middle tokens after prefilling. On Llama3.1-8B-Instruct, it reports that at a budget of $B=2$ k tokens SAGE-KV achieves the same accuracy as StreamLLM at $B=8$ k, summarized as about 4x higher memory efficiency (Wang et al., 11 Mar 2025).

SirLLM is presented as an extension and refinement of StreamLLM for infinite-length dialogue. It preserves the attention-sink mechanism but replaces the recent-window rule with token entropy selection plus a decay factor, so that older high-information tokens can survive in the cache. On the Grocery Shopping benchmark, this changes long-term recall dramatically; for Yi-6B, grocery recall accuracy $\text{ACC}_g$ rises from 25.73% under StreamLLM to 99.27% under SirLLM while keeping commonsense QA accuracy nearly unchanged (Yao et al., 2024).

SnapStream combines StreamingLLM-style rolling KV with SnapKV compression and implements the result inside a production system with static graphs and continuous batching. Its cache retains sink tokens, a recent ring buffer, and Top- $K$ tokens selected from the middle region. In a 16-way tensor-parallel deployment of DeepSeek-671B on SambaNova SN40L accelerators at 128k context length, it reports 4x improved on-chip memory usage and up to 1832 tokens per second, while introducing minimal accuracy degradation on LongBench-v2, AIME24, and LiveCodeBench (Li et al., 5 Nov 2025).

Taken together, these successors clarify the lasting role of StreamLLM: it is the simplest practically effective sink-based baseline, and the main axis along which later work improves it is the replacement of purely positional middle-token eviction with some form of content-aware retention.

In later literature, the term streaming LLM is used more broadly than the specific StreamingLLM sink-cache method. A recent taxonomy defines streaming LLMs in terms of data flow and interaction concurrency, distinguishing output-streaming, sequential-streaming, and concurrent-streaming paradigms. Within that broader view, StreamingLLM is one particular approach to KV-cache and attention management for efficient long-context generation, not a general definition of streaming intelligence (Tong et al., 4 Mar 2026).

A separate systems line uses “streaming” to mean that the prompt itself arrives incrementally rather than being fully available before prefill. Stream2LLM, for example, treats context as a stream and extends vLLM with adaptive scheduling and preemption for append-mode and update-mode prompt evolution, which is conceptually distinct from the sink-token KV-retention problem addressed by StreamingLLM (Bachkaniwala et al., 29 Mar 2026).

The name should also not be confused with LLM-Streamline, a layer-wise structured compression method that prunes a contiguous block of transformer layers and replaces it with a lightweight network. That work concerns architectural depth reduction rather than sink-based streaming inference (Chen et al., 2024).

Within this landscape, StreamLLM in the narrow, historically central sense refers to the attention-sink KV-cache method introduced in “Efficient Streaming LLMs with Attention Sinks” (Xiao et al., 2023). Its significance lies in showing that stable million-token streaming can be obtained from standard decoder-only transformers by preserving a few initial anchor positions and a recent rolling window, with no fine-tuning and with substantial efficiency gains.