---
title: 'StateKV: Efficient Inference for Long-Video VLMs'
url: https://www.emergentmind.com/topics/statekv
type: topic
---

# StateKV: Efficient Inference for Long-Video VLMs

StateKV is an inference-time method for adapting pretrained long-video vision-language models to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state while preserving a second full per-frame cache for final decoding [2605.31598]. It is a training-free modification of KV-cache construction and reuse rather than a retrained architecture: the backbone transformer remains frozen, StateKV requires no fine-tuning or architectural changes, and it changes only how attention context is exposed during video prefill [2605.31598]. In the reported evaluations across three long-video benchmarks and seven models from three families, it remains close to full self-attention while consistently outperforming a sliding-window / recency-based streaming baseline, ReKV [2605.31598].

## 1. Problem formulation and scope

StateKV is motivated by the long-video VLM inference bottleneck created by spatiotemporal self-attention. If a video has \(N\) frames and each frame contributes \(T\) tokens, then the video token sequence length is \(L = TN\), and full self-attention scales on the order of \(O((TN)^2)\), or \(O(N^2)\) when \(T\) is treated as fixed [2605.31598]. The practical consequence is that later frames become more expensive to process than earlier frames, and total video-processing cost grows quadratically with the number of frames [2605.31598]. The paper distinguishes the video prefill stage, where the visual KV cache is built and cost is dominant, from the subsequent text decoding stage, which is often cheaper when the generated answer is short [2605.31598].

This places StateKV within a broader family of KV- and state-management methods, but with a distinct scope. In long-context LLM inference, related systems address prompt or decode KV bottlenecks through semantic retrieval, hierarchical memory, structured retention, or adaptive eviction; examples include sentence-level semantic KV routing in SentenceKV, hierarchical semantic memory with token-level reconstruction in SeKV, structured composite-token retention in KVCompose, and boundary-triggered CPU-GPU retrieval in LouisKV [2504.00970] [2606.31145] [2509.05165] [2510.11292]. StateKV differs because it targets frozen pretrained long-video VLMs and uses a stateful approximation specifically to linearize video prefill rather than to compress the final decoding cache [2605.31598].

## 2. Two-memory architecture

The central mechanism is a two-memory design maintained at each transformer layer \(\ell\). After processing frame \(n\), StateKV keeps a detailed state
\[
D_n^\ell = \{(K_{1:n}^\ell, V_{1:n}^\ell)\}
\]
and a compressed state
\[
C_n^\ell = \{(\bar{K}_n^\ell, \bar{V}_n^\ell)\}, \qquad |\bar{K}_n^\ell| = |\bar{V}_n^\ell| = B
\]
with fixed capacity \(B\) [2605.31598]. The detailed state stores all visual KV pairs from all processed frames up to \(n\), whereas the compressed state stores a fixed-budget set of \(B\) key/value entries that summarize the cross-frame context needed during prefill [2605.31598].

When processing frame \(n\), StateKV does not attend to the full past video. Instead, at layer \(\ell\), the current frame’s queries \(Q_n^\ell\) attend only to the previous compressed memory \((\bar K_{n-1}^\ell,\bar V_{n-1}^\ell)\) and the current frame’s own keys and values \((K_n^\ell,V_n^\ell)\) [2605.31598]. After frame \(n\) is processed, its full \((K_n^\ell,V_n^\ell)\) are appended to the detailed cache, and a new compressed state is selected from the union of the previous compressed state and current-frame tokens [2605.31598].

A defining property is that the recurrent state stores actual token-level K/V memories, not pooled frame summaries, learned compression vectors, or low-rank approximations [2605.31598]. The compressed state is therefore a bounded retained subset of original visual token K/Vs. At the same time, the full detailed cache is preserved for final text decoding, so the compressed state is only a prefill-time approximation tool rather than the model’s final conditioning memory [2605.31598].

## 3. Importance scoring and recurrent update

StateKV updates its recurrent state using attention-based importance. For frame \(n\), the cross-frame importance of a candidate key token \(j\) is defined from attention mass received from the current frame:
\[
s_{n,j} = \sum_{i=1}^{T} A_{n,i,j},
\]
and one per-layer instantiation is
\[
s_{n,j}^\ell = \frac{1}{T} \sum_{i=1}^{T} A_{n,i,j}^\ell
\]
where \(A_{n,i,j}^\ell\) is normalized attention from query token \(i\) in frame \(n\) to candidate key token \(j\) at layer \(\ell\) [2605.31598]. The candidate pool for the next compressed state is incremental:
\[
\mathcal{U}_n^\ell = \bar{S}_{n-1}^\ell \cup \{1,\dots,T\}_{\text{(frame }n\text{ tokens)}},
\]
so only the previous retained sinks and the current frame compete for the next state [2605.31598]. StateKV then keeps the top-\(B\) candidates:
\[
\bar{S}_n^\ell = \mathrm{TopK}\bigl(\{ s_{n,j}^\ell : j \in \mathcal{U}_n^\ell \},\, B\bigr),
\]
and updates
\[
C_n^\ell \leftarrow \{(\bar{K}_n^\ell, \bar{V}_n^\ell)\} = \{(K_j^\ell, V_j^\ell) : j \in \bar{S}_n^\ell\}
\]
[2605.31598].

The method is motivated by two empirical assumptions. First, there exists a small set of historical tokens \(S_n\) with \(|S_n|=K\), \(K \ll |\mathcal H_{n-1}|\), such that most historical attention mass is concentrated on them [2605.31598]. Second, the useful sink set for the next frame can be approximated from the previous sink set plus the new frame:
\[
S_{n+1} \approx \mathrm{TopK}\bigl(S_n \cup \{\text{tokens in frame } n+1\}\bigr)
\]
[2605.31598]. The supplementary evidence reports that top-\(B\) historical tokens capture a large fraction of historical attention mass and that the next oracle state is highly recoverable from the previous state plus the next frame [2605.31598].

The implementation maintains and prunes sinks separately within each layer and KV head [2605.31598]. Because the compressed cache size is not equal to the true number of previously seen tokens, StateKV also introduces a virtual sequence length counting all processed tokens up to frame \(n\), independent of how many tokens are physically retained in the compressed state [2605.31598]. This matters for RoPE consistency: if context extension scaling such as YARN is used, the same scaling configuration must be fixed before cache building and reused during decoding, otherwise cached keys become incompatible with later queries [2605.31598].

## 4. Computational properties and inference procedure

StateKV changes the scaling law of the expensive video prefill stage. Under full self-attention, total video prefill is \(O(N^2)\) in the number of frames [2605.31598]. Under StateKV, each frame attends only to current-frame tokens \(T\) and compressed memory \(B\), so per-frame cost is approximately
\[
O(T(T+B)),
\]
which is constant if \(T\) and \(B\) are fixed, making total prefill \(O(N)\) [2605.31598]. The detailed decoding cache still grows linearly with \(N\), so generation conditioned on all video tokens remains \(O(N)\) in context size, but the paper stresses that video prefill is the dominant cost and is therefore the main target [2605.31598].

The inference procedure is causal and naturally streaming-compatible. For each frame, the model encodes the frame into \(T\) visual tokens, runs each transformer layer against the previous compressed memory plus the current frame, appends the frame’s full K/V to the detailed cache, computes attention-based importance over the previous compressed state and current frame, keeps top-\(B\) tokens, and advances the virtual sequence length [2605.31598]. After all frames are processed, the compressed state is no longer used for conditioning; standard autoregressive text decoding runs over the full detailed cache \(D_N^\ell\) [2605.31598].

The paper explicitly connects this to long-horizon and streaming settings and shows extrapolated compute curves out to 3600 frames, about 1 hour at 1 FPS [2605.31598]. This suggests a broader implication: StateKV’s fixed-budget recurrent state is primarily a prefill-time mechanism for keeping marginal per-frame cost approximately constant while preserving full decoding context afterward.

## 5. Empirical performance

StateKV is evaluated on three long-video benchmarks: VideoMME in the subtitles-free setting, MLVU, and the Real-Time Visual Perception subset of OVOBench [2605.31598]. Videos are sampled at 1 FPS and each example is capped at 512 frames [2605.31598]. The evaluated models are seven pretrained VLMs across three families and multiple scales: InternVL3-1B, InternVL3-2B, InternVL3-8B, Qwen3-VL-2B, Qwen3-VL-4B, Qwen3-VL-8B, and Eagle2.5-8B [2605.31598]. The main comparison uses a streaming version of exact full self-attention and ReKV, a sliding-window / recency-based streaming baseline, with compute-matched settings \(R=16\) for ReKV and \(B=4096\) for StateKV in the main cross-backbone table [2605.31598].

Across settings, the paper summarizes that StateKV stays within roughly 1 point on average of full self-attention while improving over ReKV by about 10 points on average [2605.31598]. Concrete examples are representative. On InternVL3-1B, Full SA scores 46.19 on VideoMME, 47.05 on MLVU, and 55.79 on OVOBench, while StateKV scores 45.8, 46.35, and 55.44; ReKV falls to 37.11, 33.44, and 37.75 [2605.31598]. On Qwen3-VL-4B, Full SA scores 66.59, 68.98, and 64.76, while StateKV scores 65.89, 67.80, and 64.87; ReKV scores 52.63, 49.91, and 49.22 [2605.31598]. On Eagle2.5-8B, Full SA scores 69.81, 73.85, and 69.30, while StateKV scores 67.96, 70.52, and 68.70 [2605.31598].

The compute-accuracy frontier is also a central result. At 512 frames on VideoMME, StateKV follows a smooth log-linear relationship between compute and accuracy and yields a better frontier than ReKV and than smaller full-attention operating points [2605.31598]. A highlighted example is that InternVL3-8B with StateKV and \(B=4096\) achieves 62.5% VideoMME at similar compute cost to InternVL3-1B Full SA, which achieves 46.2% [2605.31598]. The supplementary further reports that InternVL3-8B with the largest tested StateKV cache budget becomes cheaper than InternVL3-1B Full SA after around 1800 frames, about half an hour at 1 FPS [2605.31598].

Mechanistic ablations support the underlying assumptions. On 16 long VideoMME videos with 128 sampled frames, the fraction of total historical attention mass captured by top-\(B\) historical tokens reaches 0.71 / 0.69 / 0.72 at \(B=256\), 0.83 / 0.80 / 0.82 at \(B=1024\), and 0.93 / 0.93 / 0.93 at \(B=4096\) for InternVL3 1B / 2B / 8B respectively [2605.31598]. Weighted candidate-pool recall for recovering the next oracle state from previous state plus next frame reaches 0.90 / 0.95 / 0.92 at \(B=16\) and 0.96 / 0.97 / 0.96 at \(B=256\) [2605.31598]. These measurements support the bounded-sink and slow-evolution assumptions that motivate the recurrent update rule.

## 6. Relation to adjacent KV/state-management methods

StateKV belongs to a broader design space of methods that manage transformer state under memory or bandwidth constraints, but it occupies a distinctive point within that space. SentenceKV changes the retrieval unit from tokens or fixed chunks to sentence buckets, keeping sentence summaries on GPU and selected token KVs on CPU; it is a semantic retrieval-and-offloading variant of KV management for long-context LLMs rather than a prefill-time recurrent approximation for video [2504.00970]. SeKV organizes context into entropy-guided semantic spans, stores span summaries on GPU and low-rank SVD factors on CPU, and selectively zooms in to reconstruct token-level detail during decoding; it is a hierarchical semantic memory with trained routing and on-demand reconstruction [2606.31145]. KVCompose keeps a structured dense cache layout while allowing head-specific token content through composite tokens and global layer-adaptive budgeting, targeting long-context autoregressive LLM inference with standard serving-engine compatibility [2509.05165]. LouisKV preserves the full cache in CPU memory and retrieves prompt clusters and decode segments only at semantic boundaries, showing that retrieval cadence itself can be state-change driven rather than token-clock driven [2510.11292]. KVBuffer, in the linear-attention setting, explicitly combines a persistent aggregated state with a small recent tokenwise buffer, demonstrating a hybrid state-plus-buffer design rather than a pure state-only serving path [2605.19049]. CONF-KV, by contrast, uses next-token confidence to choose a per-step KV budget and then ranks tokens by EMA attention and recency, making it an online confidence-conditioned retention policy for long-horizon LLM decoding [2605.24786].

These comparisons suggest a useful taxonomy. StateKV proper is a training-free, inference-time approximation to full long-video self-attention during prefill [2605.31598]. SentenceKV, SeKV, KVCompose, LouisKV, KVBuffer, and CONF-KV are state-management methods for long-context LLM or linear-attention serving that compress, retrieve, quantize, or schedule KV state under different constraints [2504.00970] [2606.31145] [2509.05165] [2510.11292] [2605.19049] [2605.24786]. The common thread is that all of them treat cached attention state as a first-class systems object, but StateKV’s specific contribution is to separate compressed state for video prefill from full cache for decoding in frozen pretrained VLMs [2605.31598].

## 7. Limitations and significance

The paper is explicit that StateKV’s assumptions are empirical, not universal [2605.31598]. Its mechanistic validation is restricted to the tested models, datasets, and video regimes, so untested future backbones might not exhibit the same concentration of inter-frame attention on a small, slowly evolving sink set [2605.31598]. StateKV also does not compress decoding context: final generation still conditions on the full detailed cache, so decoding memory and context size continue to grow with video length [2605.31598]. Another practical caveat is that importance scoring requires access to attention weights or sufficient statistics, which complicates use of standard fused kernels; the paper therefore reports an eager attention path for cache building and also presents a custom Triton kernel that accumulates per-key scores without materializing the full attention matrix [2605.31598].

A plausible implication is that StateKV is strongest when useful cross-frame dependencies are concentrated on persistent anchors or a small set of salient tokens, and weaker when relevant evidence is broadly diffuse across history. The paper does not provide a dedicated failure-case benchmark for highly diffuse temporal dependencies, so that interpretation remains an implication of the method’s assumptions rather than a directly measured claim [2605.31598].

Its broader significance lies in what it implies about pretrained long-video VLMs. The reported results indicate that long-video attention in the tested models is structured enough that streaming prefill can be framed as self-attention approximation via carried state rather than strict recency truncation [2605.31598]. In that sense, StateKV is both a concrete method and a broader design claim: long-range visual context can be approximated during prefill by a fixed-capacity recurrent KV state, while exact full-resolution visual memory remains available for the decoding stage [2605.31598].

Source: https://www.emergentmind.com/topics/statekv