StateKV: Efficient Inference for Long-Video VLMs

• StateKV is an inference-time method that adapts long-video vision-language models by modifying KV-cache construction without retraining.
• It employs a two-memory design with detailed and compressed KV states to efficiently carry cross-frame context during video prefill.
• Empirical evaluations demonstrate that StateKV closely matches full self-attention performance while achieving linear computational scaling over video frames.

StateKV is an inference-time method for adapting pretrained long-video vision-LLMs to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state while preserving a second full per-frame cache for final decoding (Eyzaguirre et al., 29 May 2026). It is a training-free modification of KV-cache construction and reuse rather than a retrained architecture: the backbone transformer remains frozen, StateKV requires no fine-tuning or architectural changes, and it changes only how attention context is exposed during video prefill (Eyzaguirre et al., 29 May 2026). In the reported evaluations across three long-video benchmarks and seven models from three families, it remains close to full self-attention while consistently outperforming a sliding-window / recency-based streaming baseline, ReKV (Eyzaguirre et al., 29 May 2026).

1. Problem formulation and scope

StateKV is motivated by the long-video VLM inference bottleneck created by spatiotemporal self-attention. If a video has $N$ frames and each frame contributes $T$ tokens, then the video token sequence length is $L = TN$, and full self-attention scales on the order of $O((TN)^2)$, or $O(N^2)$ when $T$ is treated as fixed (Eyzaguirre et al., 29 May 2026). The practical consequence is that later frames become more expensive to process than earlier frames, and total video-processing cost grows quadratically with the number of frames (Eyzaguirre et al., 29 May 2026). The paper distinguishes the video prefill stage, where the visual KV cache is built and cost is dominant, from the subsequent text decoding stage, which is often cheaper when the generated answer is short (Eyzaguirre et al., 29 May 2026).

This places StateKV within a broader family of KV- and state-management methods, but with a distinct scope. In long-context LLM inference, related systems address prompt or decode KV bottlenecks through semantic retrieval, hierarchical memory, structured retention, or adaptive eviction; examples include sentence-level semantic KV routing in SentenceKV, hierarchical semantic memory with token-level reconstruction in SeKV, structured composite-token retention in KVCompose, and boundary-triggered CPU-GPU retrieval in LouisKV (Zhu et al., 1 Apr 2025, Abaskohi et al., 30 Jun 2026, Akulov et al., 5 Sep 2025, Wu et al., 13 Oct 2025). StateKV differs because it targets frozen pretrained long-video VLMs and uses a stateful approximation specifically to linearize video prefill rather than to compress the final decoding cache (Eyzaguirre et al., 29 May 2026).

2. Two-memory architecture

The central mechanism is a two-memory design maintained at each transformer layer $\ell$. After processing frame $n$, StateKV keeps a detailed state

$D_n^\ell = \{(K_{1:n}^\ell, V_{1:n}^\ell)\}$

and a compressed state

$$C_n^\ell = \{(\bar{K}_n^\ell, \bar{V}_n^\ell)\}, \qquad |\bar{K}_n^\ell| = |\bar{V}_n^\ell| = B$$

with fixed capacity $T$0 (Eyzaguirre et al., 29 May 2026). The detailed state stores all visual KV pairs from all processed frames up to $T$1, whereas the compressed state stores a fixed-budget set of $T$2 key/value entries that summarize the cross-frame context needed during prefill (Eyzaguirre et al., 29 May 2026).

When processing frame $T$3, StateKV does not attend to the full past video. Instead, at layer $T$4, the current frame’s queries $T$5 attend only to the previous compressed memory $T$6 and the current frame’s own keys and values $T$7 (Eyzaguirre et al., 29 May 2026). After frame $T$8 is processed, its full $T$9 are appended to the detailed cache, and a new compressed state is selected from the union of the previous compressed state and current-frame tokens (Eyzaguirre et al., 29 May 2026).

A defining property is that the recurrent state stores actual token-level K/V memories, not pooled frame summaries, learned compression vectors, or low-rank approximations (Eyzaguirre et al., 29 May 2026). The compressed state is therefore a bounded retained subset of original visual token K/Vs. At the same time, the full detailed cache is preserved for final text decoding, so the compressed state is only a prefill-time approximation tool rather than the model’s final conditioning memory (Eyzaguirre et al., 29 May 2026).

3. Importance scoring and recurrent update

StateKV updates its recurrent state using attention-based importance. For frame $L = TN$0, the cross-frame importance of a candidate key token $L = TN$1 is defined from attention mass received from the current frame: $L = TN$2 and one per-layer instantiation is

$L = TN$3

where $L = TN$4 is normalized attention from query token $L = TN$5 in frame $L = TN$6 to candidate key token $L = TN$7 at layer $L = TN$8 (Eyzaguirre et al., 29 May 2026). The candidate pool for the next compressed state is incremental: $L = TN$9 so only the previous retained sinks and the current frame compete for the next state (Eyzaguirre et al., 29 May 2026). StateKV then keeps the top-$O((TN)^2)$0 candidates: $O((TN)^2)$1 and updates

$O((TN)^2)$2

(Eyzaguirre et al., 29 May 2026).

The method is motivated by two empirical assumptions. First, there exists a small set of historical tokens $O((TN)^2)$3 with $O((TN)^2)$4, $O((TN)^2)$5, such that most historical attention mass is concentrated on them (Eyzaguirre et al., 29 May 2026). Second, the useful sink set for the next frame can be approximated from the previous sink set plus the new frame: $O((TN)^2)$6 (Eyzaguirre et al., 29 May 2026). The supplementary evidence reports that top-$O((TN)^2)$7 historical tokens capture a large fraction of historical attention mass and that the next oracle state is highly recoverable from the previous state plus the next frame (Eyzaguirre et al., 29 May 2026).

The implementation maintains and prunes sinks separately within each layer and KV head (Eyzaguirre et al., 29 May 2026). Because the compressed cache size is not equal to the true number of previously seen tokens, StateKV also introduces a virtual sequence length counting all processed tokens up to frame $O((TN)^2)$8, independent of how many tokens are physically retained in the compressed state (Eyzaguirre et al., 29 May 2026). This matters for RoPE consistency: if context extension scaling such as YARN is used, the same scaling configuration must be fixed before cache building and reused during decoding, otherwise cached keys become incompatible with later queries (Eyzaguirre et al., 29 May 2026).

4. Computational properties and inference procedure

StateKV changes the scaling law of the expensive video prefill stage. Under full self-attention, total video prefill is $O((TN)^2)$9 in the number of frames (Eyzaguirre et al., 29 May 2026). Under StateKV, each frame attends only to current-frame tokens $O(N^2)$0 and compressed memory $O(N^2)$1, so per-frame cost is approximately

$O(N^2)$2

which is constant if $O(N^2)$3 and $O(N^2)$4 are fixed, making total prefill $O(N^2)$5 (Eyzaguirre et al., 29 May 2026). The detailed decoding cache still grows linearly with $O(N^2)$6, so generation conditioned on all video tokens remains $O(N^2)$7 in context size, but the paper stresses that video prefill is the dominant cost and is therefore the main target (Eyzaguirre et al., 29 May 2026).

The inference procedure is causal and naturally streaming-compatible. For each frame, the model encodes the frame into $O(N^2)$8 visual tokens, runs each transformer layer against the previous compressed memory plus the current frame, appends the frame’s full K/V to the detailed cache, computes attention-based importance over the previous compressed state and current frame, keeps top-$O(N^2)$9 tokens, and advances the virtual sequence length (Eyzaguirre et al., 29 May 2026). After all frames are processed, the compressed state is no longer used for conditioning; standard autoregressive text decoding runs over the full detailed cache $T$0 (Eyzaguirre et al., 29 May 2026).

The paper explicitly connects this to long-horizon and streaming settings and shows extrapolated compute curves out to 3600 frames, about 1 hour at 1 FPS (Eyzaguirre et al., 29 May 2026). This suggests a broader implication: StateKV’s fixed-budget recurrent state is primarily a prefill-time mechanism for keeping marginal per-frame cost approximately constant while preserving full decoding context afterward.

5. Empirical performance

StateKV is evaluated on three long-video benchmarks: VideoMME in the subtitles-free setting, MLVU, and the Real-Time Visual Perception subset of OVOBench (Eyzaguirre et al., 29 May 2026). Videos are sampled at 1 FPS and each example is capped at 512 frames (Eyzaguirre et al., 29 May 2026). The evaluated models are seven pretrained VLMs across three families and multiple scales: InternVL3-1B, InternVL3-2B, InternVL3-8B, Qwen3-VL-2B, Qwen3-VL-4B, Qwen3-VL-8B, and Eagle2.5-8B (Eyzaguirre et al., 29 May 2026). The main comparison uses a streaming version of exact full self-attention and ReKV, a sliding-window / recency-based streaming baseline, with compute-matched settings $T$1 for ReKV and $T$2 for StateKV in the main cross-backbone table (Eyzaguirre et al., 29 May 2026).

Across settings, the paper summarizes that StateKV stays within roughly 1 point on average of full self-attention while improving over ReKV by about 10 points on average (Eyzaguirre et al., 29 May 2026). Concrete examples are representative. On InternVL3-1B, Full SA scores 46.19 on VideoMME, 47.05 on MLVU, and 55.79 on OVOBench, while StateKV scores 45.8, 46.35, and 55.44; ReKV falls to 37.11, 33.44, and 37.75 (Eyzaguirre et al., 29 May 2026). On Qwen3-VL-4B, Full SA scores 66.59, 68.98, and 64.76, while StateKV scores 65.89, 67.80, and 64.87; ReKV scores 52.63, 49.91, and 49.22 (Eyzaguirre et al., 29 May 2026). On Eagle2.5-8B, Full SA scores 69.81, 73.85, and 69.30, while StateKV scores 67.96, 70.52, and 68.70 (Eyzaguirre et al., 29 May 2026).

The compute-accuracy frontier is also a central result. At 512 frames on VideoMME, StateKV follows a smooth log-linear relationship between compute and accuracy and yields a better frontier than ReKV and than smaller full-attention operating points (Eyzaguirre et al., 29 May 2026). A highlighted example is that InternVL3-8B with StateKV and $T$3 achieves 62.5% VideoMME at similar compute cost to InternVL3-1B Full SA, which achieves 46.2% (Eyzaguirre et al., 29 May 2026). The supplementary further reports that InternVL3-8B with the largest tested StateKV cache budget becomes cheaper than InternVL3-1B Full SA after around 1800 frames, about half an hour at 1 FPS (Eyzaguirre et al., 29 May 2026).

Mechanistic ablations support the underlying assumptions. On 16 long VideoMME videos with 128 sampled frames, the fraction of total historical attention mass captured by top-$T$4 historical tokens reaches 0.71 / 0.69 / 0.72 at $T$5, 0.83 / 0.80 / 0.82 at $T$6, and 0.93 / 0.93 / 0.93 at $T$7 for InternVL3 1B / 2B / 8B respectively (Eyzaguirre et al., 29 May 2026). Weighted candidate-pool recall for recovering the next oracle state from previous state plus next frame reaches 0.90 / 0.95 / 0.92 at $T$8 and 0.96 / 0.97 / 0.96 at $T$9 (Eyzaguirre et al., 29 May 2026). These measurements support the bounded-sink and slow-evolution assumptions that motivate the recurrent update rule.

6. Relation to adjacent KV/state-management methods

StateKV belongs to a broader design space of methods that manage transformer state under memory or bandwidth constraints, but it occupies a distinctive point within that space. SentenceKV changes the retrieval unit from tokens or fixed chunks to sentence buckets, keeping sentence summaries on GPU and selected token KVs on CPU; it is a semantic retrieval-and-offloading variant of KV management for long-context LLMs rather than a prefill-time recurrent approximation for video (Zhu et al., 1 Apr 2025). SeKV organizes context into entropy-guided semantic spans, stores span summaries on GPU and low-rank SVD factors on CPU, and selectively zooms in to reconstruct token-level detail during decoding; it is a hierarchical semantic memory with trained routing and on-demand reconstruction (Abaskohi et al., 30 Jun 2026). KVCompose keeps a structured dense cache layout while allowing head-specific token content through composite tokens and global layer-adaptive budgeting, targeting long-context autoregressive LLM inference with standard serving-engine compatibility (Akulov et al., 5 Sep 2025). LouisKV preserves the full cache in CPU memory and retrieves prompt clusters and decode segments only at semantic boundaries, showing that retrieval cadence itself can be state-change driven rather than token-clock driven (Wu et al., 13 Oct 2025). KVBuffer, in the linear-attention setting, explicitly combines a persistent aggregated state with a small recent tokenwise buffer, demonstrating a hybrid state-plus-buffer design rather than a pure state-only serving path (Zou et al., 18 May 2026). CONF-KV, by contrast, uses next-token confidence to choose a per-step KV budget and then ranks tokens by EMA attention and recency, making it an online confidence-conditioned retention policy for long-horizon LLM decoding (Li et al., 24 May 2026).

These comparisons suggest a useful taxonomy. StateKV proper is a training-free, inference-time approximation to full long-video self-attention during prefill (Eyzaguirre et al., 29 May 2026). SentenceKV, SeKV, KVCompose, LouisKV, KVBuffer, and CONF-KV are state-management methods for long-context LLM or linear-attention serving that compress, retrieve, quantize, or schedule KV state under different constraints (Zhu et al., 1 Apr 2025, Abaskohi et al., 30 Jun 2026, Akulov et al., 5 Sep 2025, Wu et al., 13 Oct 2025, Zou et al., 18 May 2026, Li et al., 24 May 2026). The common thread is that all of them treat cached attention state as a first-class systems object, but StateKV’s specific contribution is to separate compressed state for video prefill from full cache for decoding in frozen pretrained VLMs (Eyzaguirre et al., 29 May 2026).

7. Limitations and significance

The paper is explicit that StateKV’s assumptions are empirical, not universal (Eyzaguirre et al., 29 May 2026). Its mechanistic validation is restricted to the tested models, datasets, and video regimes, so untested future backbones might not exhibit the same concentration of inter-frame attention on a small, slowly evolving sink set (Eyzaguirre et al., 29 May 2026). StateKV also does not compress decoding context: final generation still conditions on the full detailed cache, so decoding memory and context size continue to grow with video length (Eyzaguirre et al., 29 May 2026). Another practical caveat is that importance scoring requires access to attention weights or sufficient statistics, which complicates use of standard fused kernels; the paper therefore reports an eager attention path for cache building and also presents a custom Triton kernel that accumulates per-key scores without materializing the full attention matrix (Eyzaguirre et al., 29 May 2026).

A plausible implication is that StateKV is strongest when useful cross-frame dependencies are concentrated on persistent anchors or a small set of salient tokens, and weaker when relevant evidence is broadly diffuse across history. The paper does not provide a dedicated failure-case benchmark for highly diffuse temporal dependencies, so that interpretation remains an implication of the method’s assumptions rather than a directly measured claim (Eyzaguirre et al., 29 May 2026).

Its broader significance lies in what it implies about pretrained long-video VLMs. The reported results indicate that long-video attention in the tested models is structured enough that streaming prefill can be framed as self-attention approximation via carried state rather than strict recency truncation (Eyzaguirre et al., 29 May 2026). In that sense, StateKV is both a concrete method and a broader design claim: long-range visual context can be approximated during prefill by a fixed-capacity recurrent KV state, while exact full-resolution visual memory remains available for the decoding stage (Eyzaguirre et al., 29 May 2026).