Linear Scaling Video VLMs for Long Video Understanding

Abstract: Video vision-LLMs (VLMs) are increasingly used in long-horizon and streaming settings, yet most video encoders still rely on spatiotemporal self-attention, causing compute and latency to grow quadratically with the number of frames. Existing efficiency methods improve scalability but often lose accuracy relative to full self-attention, for example through aggressive frame/token dropping or coarse attention approximations. We introduce StateKV, an inference-time method that adapts pretrained long-video VLMs to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models spanning three families and multiple scales, StateKV remains close to full self-attention and consistently outperforms dominant sliding-window / recency-based streaming approximations, without fine-tuning or architectural changes. StateKV also reduces video-prefill cost measured FLOPs, enabling stronger accuracy at a fixed compute budget by running larger models. These results suggest a practical step toward scalable long-video understanding.

• The paper introduces StateKV, which reformulates video inference using a fixed-capacity recurrent state to achieve linear compute complexity.
• By leveraging importance-driven token selection and a dual-cache architecture, StateKV maintains near full-attention accuracy while significantly reducing compute cost.
• Experimental results across multiple benchmarks show that StateKV enables real-time processing on long videos and supports larger VLM backbones under fixed compute constraints.

Linear Scaling Video VLMs for Long Video Understanding: An In-Depth Analysis

Introduction

The computational inefficiency of prevalent self-attention mechanisms in video Vision-LLMs (VLMs) severely limits their practical deployment on long-horizon and streaming video tasks. Current paradigms result in quadratic scaling of compute and latency with video length, making real-time and long-duration settings intractable. "Linear Scaling Video VLMs for Long Video Understanding" (2605.31598) introduces StateKV, an inference-time method enabling linear-time video prefill for frozen, pretrained VLMs, directly addressing the core scalability bottleneck without sacrificing representational power for downstream text decoding.

Figure 1: Overview of StateKV architecture and compute-accuracy characteristics versus conventional approaches.

Methodology: StateKV

StateKV reframes streaming video inference as an approximation of full self-attention via a fixed-capacity, importance-driven recurrent state—termed the compressed state—coupled with a full per-frame cache. For each transformer layer, StateKV maintains:

• A detailed state (dstate) accumulating all per-frame video tokens and supporting generation.
• A small, fixed-capacity compressed state (cstate), refreshed per-frame and used exclusively for cross-frame context during cache construction.

The mechanism is grounded in two empirical assumptions about the behavior of attention in pretrained VLMs: (i) that long-range attention overwhelmingly focuses on a small set of tokens ("temporal sinks"), and (ii) that the useful cross-frame context set evolves slowly and can be captured via local incremental updates.

Figure 2: Single-layer breakdown of StateKV’s two-cache structure and update logic.

Each incoming frame is encoded by attending to its own tokens and the compressed state from the prior frame, updating both caches accordingly. Token selection for cstate is governed by attention-derived importance scores, where only the top-$B$ tokens (by cumulative attention) are retained for rollback, ensuring compute stays independent of frame count.

Generation proceeds using the complete dstate, guaranteeing the model conditions on full framewise details, even though prefill is strictly linear in the number of frames.

Experimental Results

Compute-Accuracy Pareto Frontier

Empirical results on VideoMME, MLVU, and OVOBench across multiple open VLM backbones (InternVL3-1B/2B/8B, Qwen3-VL-2B/4B/8B, Eagle2.5-8B) rigorously demonstrate that StateKV closely tracks full self-attention accuracy at a fraction of the compute budget, outperforming both sliding-window/recency methods (e.g., ReKV) and fixed-budget compression strategies.

For instance, increasing the StateKV cache budget $B$ continuously trades off FLOPs for accuracy, preserving a log-linear compute-accuracy relationship. Importantly, StateKV at moderate budgets delivered accuracy within 1–2% of full self-attention while enabling the practical use of much larger backbones under fixed compute constraints.

Figure 3: Total compute to preprocess a 512-frame video versus accuracy on VideoMME for different backbone scales and attention approximations.

Context Scaling and Robustness

Scalability analysis with increasing cache budget reveals that larger models, like InternVL3-8B, are able to fully saturate accuracy at large $B$, matching full attention, whereas ReKV and sliding-window techniques consistently underperform, with gaps often exceeding 10 percentage points at the same compute level.

Figure 4: VideoMME accuracy as a function of memory budget, demonstrating StateKV’s strong approximation of full attention across video lengths and scales.

Notably, ReKV’s strict recency bias introduces instability and model-specific failure cases, especially on more parameter-rich or non-CLIP-style backbones. StateKV does not suffer from such volatility, suggesting its empirical assumptions generalize robustly across architectures.

Linear Compute Scaling

The quadratic cost of vanilla self-attention is evident in both marginal and cumulative compute metrics. StateKV realizes a constant per-frame cost in prefill, yielding strict $O(N)$ scaling and well-defined break-even points: beyond a (model-specific) number of frames, even large StateKV models become both more accurate and computationally cheaper than smaller full-attention baselines for long-horizon or streaming tasks.

Figure 5: Marginal and cumulative FLOPs per frame, illustrating compute break-even points where StateKV surpasses full self-attention efficiency.

Furthermore, wall-time measurements with and without Triton-fused attention kernels confirm that the theoretical compute savings translate into real-world speedups once sufficient sequence length is reached.

Mechanistic Validation

A comprehensive analysis on attention patterns in InternVL3-1B/2B/8B, using a set of 16 long videos, empirically confirms the core assumptions:

• Concentration: Top-$B$ tokens (for moderate $B$) capture $\geq 90\%$ of cross-frame attention mass.
• Slow Evolution: The set of important tokens largely persists frame-to-frame, with high recall of future sinks from the prior state plus current frame. Attention-based selection captures significantly more useful information than recency-based retention at matched budgets.

Practical and Theoretical Implications

StateKV represents a major algorithmic advance for scalable, real-time, and long-horizon video VLM deployment. By decoupling compute cost from video length during prefill, it enables:

• Deployment of larger, more accurate backbones under fixed FLOPs or latency budgets.
• Applicability to streaming settings, as constant per-frame cost supports truly unbounded video sequence processing.
• Direct drop-in support for existing VLMs, since StateKV is fully test-time and requires no retraining or architecture modification.

Theoretically, StateKV highlights the emergent sparsity and token-economy in video VLM attention, suggesting that future backbone pretraining regimes could further exploit or even regularize for this structure, driving further gains in efficiency and scalability.

Limitations and Future Directions

This study grounds its design in present pretrained VLMs and benchmark video distributions; future architectures or learning objectives could alter attention structure, and in such cases new assumptions or adaptation strategies might be required. Investigating learnable, adaptive memory management and extending analysis to non-incremental, query-aware or multi-modal settings present promising avenues.

Conclusion

StateKV enables video VLMs to achieve linear-time inference for long and streaming videos by leveraging the empirical structure of attention in pretrained models. Across diverse models and long-video tasks, StateKV consistently offers stronger accuracy-compute tradeoffs than sliding-window or fixed-compression baselines, without incurring the complexity of retraining or redesign. This method provides a practical, theoretically-motivated step toward scalable long-video understanding and establishes a blueprint for future research in efficient multimodal sequence modeling.