V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models

Abstract: Video LLMs (VideoLLMs) show strong capability in video understanding, yet long-context inference is still dominated by massive redundant visual tokens in the prefill stage. We revisit token compression for VideoLLMs under a tight budget and identify a key bottleneck, namely insufficient spatio-temporal information coverage. Existing methods often introduce discontinuous coverage through coarse per-frame allocation or scene segmentation, and token merging can further misalign spatio-temporal coordinates under MRoPE-style discrete (t,h,w) bindings. To address these issues, we propose V-CAST (Video Curvature-Aware Spatio-Temporal Pruning), a training-free, plug-and-play pruning policy for long-context video inference. V-CAST casts token compression as a trajectory approximation problem and introduces a curvature-guided temporal allocation module that routes per-frame token budgets to semantic turns and event boundaries. It further adopts a dual-anchor spatial selection mechanism that preserves high-entropy visual evidence without attention intervention, while keeping retained tokens at their original coordinates to maintain positional alignment. Extensive experiments across multiple VideoLLMs of different architectures and scales demonstrate that V-CAST achieves 98.6% of the original performance, outperforms the second-best method by +1.1% on average, and reduces peak memory and total latency to 86.7% and 86.4% of vanilla Qwen3-VL-8B-Instruct.

• The paper introduces a training-free, plug-and-play method that allocates tokens to high-curvature temporal transitions and selects spatially diverse, high-activation tokens without breaking positional coordinates.
• V-CAST retains 98.6% of baseline accuracy at 25% token retention on Qwen3-VL-8B, improves long-video results by up to 1.5 points, and enables MoE inference that otherwise runs out of memory.
• The method cuts prefill latency by 51.2%, lowers peak memory by 13.0%, and outperforms merging approaches by preserving on-grid MRoPE alignment, though performance declines under extreme compression and remains untested for open-ended generation.

Video LLMs (VideoLLMs) inherit the quadratic prefill cost and linear KV-cache growth of attention-based inference, and video inputs amplify both because the visual token count scales as $N = T \cdot P$ with the number of frames. V-CAST (2603.27650) addresses this bottleneck with a training-free, plug-and-play token pruning method that is organized around a single diagnostic claim: under tight token budgets, the dominant failure mode of prior compression schemes is insufficient spatio-temporal information coverage, arising from either discontinuous temporal allocation or positional misalignment introduced by token merging under multimodal rotary position embeddings (MRoPE).

Diagnosis: two failure modes in existing compression

The authors first characterize why existing pipelines degrade under aggressive budgets. Uniform per-frame allocation (e.g., VisionZip [(Renninger et al., 7 Jan 2025)-style approaches]) cannot upweight brief evidence peaks, so decisive moments are diluted. Segment-based pipelines such as FastVID and HoliTom partition videos into temporal windows, which induces boundary-sensitive budget jumps and near-uniform allocation inside segments, weakening continuity across transitions. Global-uniqueness budgeting, as in VidCom², allocates by frame distinctiveness and can over-concentrate tokens on a few globally salient frames while under-covering transitional segments.

The second failure mode is more consequential for modern architectures. Qwen3-VL binds each visual token to discrete $(t,h,w)$ coordinates via a three-dimensional MRoPE. Token merging (average pooling) moves the aggregated token off the original grid, breaking the one-to-one correspondence with its positional encoding. The paper supports this with a striking negative result: the carefully designed HoliTom underperforms a naive $2\times2$ AvgPool baseline by 2.4 points on MVBench, 1.8 on LongVideoBench, and 0.8 on VideoMME, while a trivial pruning baseline keeping one token per $2\times2$ cell surpasses HoliTom by +2.2 on VideoMME. This implies that on MRoPE-based VideoLLMs, pruning—which preserves on-grid coordinates—is structurally preferable to merging, independent of the sophistication of the selection criterion.

Method: curvature-guided allocation plus dual-anchor selection

V-CAST formulates compression as selecting an index set $\mathcal{I}$ of size $B = \lfloor rN \rfloor$ while preserving order and original coordinates. It proceeds in two stages.

Curvature-guided temporal allocation. Each frame is summarized by its mean token vector, normalized to the unit hypersphere, yielding a semantic trajectory $\{\hat{\mathbf{x}}_t\}$. Semantic curvature at frame $t$ is defined as $$\kappa_t = 1 - \cos(\mathbf{v}^{\text{in}}_t, \mathbf{v}^{\text{out}}_t)$$, where $\mathbf{v}^{\text{in/out}}$ are the incoming and outgoing trajectory directions. Budget density is obtained in closed form from an entropy-regularized objective, $(t,h,w)$0 with temperature $(t,h,w)$1, then converted to integer per-frame budgets via largest-remainder rounding. The entropy term prevents collapse onto a few frames; the exponential weighting routes capacity to event boundaries and semantic turns. The method is attention-agnostic: it operates on visual features before the LLM, which keeps it compatible with FlashAttention and avoids the activation overhead of attention-score-based compression.

Dual-anchor spatial selection. Within each frame, tokens are ranked by $(t,h,w)$2, combining a contextual diversity anchor (deviation from the frame mean, favoring subjects and distinctive regions) with a feature-activation anchor (the $(t,h,w)$3-norm as a saliency proxy). The top-$(t,h,w)$4 tokens are retained at their original spatial coordinates, preserving MRoPE alignment. Both stages are linear in the token count and computable with partial selection.

Empirical results

The evaluation covers MVBench, LongVideoBench, MLVU, VideoMME, and EgoSchema on Qwen3-VL-8B/32B, Qwen3-VL-30B-A3B (MoE), LLaVA-OneVision-7B, and LLaVA-Video-7B, against VisionZip, FastVID, HoliTom, VidCom², and FlashVID. Headline findings:

Setting	Result	Implication
Qwen3-VL-8B, 32 frames, $(t,h,w)$5	63.3 avg (98.6% of vanilla), best of all methods	Near-lossless 4× compression on MRoPE models
Qwen3-VL-8B, 64 frames, $(t,h,w)$6	65.0 avg (97.0%), +1.0 over VidCom²	Gains grow with input length
256 frames, long-video benchmarks	+1.5 over second-best (VidCom²)	Curvature budgeting matters most at long contexts
Qwen3-VL-32B, $(t,h,w)$7	66.5 avg (98.1%), +0.9 over VidCom²	Transfers to larger dense models
Qwen3-VL-30B-A3B, 64 frames	68.1 avg; vanilla model and FlashVID both OOM	Enables inference where the uncompressed MoE model fails
LLaVA-OV / LLaVA-Video, $(t,h,w)$8	99.5% / 95.4% retention	Generalizes to standard RoPE backbones

Two efficiency results stand out. On Qwen3-VL-8B at $(t,h,w)$9, V-CAST reduces prefill latency by 51.2%, peak memory by 13.0%, and total latency to 86.4% of vanilla, with 1.16× throughput—the best efficiency profile among all baselines. By contrast, FlashVID increases peak memory by 84.0% because its attention-guided compression executes inside the LLM and interferes with FlashAttention; it runs out of memory beyond 64 frames and even causes the vanilla 30B-A3B MoE model to OOM in the reported setup. This is a strong practical argument that compression should be attention-agnostic to remain compatible with fused attention kernels.

Ablations support each design choice. Curvature-aware allocation beats uniform, random, and global-uniqueness routing under both 32- and 64-frame settings; the dual-anchor score beats either anchor alone (64.7 vs. 63.4 diversity-only and 61.0 activation-only average). Notably, curvature-aware budgeting is shown to be a drop-in improvement when grafted onto VisionZip and VidCom², suggesting the temporal allocation component is a general contribution independent of the spatial selection rule. Sensitivity analysis shows the average score varies only between 65.0 and 65.3 as $2\times2$0 ranges over $2\times2$1 and under perturbed anchor weights, indicating low hyperparameter fragility.

Limitations and open questions

Several caveats bear on the results. First, the headline retention figures (98.6%, 98.2%) are computed at $2\times2$2; at $2\times2$3 the gap to the uncompressed model widens (94.9% retention at 64 frames), so the method is not loss-free at extreme compression, and the paper does not characterize where performance collapses. Second, the curvature signal is computed from mean-pooled frame embeddings on the hypersphere; the paper assumes this scalar suffices to capture semantic transitions, and it offers no analysis of failure cases where decisive content changes without a directional turn in embedding space. Third, the theoretical section is descriptive rather than formal—there is no approximation-error bound relating curvature-weighted selection to trajectory approximation quality, despite the trajectory-approximation framing. Fourth, the $2\times2$4-norm saliency anchor is justified only empirically; its interaction with register tokens or attention-sink behavior in ViT encoders is unexamined. Finally, all evaluations are on multiple-choice video QA benchmarks; the effect of pruning on open-ended generation, captioning fidelity, or temporal grounding tasks is not measured, leaving open whether curvature-guided budgets preserve fine-grained localization ability.

Conclusion

V-CAST contributes a diagnosis—discontinuous temporal coverage and MRoPE misalignment are the binding constraints on VideoLLM token compression—and a simple remedy: allocate per-frame budgets by semantic curvature with entropy regularization, select spatial tokens by a diversity-plus-norm score, and prune rather than merge to keep coordinates on-grid. The empirical case is consistent across dense, MoE, and RoPE-based backbones, with the strongest advantages at 256-frame inputs and the best efficiency profile among tested baselines. The main open questions are the behavior of the method beyond 75–80% compression, the robustness of the mean-embedding curvature signal on content that transitions without directional change, and generalization beyond multiple-choice evaluation.