KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs

Abstract: Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision LLMs (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.

• The paper presents the KTV framework, a training-free method for video question answering that aggressively reduces visual tokens to as few as 504 for a 60-minute video, matching or exceeding performance benchmarks.
• KTV uses a two-stage process with clustering-based keyframe selection and importance-and-redundancy-aware token pruning to deliver unbiased, diversity-maximizing summaries of videos
• Empirically, KTV-7B, with sparse (504 tokens) and dense (1872 tokens) settings, outperformed baseline models in six benchmarks, showing high efficiency and utility without extensive training

KTV is a two-stage, training-free framework for video question answering that reduces both temporal and spatial redundancy before any visual content reaches the LLM. Built on LLaVA-v1.6 with no video-specific fine-tuning, it first selects a compact set of keyframes via clustering of frame-level features, then prunes visual tokens within each keyframe using a combination of importance and redundancy scores. The central empirical claim is that aggressive compression—down to 504 visual tokens for a 60-minute video of 10,800 frames, fewer than the 576 tokens LLaVA-v1.6 encodes from a single image—can match or exceed training-free baselines and even surpass some training-based systems on several benchmarks.

Motivation: the limits of question-relevant frame selection

The paper's principal methodological argument targets CLIP-based, question-relevant keyframe selection used in prior work such as X-Pool, FreeVideo, and SparseVLM. The authors contend that selecting frames by textual similarity to the question produces "semantic traps": frames superficially matching question keywords are retained while crucial contextual evidence—particularly temporally displaced causes or effects referenced only implicitly ("after he touched the table")—is discarded. Because causal and temporal reasoning questions frequently reference events whose visual appearance differs from the queried outcome, question-relevant selection systematically biases the visual context available to the LLM.

KTV instead performs question-agnostic keyframe selection. DINOv2 features are extracted for all $T$ frames, K-means groups them into $m$ clusters (with $m=6$ in all experiments to match baseline sampling budgets), and the frame nearest each centroid is retained as the cluster representative. The selected frames are re-ordered chronologically before encoding. This design deliberately decouples frame selection from the query, yielding what the authors characterize as an unbiased, diversity-maximizing summary of the video.

Key token selection within keyframes

The second stage addresses spatial redundancy. Each keyframe is encoded by the frozen CLIP-L vision encoder into $L$ patch tokens, from which only the top-$k$ are kept. Token scoring combines two criteria:

• Importance: the attention score between each token and the frame's [CLS] token, following prior training-free pruning work.
• Redundancy: the average cosine similarity between a token and all other tokens in the same frame; high similarity indicates redundancy.

Both scores are min-max normalized per frame and combined as $$S_{i,j} = \text{Norm}(S^{cls}_{i,j})\cdot\alpha + (1-\text{Norm}(S^{sim}_{i,j}))\cdot(1-\alpha)$$, where $\alpha$ balances importance against diversity. Ablations place the optimum at $\alpha \in [0.8, 0.9]$, indicating importance dominates but redundancy-awareness contributes measurably.

Token budgets are allocated across frames via a descending set of retention ratios $\beta = \{\beta_1,\dots,\beta_m\}$ assigned according to CLIP text-frame similarity between each keyframe and the question. This reintroduces question awareness at the token level rather than the frame level: frames more relevant to the query receive proportionally more tokens ($k_i = \beta_i \times L$), while global scene coverage is preserved by the clustering stage. Three configurations—sparse (504 total tokens), normal (936), and dense (1872)—span the operating range.

Empirical results

Using LLaVA-v1.6-7B and 34B on seven Multiple-Choice VideoQA benchmarks (NExT-QA, EgoSchema, IntentQA, STAR, VideoMME, MVBench, MLVU-Test), KTV compares favorably against IG-VLM, SF-LLaVA, and DYTO:

Method	Tokens	NExT-QA	EgoSchema	STAR	VideoMME	MVBench
IG-VLM-7B	—	63.1	35.8	48.6	39.7	42.9
SF-LLaVA-7B	3680	64.0	44.2	48.8	39.4	43.3
DYTO-7B	3680	65.7	48.6	50.7	41.2	—
KTV-7B-sparse	504	64.5	49.6	52.3	43.6	46.2
KTV-7B-dense	1872	65.8	51.0	52.7	44.0	46.0
DYTO-34B	3680	72.9	56.8	51.1	53.4	—
KTV-34B-dense	1872	72.7	57.0	54.7	53.2	51.5

Several results stand out. KTV-7B-sparse outperforms both IG-VLM and SF-LLaVA across all six reported benchmarks using only 504 tokens, with inference time of 1.19s—36.4% of SF-LLaVA-7B's cost. On MLVU-Test, KTV-34B-dense reaches an M-Avg of 45.0, exceeding VILA-1.5-40B by +0.8% and trailing Video-LLaMA2-72B and InternVL2-76B by less than 1%, despite those models using roughly twice the LLM capacity. Subcategory results reinforce this: KTV-34B-sparse attains the highest Needle QA score (53.3) among all compared methods including training-based ones, and KTV-34B-dense leads Sports QA at 58.3. The claim that a training-free pipeline can rival 70B+ instruction-tuned models on long-video benchmarks is the paper's strongest assertion, and it rests on MLVU-Test comparisons where model sizes differ substantially—a caveat worth noting when interpreting the margin.

Ablation findings

The ablation study isolates three components on NExT-QA, EgoSchema, and IntentQA with KTV-7B-normal. Replacing clustered keyframes (CK) with question-relevant keyframes (QK) degrades accuracy—for example, −4.3% on EgoSchema relative to CK—directly supporting the semantic-trap hypothesis. Removing either the importance score (IS) or redundancy score (RS) also hurts performance (−1.8% and −1.0% respectively on EgoSchema), confirming their complementarity. Notably, IS-only pruning consistently outperforms RS-only pruning, leading the authors to conclude that semantic relevance matters more than intra-frame diversity for token selection—an observation that partially tensions with the framework's stated goal of preserving visual diversity. Uniform temporal sampling fares worst overall, particularly on long videos, validating the clustering approach over fixed-stride baselines.

Limitations and open questions

The paper concedes several constraints implicitly through its design choices. First, the number of keyframes is fixed at $m$0 for comparability with baselines; whether dynamic cluster counts scaled to video length would improve long-video performance is untested. Second, token budget allocation still depends on CLIP text-frame similarity, so the framework does not fully escape the cross-modal alignment biases it criticizes at the frame level—the interaction between agnostic frame selection and question-aware token allocation is not ablated independently. Third, evaluation is restricted to Multiple-Choice VideoQA without subtitles; open-ended generation, dense captioning, and audio-conditioned tasks remain unexamined. Fourth, the DINOv2 clustering pass adds a full-video forward pass whose cost is not included in the reported inference-time comparisons, which measure VLM latency only. Finally, the finding that dense configurations sometimes underperform sparse ones on individual subcategories (e.g., MVBench for the 34B model) suggests the relationship between token budget and accuracy is benchmark-dependent and not fully characterized.

Conclusion

KTV demonstrates that hierarchical redundancy reduction—question-agnostic clustering for temporal compression followed by importance-and-redundancy-weighted token pruning for spatial compression—enables competitive Multiple-Choice VideoQA performance at visual token budgets below those of a single image. Its strongest results, on EgoSchema, STAR, and MLVU-Test, indicate that preserving global scene coverage matters more than question-keyword alignment when selecting frames, while per-token question awareness recovers relevance at finer granularity. The main open questions concern adaptive keyframe counts, the residual dependence on CLIP similarity for budget allocation, and generalization beyond multiple-choice evaluation.