Key-Frame Sampler Techniques & Applications

• Key-frame sampler is a method that selects a compact subset of frames from videos to retain essential task-related information under limited resources.
• It leverages diverse signals—including temporal coverage, cross-modal relevance, and geometric diversity—to address varied applications like action recognition and video summarization.
• Recent approaches range from training-free heuristics to learned, reinforcement-guided models that optimize frame selection for enhanced downstream performance.

A key-frame sampler is a procedure that maps a video or temporal sequence to a compact subset of frames, frame indices, or frame-selection weights, chosen to preserve task-relevant information under a constrained budget of visual tokens, storage, or computation. In the recent literature, this role appears in several technically distinct forms: fixed sparse temporal sampling for long-video video-LLMs, motion-guided frame selection for action recognition, text-conditioned frame retrieval for VideoLLMs, graph- and shot-based summarization, reconstruction-driven subset selection for pose estimation and video compression, and CTC-guided temporal sparsification in speech recognition (Tan et al., 2024, Zhi et al., 2021, Liang et al., 2024, Zhang et al., 2020, Fan et al., 2023). The unifying idea is that a small fraction of frames often carries most of the information needed for downstream reasoning, prediction, or identification, but the criteria for “informativeness” differ sharply across domains.

1. Problem formulations and scope

A common formalization is to treat key-frame sampling as subset selection under a fixed budget. In "Adaptive Keyframe Sampling for Long Video Understanding" (Tang et al., 28 Feb 2025), the input is a video

$$\mathbf{V}\in\mathbb{R}^{T\times W\times H\times C}$$

together with a prompt $\mathbf{Q}$, and the output is an index set

$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$

where $M$ is the allowed number of key frames. That paper first states an ideal objective

$$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$

then replaces it with a surrogate combining prompt-frame relevance and temporal coverage. In "MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding" (Li et al., 5 Feb 2025), the problem is instead posed on a matrix of frame embeddings $\mathbf{Q}\in\mathbb{R}^{n\times d}$, and the selected rows maximize geometric volume in the reduced embedding space. In "MGSampler: An Explainable Sampling Strategy for Video Action Recognition" (Zhi et al., 2021), the goal is to compress a trimmed video of length $T$ into a fixed-size set of $N$ frames for an existing action backbone.

These formulations already indicate that “key-frame sampler” is not tied to one representation level. Some samplers operate directly on decoded frames, some on frame embeddings, some on similarity matrices, and some on latent temporal states. In "MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding" (Tan et al., 26 Feb 2026), the sampler maps a query-frame similarity matrix

$\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$

to frame-sampling probabilities $\mathbf{p}\in\mathbb{R}^{N_f}$. In "Key Frame Proposal Network for Efficient Pose Estimation in Videos" (Zhang et al., 2020), the sampler predicts a relaxed binary vector $\mathbf{Q}$0 over frame positions. In "Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition" (Fan et al., 2023), the “frames” are acoustic timesteps, and key frames are defined by intermediate CTC outputs rather than by video content.

A limiting case is especially important: some systems use the term “key frame” without learning a frame selector at all. "Koala: Key frame-conditioned long video-LLM" (Tan et al., 2024) does not introduce a standalone learned key-frame sampler. Instead, it samples a fixed number of coarsely spaced frames across the full video duration and uses their tokens as global conditioning context for additional tokenizers over denser local segments. In that setting, key-frame sampling is deterministic preprocessing, while selection of what matters is deferred to downstream query-conditioned attention.

2. Selection criteria and signals

The literature uses several distinct signals to define frame importance.

Family	Selection signal	Representative papers
Fixed temporal coverage	Coarse global spacing, equal-motion mass, recursive temporal bins	(Tan et al., 2024, Zhi et al., 2021, Tang et al., 28 Feb 2025)
Cross-modal relevance	CLIP or BLIP similarity between text/query and frame	(Liang et al., 2024, Tang et al., 28 Feb 2025, Tan et al., 26 Feb 2026, Zhang et al., 23 Jun 2026)
Structural representativeness	Shot cohesion, graph smoothness, within-shot typicality	(Zhao et al., 18 Mar 2026, Zhang et al., 2024, Sahami et al., 2021)
Geometric diversity	Maximum-volume subset in embedding space	(Li et al., 5 Feb 2025)
Reconstruction utility	Ability of retained frames to reconstruct missing content	(Zhang et al., 2020, Fu et al., 2023, Pertsch et al., 2019)
Domain-specific sparsification	Non-blank CTC outputs, start/end anchor frames	(Fan et al., 2023, Yang et al., 2024)

Query- or text-conditioned relevance is central in recent long-video methods. "KeyVideoLLM: Towards Large-scale Video Keyframe Selection" (Liang et al., 2024) frames key-frame selection as a text-video frame similarity problem. It uniformly pre-samples $\mathbf{Q}$1 candidate frames, embeds them with CLIP-ViT-B/32, embeds the text, scores each frame by cosine similarity

$\mathbf{Q}$2

then takes the top-$\mathbf{Q}$3 frames and restores temporal order. "MSJoE" (Tan et al., 26 Feb 2026) generalizes this by having the MLLM generate multiple visual queries, then using frozen CLIP to compute a full query-frame similarity matrix before a learnable sampler predicts frame probabilities.

Motion and temporal change form another major criterion. MGSampler estimates per-frame motion either from adjacent RGB-frame differences or shallow feature differences, normalizes them to a salience distribution, reshapes the distribution with a power $\mathbf{Q}$4, and samples one frame from each equal-mass interval of the cumulative motion curve (Zhi et al., 2021). The result is “motion sensitive” yet “motion uniform”: it avoids both uniform under-sampling of short decisive actions and naïve top-$\mathbf{Q}$5 collapse onto a single motion peak. In a very different domain, the Conformer ASR key-frame mechanism identifies informative timesteps by the intermediate CTC rule

$\mathbf{Q}$6

with consecutive duplicate non-blank outputs removed (Fan et al., 2023).

Several methods explicitly target coverage or structure rather than local salience alone. AKS defines a surrogate objective combining relevance with a temporal coverage term inspired by Ripley’s $\mathbf{Q}$7-function (Tang et al., 28 Feb 2025). AdaQ turns frame relevance scores into a soft distribution

$\mathbf{Q}$8

and then samples from an adaptive quasi-Gaussian $\mathbf{Q}$9-$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$0 interval whose width depends on score variance (Zhang et al., 23 Jun 2026). InfoShot first partitions a video into semantically consistent shots and then selects, from each shot, one “common” frame based on within-shot typicality and one “unique” frame based on within-shot volatility (Zhao et al., 18 Mar 2026). This design directly encodes the idea that a sampler should preserve both stable shot semantics and brief deviations.

Finally, some samplers define informativeness through recoverability. K-FPN seeks the smallest subset of rows of a temporal feature matrix $$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$1 such that the full sequence remains reconstructable under a learned dynamics dictionary, formalized by

$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$2

subject to row-selection constraints (Zhang et al., 2020). FrameRS similarly uses a self-supervised masked reconstructor and labels the best frame subset as the one minimizing reconstruction MSE (Fu et al., 2023). KeyIn uses a differentiable latent-variable model that predicts keyframes and their occurrence times so that a separate inpainter can reconstruct the rest of the sequence (Pertsch et al., 2019).

3. Training-free and heuristic samplers

A large class of key-frame samplers is explicitly training-free.

MGSampler is a representative example in action recognition. After computing motion magnitudes $$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$3, it normalizes them to $$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$4, applies power smoothing

$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$5

forms the cumulative motion distribution

$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$6

and samples one index from each equal partition of $$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$7 by inverse-CDF style lookup (Zhi et al., 2021). This gives a sparse and interpretable sampler whose decisions can be visualized directly on the cumulative motion curve.

Text-conditioned but training-free retrieval appears in KeyVideoLLM, AKS, and AdaQ. KeyVideoLLM relies on pretrained CLIP only; AKS scores candidate frames with BLIP ITM or CLIP and then uses a recursive split-or-stop algorithm that alternates between TOP-like and BIN-like behavior depending on whether

$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$8

within the current temporal segment (Tang et al., 28 Feb 2025). AdaQ is also training-free, but unlike hard top-$$\mathcal{I}\subseteq\{1,2,\ldots,T\}, \quad |\mathcal{I}|=M,$$9 retrieval it adapts the softness of its relevance distribution through

$M$0

then performs probability-weighted sampling without replacement from an effective $M$1-$M$2 interval (Zhang et al., 23 Jun 2026). The paper’s main claim is that this soft, example-adaptive coverage control is better suited to the local-versus-global tradeoff in long-video QA than hard top-$M$3.

MaxInfo is training-free but uses a different principle: geometric diversity. After CLIP-based frame embedding and truncated SVD, it selects frame rows by maximizing the rectangular volume

$M$4

of the reduced embedding submatrix, implemented with a rectangular MaxVol algorithm (Li et al., 5 Feb 2025). The method’s rationale is that redundant frames collapse volume, whereas representative and diverse frames expand it.

Structure-aware summarization methods are older but conceptually important. The graph-sampling method of "Fast Graph Sampling for Short Video Summarization using Gershgorin Disc Alignment" (Sahami et al., 2021) constructs a similarity path graph, defines

$M$5

and uses the E-optimality criterion $M$6 as a proxy for worst-case reconstruction quality. "Shot Segmentation Based on Von Neumann Entropy for Key Frame Extraction" (Zhang et al., 2024) instead computes a frame-similarity matrix, partitions the sequence into shots by minimizing the sum of per-shot Von Neumann entropies, and selects the first frame of each shot as the key frame. "Deep Unsupervised Key Frame Extraction for Efficient Video Classification" (Tang et al., 2022) combines CNN features with Temporal Segment Density Peaks Clustering, aiming to preserve temporal information while estimating key frames without direct supervision. "Key Frame Extraction with Attention Based Deep Neural Networks" (Arslan et al., 2023) uses an attention-augmented autoencoder and k-means in latent space, selecting the frame nearest each cluster center.

InfoShot occupies an intermediate position between summarization and long-video inference. It is training-free, shot-aware, and task-agnostic. For each shot $M$7, it computes typicality

$M$8

and local volatility

$M$9

then selects one representative and one deviation-sensitive frame per shot (Zhao et al., 18 Mar 2026). This explicitly encodes two complementary notions of keyness inside each temporal segment.

4. Learned and task-coupled samplers

Learned samplers typically arise when frame selection is tightly coupled to a downstream model or reconstruction objective.

Koala is a useful starting point because it clarifies a frequent misconception. The system samples a fixed set of key frames $$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$0, encodes them with a pretrained short-video tokenizer

$$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$1

and uses those tokens both as direct LLM input and as conditioning context for a Conditioned Segment tokenizer and a Conditioned Video tokenizer (Tan et al., 2024). The novelty lies in key-frame-conditioned hierarchical tokenization, not in a learned frame scorer.

MSJoE is the opposite design: a truly learnable key-frame sampler coupled to an MLLM. The MLLM first generates up to four visually grounded textual queries from the question and a low-cost preview, frozen CLIP computes a similarity matrix over dense video frames, and a lightweight 1D U-Net with about 2 million parameters maps that matrix to per-frame probabilities (Tan et al., 26 Feb 2026). Frame indices are then sampled without replacement, and both the MLLM and sampler are trained by reinforcement learning. The sampler uses REINFORCE, while the MLLM uses GRPO, so query reasoning, frame selection, and answer generation co-adapt.

The Conformer ASR key-frame mechanism is learned in a domain-specific way. After an intermediate encoder stage, an intermediate CTC branch labels frames as blank or non-blank; the latter define key positions $$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$2. KFSA uses these positions to mask later self-attention, whereas KFDS physically drops blank-dominated frames and feeds the shortened sequence into the second encoder stage (Fan et al., 2023). The method is enabled only after 40 epochs so that the intermediate CTC branch becomes reliable.

Reconstruction-based learned samplers form another family. K-FPN outputs relaxed selection scores $$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$3 and minimizes

$$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$4

so frames are selected because they make the full deep-feature sequence reconstructable from a learned dynamics dictionary (Zhang et al., 2020). FrameRS first pretrains a masked video reconstructor, then labels the best subset of visible temporal positions as the one yielding minimum reconstruction loss, and trains a lightweight selector on the encoder features to classify the best $$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$5-of-$$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$6 subset (Fu et al., 2023). KeyIn makes both the keyframe content and the keyframe times latent variables; during training it uses a differentiable relaxation in which the target keyframe is

$$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$7

with $$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$8 a learned distribution over occurrence times (Pertsch et al., 2019).

The term “sampler” can also broaden beyond subset selection of an observed video. "ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler" (Yang et al., 2024) addresses bounded interpolation between a start frame $$\mathrm{KS}_M(\mathbf{Q},\mathbf{F})=\arg\max_{|\mathcal{I}|=M}G'(\{\mathbf{F}_t\mid t\in\mathcal{I}\}),$$9 and an end frame $\mathbf{Q}\in\mathbb{R}^{n\times d}$0. Its contribution is a training-free sequential bidirectional diffusion process that bridges two anchor frames without the off-manifold artifacts associated with parallel branch fusion. This is a keyframe-conditioned generative sampler rather than a key-frame selector.

5. Applications and empirical evidence

In long-video MLLMs and VLLMs, key-frame samplers are primarily evaluated by QA accuracy under strict frame budgets. Koala reports that its lightweight, self-supervised long-video extension outperforms state-of-the-art large models by 3–6% in absolute accuracy across all tasks, and the base-to-Koala gains on three cited benchmarks are $\mathbf{Q}\in\mathbb{R}^{n\times d}$1 on EgoSchema, $\mathbf{Q}\in\mathbb{R}^{n\times d}$2 on Seed-Bench Procedure Understanding, and $\mathbf{Q}\in\mathbb{R}^{n\times d}$3 on Seed-Bench Action Recognition (Tan et al., 2024). AKS improves all three tested MLLMs; for LLaVA-Video-7B, UNI, TOP, BIN, and ADA obtain $\mathbf{Q}\in\mathbb{R}^{n\times d}$4, $\mathbf{Q}\in\mathbb{R}^{n\times d}$5, $\mathbf{Q}\in\mathbb{R}^{n\times d}$6, and $\mathbf{Q}\in\mathbb{R}^{n\times d}$7 on LongVideoBench, and $\mathbf{Q}\in\mathbb{R}^{n\times d}$8, $\mathbf{Q}\in\mathbb{R}^{n\times d}$9, $T$0, and $T$1 on VideoMME, respectively (Tang et al., 28 Feb 2025). AdaQ reports that Qwen3-VL-8B with 64 frames improves from $T$2 to $T$3 on LongVideoBench, from $T$4 to $T$5 on Video-MME, from $T$6 to $T$7 on LVBench, and from $T$8 to $T$9 on MLVU with CLIP-based sampling, and also states that Qwen3-VL-8B + AdaQ with 64 frames exceeds GPT-4o by 15.8 percentage points on average across four benchmarks (Zhang et al., 23 Jun 2026). MaxInfo reports $N$0 on LongVideoBench and $N$1 on EgoSchema for LLaVA-Video-7B, and a $N$2 improvement for LLaVA-Video-72B (Li et al., 5 Feb 2025).

Text-conditioned frame retrieval also shows strong data-management benefits. KeyVideoLLM reports a compression rate of up to $N$3, a 100.0% selection success rate across all tested datasets, and speedups up to $N$4 over Katna; average runtime is about $N$5–$N$6 seconds per video (Liang et al., 2024). In supervised instruction tuning, CLIP-QA improves Video-LLaVA over uniform sampling, for example from $N$7 to $N$8 on MSVD and from $N$9 to $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$0 on MSRVTT. InfoShot, which is task-agnostic rather than query-conditioned, reaches average SynFlash recall $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$1 versus $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$2 for uniform, and on Video-MME at $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$3 fps it ties uniform at $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$4, showing that a shot-aware sampler can preserve transient events without degrading standard QA (Zhao et al., 18 Mar 2026).

In action recognition and speech, the evidence emphasizes efficiency-accuracy tradeoffs. MGSampler improves TSM with 8 frames from $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$5 to $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$6 on Something-Something V1 and from $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$7 to $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$8 on V2 using feature-level difference, while test-time latency rises only slightly, for example from $\mathbf{S} \in \mathbb{R}^{N_q \times N_f}$9 ms to $\mathbf{p}\in\mathbb{R}^{N_f}$0 ms on Something-Something V2 (Zhi et al., 2021). The Conformer key-frame mechanism discards more than 60% of frames while maintaining or improving error rates: on AISHELL-1, KFDS with $\mathbf{p}\in\mathbb{R}^{N_f}$1 yields $\mathbf{p}\in\mathbb{R}^{N_f}$2 CER with a $\mathbf{p}\in\mathbb{R}^{N_f}$3 drop ratio, and on LibriSpeech it reaches $\mathbf{p}\in\mathbb{R}^{N_f}$4 WER with $\mathbf{p}\in\mathbb{R}^{N_f}$5 frame dropping (Fan et al., 2023). K-FPN typically keeps about $\mathbf{p}\in\mathbb{R}^{N_f}$6–$\mathbf{p}\in\mathbb{R}^{N_f}$7 of $\mathbf{p}\in\mathbb{R}^{N_f}$8 frames and reports about $\mathbf{p}\in\mathbb{R}^{N_f}$9 to $\mathbf{Q}$00 speedups over stronger framewise baselines while improving or matching pose accuracy (Zhang et al., 2020).

In summarization and re-identification, benefits depend strongly on the visual regime. The graph-sampling method using Gershgorin disc alignment attains $\mathbf{Q}$01 on VSUMM, comparable to stronger sparse-reconstruction baselines but at substantially reduced complexity (Sahami et al., 2021). The Von Neumann entropy shot-segmentation paper reports $\mathbf{Q}$02 and $\mathbf{Q}$03 on a 20-second “Rising Waves” fragment, though the same paper also prints table values on Open Video clips that do not consistently support its surrounding narrative (Zhang et al., 2024). In kākā re-identification, the proposed automated pipeline yields very high downstream video-level majority-vote accuracy, such as $\mathbf{Q}$04 on Dataset B for several clustering-based selectors and for Random 7, indicating that under a controlled feeder-camera setup the overall filtering-and-cropping pipeline may matter more than the exact clustering rule (Maddigan et al., 9 Oct 2025).

6. Conceptual issues, misconceptions, and open problems

The first recurring misconception is that a key-frame sampler must be a learned frame scorer. Koala explicitly shows otherwise: its “key frames” are a fixed sparse global pass used to compute $\mathbf{Q}$05, while long-video reasoning happens through conditioned tokenization downstream (Tan et al., 2024). Conversely, MSJoE, K-FPN, and the Conformer key-frame mechanism show that sampling can be tightly integrated with end-task learning and optimized jointly or through policy gradients (Tan et al., 26 Feb 2026, Zhang et al., 2020, Fan et al., 2023).

A second misconception is that hard top-$\mathbf{Q}$06 retrieval is sufficient once a good relevance signal exists. Several papers argue against this. AdaQ identifies two structural weaknesses of hard key-frame selection for long-video MLLMs: inflexibility across query types and brittleness to noisy similarity estimates from image-text embedding models (Zhang et al., 23 Jun 2026). AKS reports that TOP sampling helps when evidence is concentrated, BIN helps when evidence is distributed, and ADA performs best overall because it adapts between the two (Tang et al., 28 Feb 2025). InfoShot’s mutual-information argument similarly shows why a sampler optimized only for shot structure can miss within-shot transient evidence (Zhao et al., 18 Mar 2026).

A third issue is that training-free methods inherit the weaknesses of their external scorers or candidate pools. KeyVideoLLM can only retrieve relevant frames from its uniformly pre-sampled set of $\mathbf{Q}$07 candidates (Liang et al., 2024). AKS notes that if candidate extraction is too sparse, short events may already be absent before adaptive selection begins (Tang et al., 28 Feb 2025). AdaQ states that failures still arise from noisy or misaligned external similarity models and from the capability boundary of the downstream MLLM (Zhang et al., 23 Jun 2026). Similar concerns appear in wildlife and action settings, where optical-flow thresholds, cluster counts, or shallow motion features can be sensitive to nuisance motion, camera movement, or environmental regularity (Maddigan et al., 9 Oct 2025, Zhi et al., 2021).

There are also unresolved scaling questions. ViBiDSampler does not present a full multi-keyframe extension; it suggests segment-wise or recursive adaptation as a natural direction, but leaves longer videos and more than two anchors as an open systems problem (Yang et al., 2024). FrameRS is tied to a fixed $\mathbf{Q}$08-of-$\mathbf{Q}$09 subset classification problem and explicitly notes that dividing videos into 8-frame clips “didn't take the contents of the video into consideration” sufficiently (Fu et al., 2023). Koala inherits the pretrained model’s maximum input token limit and is constrained by learned temporal positional embeddings (Tan et al., 2024). InfoShot itself notes that a fixed two-frames-per-shot allocation may be suboptimal when shot lengths or event density vary widely (Zhao et al., 18 Mar 2026).

Finally, evaluation practice remains heterogeneous. Some works measure downstream QA or recognition accuracy, some measure reconstruction quality, some measure storage compression, and some use custom summarization metrics. In one case, the attention-autoencoder paper reports $\mathbf{Q}$10 as “classification accuracy,” but the detailed table corresponds to the positive-class F1 rather than a conventional overall accuracy (Arslan et al., 2023). This suggests that “key-frame sampler” is best understood as a family of task-coupled temporal compression mechanisms rather than a single benchmarkable primitive.

Across these literatures, the notion of a key-frame sampler has broadened from fixed summarization heuristics to query-conditioned retrieval, shot-aware coverage control, geometric diversity maximization, reconstructability-based selection, and jointly trained sampling policies. The shared technical question is unchanged: how to spend a very small frame budget on the frames that most change what a downstream model can infer.