Frame-wise PickScore: Video Frame Relevance

• Frame-wise PickScore is a metric that assigns scalar relevance scores to video frames based on text-conditioned multimodal embeddings.
• It integrates powerful oracle models like CLIP and efficient student architectures such as PEEK to balance accuracy with computational efficiency.
• The approach leverages listwise ranking loss to optimize frame ordering, enabling adaptive sampling for tasks like video captioning and retrieval.

Frame-wise PickScore is a metric and methodology for evaluating and ranking individual frames of a video with respect to a conditioning textual prompt, based on learned multimodal alignment. It operationalizes a scalar “relevance” score per frame, facilitating adaptive frame selection for downstream tasks such as video captioning and retrieval. The concept is instantiated in both generic CLIP-based systems as well as in efficient distillation architectures such as PEEK, offering both oracle and lightweight proxy models, respectively (Kirstain et al., 2023, Steunou et al., 29 May 2026).

1. Mathematical Formulation and Core Principles

Frame-wise PickScore extends the scalar PickScore concept, originally defined for prompt-image pairs, to the temporally indexed frames of a video segment. Given a fixed text prompt $x$ and a sequence of $N$ video frames $\{f_1,\dots,f_N\}$:

• Embedding computation: The prompt $x$ is tokenized and encoded by a text encoder (e.g., CLIP or SigLIP2) to produce $\mathbf{t}_\mathrm{emb} \in \mathbb{R}^d$. Each frame $f_i$ is preprocessed and encoded by an image encoder to yield $\mathbf{i}_\mathrm{emb}^{(i)} \in \mathbb{R}^d$.
• Frame-wise score: For each frame $i$, a PickScore is computed as

$$s_i = \langle \mathbf{t}_\mathrm{emb}, \mathbf{i}_\mathrm{emb}^{(i)} \rangle \cdot T,$$

where $T \in \mathbb{R}$ is a trainable “temperature” parameter; with some variants using cosine similarity between frame and text embeddings, optionally normalized and subsequently min–max scaled (Kirstain et al., 2023, Steunou et al., 29 May 2026).

• Normalization: PEEK refines the teacher output by min–max normalizing raw cosine similarities $N$0 per video segment, yielding oracle relevance targets $N$1.

This construction delivers a dense, per-frame alignment signal with respect to the textual prompt.

2. Model Architectures: Oracle and Student Variants

The implementation of frame-wise PickScore generally follows one of two paradigms:

A. Oracle models:

Leverage powerful, frozen dual-encoder architectures (CLIP or SigLIP2) to score each frame against the prompt or caption (Kirstain et al., 2023, Steunou et al., 29 May 2026).

• Encoders are typically Vision Transformers and large Transformer-based text encoders with high-dimensional embeddings ($N$2 CLIP, $N$3 SigLIP2).
• Oracle PickScore is computationally intensive and mainly serves as a teacher for distillation.

B. Student architectures (PEEK):

Distill oracle PickScore distributions into compact temporal models (Steunou et al., 29 May 2026).

• Each frame is encoded by a frozen MobileCLIP2 encoder ($N$4).
• Features are projected and combined with positional encodings and layer-norm, then processed by a 2-layer Transformer encoder (hidden dim 256, 4 heads).
• A final linear layer regresses a per-frame logit ($N$5), representing the student PickScore.
• The total trainable parameter count for the student is ≈1.7M, with 13.1M if including MobileCLIP2.
• The student is trained end-to-end to mimic the ranking of oracle scores.

3. Training Objectives and Optimization

Training of frame-wise PickScore systems is formulated to directly optimize frame ranking rather than absolute score regression:

• Listwise Ranking Loss (ListMLE):

Training targets are ordered min–max normalized teacher PickScores $N$6. For each segment, let $N$7 denote the permutation that sorts these in descending order. The ListMLE loss is given by:

$N$8

yielding student parameters that maximize the likelihood of correct frame ranking. - Alternative losses (MSE+pairwise hinge) are empirically inferior (Steunou et al., 29 May 2026).

• Data sampling:

Large-scale training involves batch sizes up to 1024, AdamW optimizer, and comprehensive temporal augmentation (sampling $N$9 uniformly spaced candidate frames per video segment).

4. Inference and Frame Selection

At inference, frame-wise PickScore is used for adaptive frame selection under computational or memory budget constraints:

1. For a new video segment (with caption/prompt), sample $\{f_1,\dots,f_N\}$0 frames.
2. Compute student PickScores $\{f_1,\dots,f_N\}$1 or oracle PickScores where permissible.
3. Given a frame budget $\{f_1,\dots,f_N\}$2 (number of frames to select for downstream processing), partition the sequence into $\{f_1,\dots,f_N\}$3 equal-length bins.
4. Within each bin, select the frame with the highest PickScore (stratified argmax):

$\{f_1,\dots,f_N\}$4

where $\{f_1,\dots,f_N\}$5 is the $\{f_1,\dots,f_N\}$6-th bin.

1. Forward the selected frames $\{f_1,\dots,f_N\}$7 to the next-stage model (e.g., a video captioner).

This approach ensures both relevance and temporal coverage. Optional techniques include temporal smoothing for score stability and early-exit heuristics for efficiency (Kirstain et al., 2023, Steunou et al., 29 May 2026).

5. Computational Complexity and Optimizations

Frame-wise PickScore enables near real-time application through algorithmic and architectural optimizations:

• Prompt embedding: For a fixed prompt, text encoding is performed once ($\{f_1,\dots,f_N\}$8, where $\{f_1,\dots,f_N\}$9 is the token count and $x$0 the embedding size).
• Per-frame cost: Each image/frame requires a forward pass through the encoder ($x$1 is constant for fixed architecture).
• Batch processing: Batching (e.g., 32–64 frames) optimizes hardware throughput (8–12 ms per image for fp16 CLIP at batch-size 32).
• Precision/quantization: Use of fp16 or INT8 quantization further reduces computational burden.
• Distillation: Student models (e.g., in PEEK) drastically reduce inference time (adds +5.2% to end-to-end captioning vs. 65.4% for heavier adaptive methods, on ActivityNet (Steunou et al., 29 May 2026)).
• Temporal caching/interpolation: For similar frames, embeddings or scores may be cached or interpolated to amortize cost, especially in high-frame-rate regimes.

With these strategies, frame-wise PickScore can be deployed at $x$230 fps on single GPUs or high-end CPUs under quantization (Kirstain et al., 2023).

6. Empirical Results and Evaluations

Frame-wise PickScore and its student proxies have been extensively evaluated on video-language tasks:

• Benchmarks: ActivityNet Captions and MSR-VTT, with metrics including CIDEr, BLEU-4, METEOR, ROUGE-L.
• Downstream models: Four vision-LLMs, including SmolVLM2-2.2B and Qwen* variants.
• Selector comparisons: Uniform, random, CSTA (video summarization), MaxInfo (diversity), Oracle (teacher PickScore from SigLIP2).

Findings (ActivityNet, MSR-VTT):

• For low frame budgets ($x$3), PEEK is the highest-performing selector in 14/16 settings on ActivityNet, with CIDEr improvements up to +3.00 for $x$4.
• At higher budgets ($x$5), the benefit narrows, with uniform sampling occasionally matching or exceeding PickScore on MSR-VTT.
• Ablation studies show stratified argmax selection (temporal bins + argmax per bin) outperforms top-$x$6 selection without temporal partitioning.
• ListMLE gives a +1.18 CIDEr margin over point-wise MSE plus pairwise hinge at $x$7.
• Efficiency profiles: PEEK adds only +5.2% runtime overhead, compared with +65.4% (CSTA) and +211.9% (MaxInfo) (Steunou et al., 29 May 2026).
• A plausible implication is that PickScore-based selection is especially advantageous for tight frame budgets and operationally constrained pipelines.

7. Applications and Broader Context

Frame-wise PickScore enables:

• Adaptive frame sampling for video-captioning and other video-language tasks, optimizing both informativeness and temporal coverage under memory or latency constraints.
• Model evaluation and ranking, by providing a human-aligned metric for relevance.
• Efficient knowledge distillation, where expensive oracle models transfer temporal relevance priors to compact and fast student networks.

Originally defined for text-to-image evaluation (Kirstain et al., 2023), the frame-wise paradigm broadens its impact in video understanding, operationalizing a learned, text-conditioned selection mechanism that generalizes across benchmarks and model scales (Steunou et al., 29 May 2026).