Generative Keyframe Captioning Methods

• Generative keyframe captioning is a video-to-language method that selects key frames with temporal context to generate detailed, temporally coherent narratives.
• It employs modular pipelines, progress-aware frame captioning, and codec-inspired anchor–residual strategies to balance static scene description with dynamic changes.
• These methods improve descriptive quality by preserving event progression and fine-grained visual evidence while mitigating redundancy compared to global video captioning.

Searching arXiv for recent and foundational papers on generative keyframe/video captioning to ground the article. Generative keyframe captioning is a family of video-to-language methods in which captions are generated from temporally selected frames or frame-derived semantic anchors, rather than from a single global video representation. Across the literature summarized here, the central objective is to preserve event progression and fine-grained visual evidence while avoiding the collapse of an entire clip into one coarse sentence. The field spans modular pipelines that caption representative segment frames and stitch them into narratives, end-to-end encoder–decoder pretraining for multimodal caption generation, frame-level progress-aware captioning, and codec-inspired dense captioning that treats anchors as semantic I-frames and residuals as semantic P-frames (Shin et al., 2016, Seo et al., 2022, Xue et al., 2024, Lin et al., 26 May 2026).

1. Conceptual scope and problem formulation

Generative keyframe captioning occupies an intermediate regime between image captioning and whole-video captioning. Image captioning generates descriptions for isolated images or frames and therefore has no temporal context. Whole-video captioning, by contrast, typically aggregates all video features into one global representation and outputs a single sentence. The works considered here treat that dichotomy as inadequate for dynamic visual content: single-sentence systems are compact but coarse, while isolated frame descriptions can miss temporal progression (Shin et al., 2016, Xue et al., 2024).

Within this regime, the captioning unit is not necessarily a uniformly sampled frame. One line of work uses event-like temporal segments obtained by action localization and captions the middle frame of each segment. Another line generates a caption for each frame in a short sequence while requiring temporal specificity and progressive coherence. A later codec-inspired line represents each scene-aligned segment with one exhaustive anchor caption plus residual captions that describe only localized changes (Shin et al., 2016, Xue et al., 2024, Lin et al., 26 May 2026).

A concise comparison of the main representational choices is useful.

Work	Generation unit	Organizational principle
"Beyond Caption To Narrative: Video Captioning With Multiple Sentences" (Shin et al., 2016)	Middle frame of each temporal segment	Multi-sentence narrative with coreference and connectives
"Progress-Aware Video Frame Captioning" (Xue et al., 2024)	Each frame in a short sequence	Progress-aware frame-level captions
"CodecCap: High-Fidelity Codec-Inspired Residual Modeling for Dense Video Captioning" (Lin et al., 26 May 2026)	Anchor caption plus per-second residual captions	Four-level hierarchy: anchor, residual, scene, video

A recurring misconception is that keyframe-based captioning is merely keyframe selection followed by tags or sparse summaries. The codec-inspired formulation explicitly rejects that view: the keyframe caption is a generated semantic I-frame that encodes stable visual state, and later descriptions are generated as P-frame-like residuals relative to it (Lin et al., 26 May 2026).

2. Segmented frame captioning and narrative assembly

An early explicit realization of generative keyframe captioning appears in a modular pipeline for multiple-sentence video captioning. The method first temporally segments the video using action localization rather than shot or scene boundary detection, on the grounds that shot/scene boundary detection is not reliable for all video domains and may not correspond to actual event changes. It uses improved Dense Trajectories motion features, a sliding temporal window approach for action detection, and a classifier trained on UCF101. The sliding windows have lengths $30, 60, 90, 120$ frames with step size $30$ frames. Candidate windows are rescored with non-maximum suppression, a score threshold is applied, and adjacent windows of the same predicted action class are grouped into segments. If no action is detected, the system uses the middle frame (Shin et al., 2016).

Caption generation is then performed on the representative frame of each segment. For each temporal segment, the middle frame is extracted; VGG fc7 features of dimension 4096 are fed to an LSTM trained on MS COCO, with beam size $1$. The paper explicitly states that video-level features are not used for captioning, only for segmentation. This makes the generation step a repeated image-captioning process over temporally selected frames rather than a captioning process over one aggregated video descriptor (Shin et al., 2016).

The narrative quality of the output depends on discourse-level post-processing. The method applies backward coreference resolution so that later repeated mentions can be replaced by pronouns when appropriate. Its implementation uses the Stanford CoreNLP gender annotator for singular human subjects, coreference resolution for non-singular or non-human references, POS tagging for plurals, lemmatization, and WordNet assistance. It then inserts connective words using an unsupervised nearest-neighbor method built from 500 sentence pairs from tagged Wikicorpus, a CKY parser with a PCFG trained on Penn Treebank, Sentence2Vec embeddings, and nearest-neighbor retrieval by $L2$ distance: $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$ The final output is a multi-sentence narrative in temporal order (Shin et al., 2016).

Quantitatively, the method improves over a middle-frame captioning baseline on Montreal and MPII, and improves BLEU and METEOR on an MS Video Description Corpus subset while CIDEr is slightly lower than baseline on that subset. On Montreal, the baseline scores are BLEU $0.003$, METEOR $0.070$, CIDEr $0.042$, while the proposed method reports BLEU $0.004$, METEOR $0.089$, CIDEr $30$0. On MPII, the baseline scores are BLEU $30$1, METEOR $30$2, CIDEr $30$3, while the proposed method reports BLEU $30$4, METEOR $30$5, CIDEr $30$6. On the MS Video Description Corpus subset, the baseline scores are BLEU $30$7, METEOR $30$8, CIDEr $30$9, while the proposed method reports BLEU $1$0, METEOR $1$1, CIDEr $1$2 (Shin et al., 2016).

3. End-to-end generative pretraining for sentence generation from visual context

A separate but closely related development is end-to-end generative pretraining for multimodal video captioning. The importance of this line is architectural: it addresses the fact that prior video-language pretraining frameworks often optimize discriminative objectives and therefore do not train a sentence generator. MV-GPT instead jointly trains both a multimodal video encoder and a sentence decoder, which is directly relevant wherever keyframe captioning requires an actual conditional generator rather than only an encoder or a retrieval model (Seo et al., 2022).

The architecture has four components: a text encoder, a visual encoder, a multimodal encoder, and a sentence decoder. Text is encoded with BERT-Base. Raw pixels are encoded with ViViT-Base using tubelet embeddings and a factorized spatial-then-temporal transformer. The streams are fused by a co-attentional transformer similar to ViLBERT / Look-Before-You-Speak, producing multimodal context $1$3. The decoder is a GPT-2-style autoregressive transformer conditioned on that multimodal context, and predicts the next token as

$1$4

The paper emphasizes that the visual encoder can work with coarsely sampled frames at $1$5 fps, which is important for efficiency and end-to-end training (Seo et al., 2022).

Its pretraining signal is a bidirectional utterance generation objective built from triplets of frames $1$6, present utterance $1$7, and future utterance $1$8. Forward generation predicts the future utterance from the current clip context: $1$9 while backward generation predicts the present utterance from future utterance and current frames: $L2$0 Masked language modeling is added on decoder outputs, and the effective objective combines forward generation, backward generation, MLM losses, and regularization (Seo et al., 2022).

For caption generation, the decoder is initialized with a BOS token and uses beam search with beam size $L2$1. The model is pretrained on HowTo100M and evaluated on YouCook2, ViTT, MSR-VTT, and ActivityNet-Captions using BLEU-4, CIDEr, METEOR, and ROUGE-L, with BLEU-1 used for ViTT. Reported results include $L2$2 B-4 / $L2$3 C / $L2$4 M / $L2$5 R-L on YouCook2, $L2$6 B-1 / $L2$7 C / $L2$8 M / $L2$9 R-L on ViTT, $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$0 B-4 / $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$1 C / $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$2 M / $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$3 R-L on MSR-VTT, and $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$4 B-4 / $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$5 M on ActivityNet-Captions (Seo et al., 2022).

A plausible implication is that generative keyframe captioning benefits not only from choosing the right temporal units, but also from pretraining the decoder itself as a multimodal generator. The paper explicitly notes that the most transferable part for keyframe captioning is the encoder–decoder generative pretraining framework, while the exact future-utterance mechanism depends on temporal ordering and nearby speech (Seo et al., 2022).

4. Progress-aware frame-level captioning

Progress-aware video frame captioning formulates the problem at frame level while retaining temporal dependence. The task maps a frame sequence

$\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$6

to a caption sequence

$\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$7

with three explicit requirements: accuracy, temporal specificity, and progressive coherence. The motivation is that image captioners treat each frame independently and may miss what is changing from one frame to the next, whereas whole-video captioners compress the clip into one sentence that is too coarse to capture stages such as “still intact,” “starting to cook,” and “fully mixed” (Xue et al., 2024).

The proposed model, ProgressCaptioner, is a 7B VLM initialized from LLaVA-OV-7B, using SigLIP as the vision encoder, Qwen2 as the LLM, and a projector to connect vision and language modules. The method is trained in two stages. Stage I focuses on two-frame inputs $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$8, motivated by the observation that feeding too many frames causes brief captions, memory issues, and temporal mismatches, whereas single-frame captioning loses temporal context. Candidate caption pairs are generated by multiple VLMs and filtered by two automatic critics: progression detection and caption matching. Accepted pairs are used for supervised fine-tuning, and preference triples are used for direct preference optimization. Stage II extends the same strategy to short sequences of length $\arg\min_i \| \mathbf{v}_{gen} - \mathbf{v}_i \|_2$9 to $0.003$0, again with SFT on accepted pseudo-labels and DPO on preference data (Xue et al., 2024).

The training resource, FrameCap, is built from HowToChange and COIN. Frames are sampled at $0.003$1 FPS; Stage I uses frame pairs and Stage II uses short multi-frame sequences of length $0.003$2 to $0.003$3. Candidate captions are generated by VILA, Qwen2-VL, LLaVA-Next-Video, LLaVA-Video, and LLaVA-OV, then filtered by progression detection and caption matching. Reported statistics are 7,812 videos and 101,369 frames from HowToChange, 9,030 videos and 103,791 frames from COIN, for totals of 240,241 frame-caption pairs for SFT, 21,075 preference pairs for DPO, 34,306 multi-frame sequences for SFT, and 26,188 sequence preference examples for DPO (Xue et al., 2024).

The evaluation benchmark, FrameCapEval, draws videos from HowToChange, COIN, Penn Action, and Kinetics / Kinetics600. Frames are extracted at $0.003$4 FPS, sequences are grouped with K-means on CLIP features, the number of clusters $0.003$5 is chosen using silhouette scores, and final sequence lengths range from $0.003$6 to $0.003$7. Sequences without fine-grained progression are manually filtered out. The supplement reports 306 videos from HowToChange, 271 from COIN, 51 from Penn Action, and 56 from Kinetics600, for a total of 684 videos. Evaluation uses balanced accuracy for progression detection and sequence-level accuracy for caption matching, and includes a user study with 15 participants (Xue et al., 2024).

Reported results show ProgressCaptioner achieving on HowToChange caption matching $0.003$8 and progression detection $0.003$9, on COIN caption matching $0.070$0 and progression detection $0.070$1, and on Penn caption matching $0.070$2 and progression detection $0.070$3. The user study reports the highest best-caption selection rate at $0.070$4. For keyframe selection, the paper describes a sliding two-frame window procedure: caption $0.070$5, ask an LLM whether the caption indicates action progression, keep $0.070$6 if yes, and skip it if no. Replacing uniformly sampled frames with ProgressCaptioner-selected frames in SlowFast and X3D-XS gives about $0.070$7 top-1 accuracy for both models on Kinetics test videos (Xue et al., 2024).

5. Codec-inspired anchor–residual captioning

CodecCap provides the clearest formalization of generative keyframe captioning as a dense captioning framework. Its motivating dilemma is explicit: holistic video captions are compact but often miss fine-grained evidence and temporally localized events, whereas segment-wise dense captions improve coverage but repeatedly restate stable background, objects, and actions, producing heavy redundancy. The proposed solution is to separate stable visual context from localized changes, directly mirroring video codecs (Lin et al., 26 May 2026).

The representation is

$0.070$8

comprising four granularities: keyframe captions $0.070$9, per-second residual captions $0.042$0, scene-level narratives $0.042$1, and a whole-video narrative $0.042$2. Anchors are semantic I-frames that capture scene, entities, attributes, spatial layout, and object states. Residuals are semantic P-frames that capture motion, state transitions, interactions, occlusion, object displacement, and other changes relative to the prior frame (Lin et al., 26 May 2026).

The pipeline has four stages. First, scene-aligned segmentation uses codec metadata from I-frames, content-based shot detection via PySceneDetect, and a GOP-aware strategy to distinguish codec-refresh I-frames from true scene cuts. Second, anchor captioning takes the first sampled frame in each segment as the anchor and generates an exhaustive caption covering environment, entities, visual attributes, object states, and spatial layout. Third, residual captioning operates over sliding temporal windows with delta-only enforcement: each residual is output as a JSON object $0.042$3, multi-frame context is used to avoid mistaking noise for change, and spatial normalization uses a 9-zone grid or normalized coordinates with directions following screen coordinates. If nothing changed, the output can be essentially “No visible change.” Fourth, a text-only LLM aggregates anchor and residual evidence into scene-level paragraphs and a whole-video narrative using structured acceptance rules: continuous changes need support from at least two consecutive residuals, discrete events can be accepted from one residual if context matches, anchor attributes persist unless explicitly updated, contradictions are resolved by support count, and unsupported inferences are omitted (Lin et al., 26 May 2026).

The evaluation framework is centered on caption fidelity rather than only fluency or semantic similarity. VidCapQA starts from 9,443 multiple-choice questions sourced from eight video understanding benchmarks and relabels them into 14 capability dimensions: action recognition, attribute recognition, camera movement, counting, direction, holistic understanding, object tracking, reasoning, rotation, speed, state change, temporal grounding, temporal sequence, and trajectory. Construction uses unknown-aware majority voting for relabeling, text-only leak filtering, visual validation with multiple VLMs, and a stratified 1,000-question final benchmark. On this benchmark, Gemini 3.1 Pro-generated captions achieve only $0.042$4 accuracy, which the paper uses to argue that caption representation is a critical bottleneck (Lin et al., 26 May 2026).

CodecCap is also used to construct CodecVDC-100K, described as the first public dataset with the complete anchor, residual, scene, and video hierarchy. The dataset contains 118,257 videos and 7,962 hours. Each video includes a video-level narrative, a median of 6 scene-aligned segments, an anchor caption of about 270 words per segment, and about 225 one-second residual captions per video, around 25 words each. In experiments, the residual subset is used for supervised fine-tuning of VDCTalker, which fine-tunes Qwen3.5-35B-A3B with ms-swift on 64 Nvidia H800 GPUs, using learning rate $0.042$5 and global batch size 1024. Reported gains over the base model are $0.042$6 overall accuracy on VidCapQA and improvement on 12 of 14 capability dimensions, including trajectory $0.042$7, action recognition $0.042$8, temporal sequence $0.042$9, and direction $0.004$0 (Lin et al., 26 May 2026).

This codec formulation also sharpens the distinction between dense captioning and redundancy. The paper notes that the per-second Seed baseline can score higher, but does so with severe token inflation and redundancy. CodecCap is more token-efficient because static content is anchored once. A plausible implication is that anchor–residual factorization changes the unit of supervision itself: stable scene understanding is learned from anchors, while dynamic evidence is learned from residual deltas (Lin et al., 26 May 2026).

6. Evaluation criteria, representational trade-offs, and recurring misconceptions

The literature shows a clear progression in how caption quality is measured. Early segmented-narrative work relies on standard captioning metrics such as BLEU, METEOR, and CIDEr. MV-GPT is also evaluated with BLEU-4, CIDEr, METEOR, and ROUGE-L. More recent work introduces task-specific evaluation protocols: FrameCapEval measures progression detection with balanced accuracy and caption matching with sequence-level accuracy, while VidCapQA evaluates whether a generated caption preserves enough evidence for a text-only LLM to answer multiple-choice questions and allows the LLM to output unknown when no evidence is present (Shin et al., 2016, Seo et al., 2022, Xue et al., 2024, Lin et al., 26 May 2026).

These evaluation choices reveal a substantive representational trade-off. One misconception is that caption quality is adequately characterized by fluency or semantic overlap with references. VidCapQA directly challenges that assumption by showing that even captions generated by strong VLMs can miss many visual details, and by framing captioning quality as recoverable evidence preservation rather than stylistic adequacy alone (Lin et al., 26 May 2026).

Another misconception is that more temporal density necessarily yields better supervision. The segmented narrative system already had to threshold action windows to avoid too many redundant segments. ProgressCaptioner found that feeding too many frames causes brief captions, memory issues, and temporal mismatches. CodecCap reports that per-second dense baselines can inflate tokens severely and repeat static content. Across these works, the recurring design problem is not simply how to caption more frames, but how to allocate description budget across stable context and change events (Shin et al., 2016, Xue et al., 2024, Lin et al., 26 May 2026).

A further point of contrast concerns temporal unit selection. Action localization treats changes in action as the segmentation cue. Progress-aware frame captioning uses two-frame and short-sequence supervision to distinguish real action progression from hallucinated progression. CodecCap uses scene-aligned segmentation and then models intra-scene change through residual captions. This suggests that “keyframe” is not a single fixed primitive across the literature: it can be a middle frame of an action segment, an element in a progress-aware frame sequence, or a semantic anchor for a scene-aligned segment (Shin et al., 2016, Xue et al., 2024, Lin et al., 26 May 2026).

The recent literature also makes explicit room for future extensions. CodecCap notes that the full hierarchy of CodecVDC-100K can support future joint training and help mitigate weaknesses in persistence-oriented dimensions like speed and state change. MV-GPT states that its bidirectional generation objective could be adapted to keyframe captioning by treating the keyframe or sparse frame set as $0.004$1 and surrounding context as text supervision. A plausible synthesis is that future generative keyframe captioning systems may combine hierarchical anchor–residual representations with end-to-end multimodal generative pretraining (Seo et al., 2022, Lin et al., 26 May 2026).