Sali4Vid: Saliency-Aware Dense Video Captioning

• Sali4Vid is a dense video captioning framework that jointly addresses event localization and caption generation via timestamp-derived saliency cues.
• The framework uses sigmoid-based reweighting to emphasize key frames, improving temporal precision and overall caption quality.
• It segments videos adaptively by semantic similarity to retrieve relevant captions, achieving state-of-the-art results on YouCook2 and ViTT.

Sali4Vid is a saliency-aware framework for dense video captioning that addresses event localization and event captioning jointly in untrimmed video by improving both video-side supervision and retrieval-side alignment. It introduces Saliency-aware Video Reweighting, which converts timestamp annotations into sigmoid-based frame importance weights, and Semantic-based Adaptive Caption Retrieval, which segments videos by frame similarity to capture scene transitions before retrieving captions. In the reported evaluation, the framework achieves state-of-the-art results on YouCook2 and ViTT, indicating that jointly improving temporal weighting and retrieval can improve dense video captioning quality (Jeon et al., 4 Sep 2025).

1. Problem formulation and target limitations

Dense video captioning requires a model to determine the temporal boundaries of multiple events and to generate a descriptive sentence for each event. Relative to ordinary video captioning, the task is complicated by long, untrimmed videos, adjacent events with weak separation, and the need to align each caption with the correct temporal segment. Sali4Vid is situated within end-to-end dense video captioning systems such as Vid2Seq, CM², and HiCM², where timestamps and captions are generated jointly (Jeon et al., 4 Sep 2025).

The framework is organized around two limitations identified in prior methods. First, timestamp supervision is often used only on the text side: timestamps are encoded as tokens for the decoder, while the video features themselves are treated uniformly over time. Second, retrieval-augmented methods often retrieve captions from fixed-size video chunks, which can ignore scene transitions and therefore mix unrelated actions or split coherent events across chunk boundaries. Sali4Vid addresses these two issues with a saliency-based weighting module for frame features and an adaptive retrieval module based on semantic segmentation of the video (Jeon et al., 4 Sep 2025).

A central conceptual point is that Sali4Vid does not require an externally annotated saliency dataset. Its “saliency” signal is derived from ground-truth event timestamps and used to construct continuous frame weights during training. This places it within a family of temporally supervised dense video captioning methods rather than classical human-fixation saliency prediction.

2. End-to-end architecture and data flow

Sali4Vid is built on a Vid2Seq-style architecture. The inputs comprise video frames extracted at $1$ FPS and padded or subsampled to fixed length $T=100$, frame-level visual features from CLIP ViT-L/14, speech transcript features with time tokens, and ground-truth timestamps and captions during training (Jeon et al., 4 Sep 2025).

The pipeline proceeds as follows. Frame features $spat=\{spat_i\}_{i=1}^T$ are extracted and encoded with a temporal transformer into context-aware features $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$. During training, timestamp-guided saliency weights are applied to reweight the video features. In parallel, the video is segmented by semantic changes in frame similarity, and top-$k$ captions are retrieved for each semantic segment from an external datastore. The decoder then receives the concatenation of reweighted video features, segment-level retrieved caption embeddings, and transcript features, and training uses cross-entropy on the target output sequence (Jeon et al., 4 Sep 2025).

At inference time, timestamp supervision is unavailable, so the explicit reweighting is not applied directly. The reported interpretation is that the training-time weighting improves the learned representation, while adaptive retrieval continues to provide segment-level guidance. This training/inference asymmetry is one of the distinctive features of the method: the supervision is stronger during learning than at test time, but the model is expected to retain the representational bias learned from timestamp-derived saliency.

3. Saliency-aware Video Reweighting

The first core module, Saliency-aware Video Reweighting, uses timestamp annotations not only to supervise the decoder output sequence but also to define continuous importance weights over video frames. For each annotated event $n$ with start and end times $(t_n^s,t_n^e)$, Sali4Vid defines

$$W_n^L(i)=\text{Sigmoid}\left(\alpha\left(\frac{i}{T}-\frac{t_n^s}{d}\right)\right),$$

$$W_n^R(i)=\text{Sigmoid}\left(\alpha\left(\frac{t_n^e}{d}-\frac{i}{T}\right)\right),$$

$W_n(i)=W_n^L(i)\times W_n^R(i),$

where $T=100$0 is the frame index, $T=100$1 is the total number of frames, $T=100$2 is video duration, and $T=100$3 controls the sigmoid sharpness. If multiple events are present, the frame importance is

$T=100$4

This yields a smooth, center-skewed temporal profile: low before the event, high inside the event, and low after the event, with soft transitions around boundaries (Jeon et al., 4 Sep 2025).

The reweighting step is then applied directly to frame features:

$T=100$5

The resulting reweighted feature sequence $T=100$6 is passed to the decoder. The stated effect is to emphasize frames inside event spans, suppress irrelevant frames, and improve both localization and caption generation. Because the supervision signal is the ground-truth timestamp annotation $T=100$7, no additional saliency labels are required (Jeon et al., 4 Sep 2025).

The ablation findings further characterize this design. Sigmoid weighting outperforms hard binary masks and Gaussian weighting. The reported interpretation is that binary masks are too sharp and lose boundary nuance, whereas Gaussian weights are smooth but less aligned with explicit event boundaries. The center-skew design also outperforms start-skew, end-skew, and random-skew variants, which is consistent with instructional-video settings in which event centers carry strong descriptive content while boundary information still matters.

4. Semantic-based Adaptive Caption Retrieval

The second core module, Semantic-based Adaptive Caption Retrieval, replaces fixed-size retrieval chunks with segments defined by semantic transitions. Given spatial frame features $T=100$8, Sali4Vid first computes a frame-wise change signal

$T=100$9

where $spat=\{spat_i\}_{i=1}^T$0 is cosine similarity. Small $spat=\{spat_i\}_{i=1}^T$1 indicates adjacent frames with similar semantics; large $spat=\{spat_i\}_{i=1}^T$2 indicates a likely scene or event transition (Jeon et al., 4 Sep 2025).

Rather than declaring a boundary whenever $spat=\{spat_i\}_{i=1}^T$3 exceeds a fixed threshold, the method uses momentum-based accumulation and an adaptive threshold. A running segment representation $spat=\{spat_i\}_{i=1}^T$4 is initialized from the first frame, and for each new frame $spat=\{spat_i\}_{i=1}^T$5 the method computes

$spat=\{spat_i\}_{i=1}^T$6

The adaptive threshold is

$spat=\{spat_i\}_{i=1}^T$7

where $spat=\{spat_i\}_{i=1}^T$8 and $spat=\{spat_i\}_{i=1}^T$9 are the mean and standard deviation of all $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$0, and $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$1 is a scaling factor. If $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$2, the current segment ends at $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$3, a new segment begins at $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$4, and the segment feature is reset. Otherwise, frame $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$5 is added to the current segment, and the running representation is updated by moving average:

$\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$6

This design is described as more robust than isolated thresholding because it requires sustained evidence of change, thereby reducing over-segmentation (Jeon et al., 4 Sep 2025).

After segmentation, each segment $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$7 is represented by the average of its frame features:

$\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$8

Caption retrieval is then performed segment-wise. For each segment $\mathbf{x}=\{\mathbf{x}_i\}_{i=1}^T$9, the top-$k$0 captions are retrieved from an external datastore $k$1 via

$k$2

and the retrieved caption embeddings are average-pooled:

$k$3

These per-segment caption vectors form $k$4. The reported motivation is that adaptive segmentation aligns retrieval with real semantic changes, reduces mixing of unrelated events, and improves caption relevance relative to fixed-size chunk retrieval (Jeon et al., 4 Sep 2025).

5. Optimization, empirical results, and ablations

The training objective is standard cross-entropy conditioned on the reweighted video features, segment-level retrieved captions, and transcript features:

$k$5

where $k$6 is the target output sequence and $k$7 denotes model parameters (Jeon et al., 4 Sep 2025).

The reported evaluations use YouCook2 and ViTT. YouCook2 contains 2,000 instructional videos, with average duration around 320s and approximately 7.7 localized sentences/video. ViTT contains 8,000 instructional videos, with average duration around 250s and approximately 7.1 localized tags/video. Captioning metrics are CIDEr, METEOR, SODA_c, and BLEU4. Localization is measured by F1, Recall, and Precision averaged over IoU thresholds $k$8 (Jeon et al., 4 Sep 2025).

Dataset / setting	Prior best	Sali4Vid
YouCook2 validation, CIDEr	71.84 (HiCM²)	75.80
YouCook2, F1	32.51 (HiCM²)	33.61
ViTT test, CIDEr	51.29 (HiCM²)	53.87
ViTT, F1	46.21 (Vid2Seq)	46.58

On YouCook2 validation, Sali4Vid reports 75.80 CIDEr, a gain of +3.96 CIDEr over the previous best. It also reports 13.54 METEOR and 6.35 BLEU4, both best, while 10.28 SODA_c is not the best because HiCM² reports 10.73. On ViTT test, Sali4Vid reports 53.87 CIDEr, improving on HiCM² by +2.58 CIDEr. Additional ViTT results are 10.05 METEOR, 15.08 SODA_c, and 0.91 BLEU4; the SODA_c score is slightly above HiCM²’s 15.07. The localization results show improved precision as well, which the paper interprets as evidence that the model focuses better on relevant temporal regions (Jeon et al., 4 Sep 2025).

The ablations attribute the gains to both major modules. For reweighting, sigmoid weighting is best by a large margin compared with binary and Gaussian alternatives. For retrieval, adaptive threshold + momentum-based accumulation is the strongest design, outperforming fixed-size retrieval, fixed thresholds, and adaptive thresholding without momentum. The best performance occurs around 10 retrieved captions per segment; 5, 20, and 30 are similar but slightly worse. The datastore study reports that the method remains robust with in-domain captions, COCO, CC3M, and hierarchical memory; in-domain retrieval is best for F1, whereas hierarchical memory improves CIDEr and SODA_c. Component ablations show that adaptive retrieval alone improves the baseline, saliency reweighting alone gives a larger boost, and combining both yields the best overall performance (Jeon et al., 4 Sep 2025).

The paper also reports lightweight runtime costs for the retrieval side: retrieval is around 2 ms/video for a reduced caption subset, and segmentation takes about 16 ms/video using a model-free similarity-based algorithm. Two limitations are explicitly noted: reweighting depends on timestamp annotations and is therefore not directly applicable to weakly supervised or annotation-free settings, and adaptive retrieval can still produce noisy segments in some cases.

6. Relation to adjacent saliency-oriented video methods

The term “saliency” spans several distinct problem formulations in recent video research. In Sali4Vid, saliency is derived from event timestamps and used to reweight frame features for dense video captioning. By contrast, DiffSal frames audio-visual saliency prediction as a conditional denoising diffusion process that predicts human fixation or saliency maps from synchronized audio and video. Its model combines a video encoder, an audio encoder, and a Saliency-UNet, and it reports an average relative improvement of 6.3\% over the previous state-of-the-art results by six metrics across six audio-visual saliency benchmarks: AVAD, Coutrot1, Coutrot2, DIEM, ETMD, and SumMe (Xiong et al., 2024).

A different usage appears in Sali-Cache, described as a proactive KV-cache optimization framework for long-form video understanding in vision-LLMs. There saliency is part of a memory-management mechanism: a temporal filter based on optical flow detects inter-frame redundancy, and a spatial filter based on saliency detection assigns patches to FP16, INT8, INT4, or pruning before attention computation. In experiments on LLaVA 1.6, the method reports 16.70 GB peak VRAM, 1.45 GB cache, and 2.20× compression, while maintaining 100% accuracy on BLEU, ROUGE-L, and Exact Match (Sai et al., 15 Feb 2026).

This broader comparison suggests that “saliency” in video research is not a single task definition. It may refer to timestamp-derived temporal relevance for dense captioning, predicted fixation maps in audio-visual saliency prediction, or heuristics for memory allocation in long-context vision-language inference. Within that landscape, Sali4Vid is specifically a dense video captioning framework whose main contribution is to connect timestamp supervision with frame-level weighting and to replace fixed retrieval chunks with semantically adaptive retrieval units.