HiPHer: History-Driven Video Highlighter

• HiPHer is a history-driven, preference-aware video highlighter that predicts segment-wise saliency scores using a user’s watch history.
• It leverages multimodal CLIP features and a cross-attention mechanism to condition segment representations on a global preference embedding.
• A contrastive saliency loss with a fine-grained small margin enables robust ranking, outperforming baselines in personalized video highlighting.

Searching arXiv for “HiPHer” and closely related papers to ground the article. HiPHer is a History-driven Preference-aware Video Highlighter, introduced as the core method in the HIPPO-Video work on personalized video highlighting. It is designed to predict segment-wise saliency scores for a target video conditioned on a user’s watch history, rather than producing generic highlights or conditioning only on a short natural-language query. In the formulation used by the paper, a target video $V$ is divided into segments $s_k$, a session history is represented as $\mathcal{H}=\{H_1,\dots,H_m\}$, and the task is to predict $Y=\{y_1,\dots,y_n\}$, where each $y_k$ reflects the relevance of segment $s_k$ to the specific user inferred from $\mathcal{H}$ (Lee et al., 22 Jul 2025).

1. Definition and task setting

HiPHer is defined within the broader task of personalized video highlighting, which the paper distinguishes from generic highlight detection, video summarization, moment retrieval, and query-focused summarization. Generic highlight detection and video summarization assign importance scores or summaries that are intended to reflect what is salient on average. Query-based approaches condition on a single phrase or sentence. HiPHer instead treats saliency as preference-conditioned: a segment is important if it is salient to a particular viewer, given that viewer’s recent watch history and inferred interests (Lee et al., 22 Jul 2025).

The task is formalized as follows. If a target video $V$ is uniformly divided into $n$ segments,

$V \rightarrow \{s_1,\dots,s_n\},$

and the watch history contains $s_k$0 videos,

$s_k$1

the goal is to predict

$s_k$2

where each $s_k$3 depends jointly on segment content and user preferences derived from $s_k$4 (Lee et al., 22 Jul 2025).

This formulation positions HiPHer at the intersection of user modeling and fine-grained video understanding. A plausible implication is that it extends personalization from the item level, which is common in recommender systems, to the intra-video level, where the objective is not which video to recommend next but which parts of a video are most relevant to a specific viewer.

2. Relation to HIPPO-Video

HiPHer was introduced together with HIPPO-Video, a dataset constructed specifically for preference-conditioned highlighting. The dataset contains 2,040 watch-history / saliency-score pairs, corresponding to 20,400 videos across 170 semantic categories. Each watch history contains 10 videos, and the last video in each history is used as the target video for segment-wise saliency annotation (Lee et al., 22 Jul 2025).

Each training instance therefore includes a watch history

$s_k$5

together with a segmentation of the tenth video into clips $s_k$6 and a saliency score

$s_k$7

for each segment. These scores are not generic annotations. They are assigned using the simulator’s final long-term preferences $s_k$8 and the personal review written after watching the target video, so the labels explicitly encode user-conditioned relevance (Lee et al., 22 Jul 2025).

The dataset uses an LLM-based user simulator that iteratively retrieves candidate videos, selects a “most wanted” and “least wanted” video, produces summaries and personal opinions, and updates preferences in an ADD / REFINEMENT / REMOVAL format over ten turns. This process yields session-level histories with evolving preferences, rather than isolated query–video pairs (Lee et al., 22 Jul 2025). Human verification reported that 97.56% of simulator queries were judged reasonable with Fleiss’ $s_k$9, the simulator’s most-wanted video matched human majority 71.42% of the time, and approximately 98% of saliency pairs were considered reasonable by majority agreement (Lee et al., 22 Jul 2025).

HiPHer is therefore best understood not as a standalone highlighting heuristic but as the model counterpart to a new data formulation. The method derives its input signal from full multimodal watch histories rather than from a single query string.

3. Input representation and preference modeling

HiPHer takes as input a target video $\mathcal{H}=\{H_1,\dots,H_m\}$0, segmented into $\mathcal{H}=\{H_1,\dots,H_m\}$1 segments, together with a watch history $\mathcal{H}=\{H_1,\dots,H_m\}$2. For each target segment $\mathcal{H}=\{H_1,\dots,H_m\}$3, the representation consists of a representative frame $\mathcal{H}=\{H_1,\dots,H_m\}$4 and a transcript $\mathcal{H}=\{H_1,\dots,H_m\}$5 obtained via ASR. Feature extraction is performed with CLIP: a pre-trained CLIP image encoder produces a visual feature

$\mathcal{H}=\{H_1,\dots,H_m\}$6

and a CLIP text encoder produces a textual feature

$\mathcal{H}=\{H_1,\dots,H_m\}$7

These are concatenated to form the segment representation

$\mathcal{H}=\{H_1,\dots,H_m\}$8

The paper notes that CLIP dimensions are typically 512, giving a concatenated dimension of 1024 (Lee et al., 22 Jul 2025).

History videos are encoded in the same manner. For each history video $\mathcal{H}=\{H_1,\dots,H_m\}$9, whose segments are $Y=\{y_1,\dots,y_n\}$0, HiPHer constructs a single video-level embedding by mean pooling over segments: $Y=\{y_1,\dots,y_n\}$1 with

$Y=\{y_1,\dots,y_n\}$2

The global user preference embedding is then formed by mean pooling across history videos: $Y=\{y_1,\dots,y_n\}$3 and concretely

$Y=\{y_1,\dots,y_n\}$4

Thus, HiPHer represents user preference as a single vector summarizing the average multimodal content of the watch history (Lee et al., 22 Jul 2025).

The authors explicitly note that more advanced aggregation schemes could weight videos differently, but HiPHer uses simple mean pooling. This suggests that the method’s empirical performance derives less from a sophisticated history encoder than from the introduction of history-conditioned supervision and cross-modal conditioning.

4. Architecture and conditioning mechanism

Before preference conditioning, both the segment representations $Y=\{y_1,\dots,y_n\}$5 and the preference embedding $Y=\{y_1,\dots,y_n\}$6 are passed through projection layers to align them in a shared embedding space. The projection layers are described as three sequential layers of LayerNorm and Dropout, with an implied standard linear projection structure. This yields projected segment representations $Y=\{y_1,\dots,y_n\}$7 and a projected preference embedding $Y=\{y_1,\dots,y_n\}$8 (Lee et al., 22 Jul 2025).

The core personalization mechanism is a cross-attention layer in which the target segments act as queries and the preference embedding provides keys and values. In the paper’s description, segment representations query the preference vector to obtain preference-conditioned segment features: $Y=\{y_1,\dots,y_n\}$9 Denoting the resulting representation by

$y_k$0

the intent is that each segment representation becomes conditioned on the global preference vector rather than being scored in isolation (Lee et al., 22 Jul 2025).

After cross-attention, the sequence of conditioned segment embeddings $y_k$1 is fed into a Transformer encoder with multi-head self-attention and feed-forward layers. This models contextual dependencies among segments, analogous to transformer-based video understanding models. If the encoder outputs $y_k$2 for segment $y_k$3,

$y_k$4

then a prediction head maps each $y_k$5 to a scalar saliency score,

$y_k$6

The paper does not specify a more elaborate scoring head than a standard regression mapping (Lee et al., 22 Jul 2025).

A concise summary of the architecture is given below.

Component	Function	Formulation
Segment encoding	Encode target segments from frame and transcript	$y_k$7
History aggregation	Build user preference vector from watched videos	$y_k$8
Preference conditioning	Inject user preference into segment representations	$y_k$9
Context modeling	Model inter-segment structure	$s_k$0

From an architectural perspective, HiPHer is comparatively simple. It does not explicitly separate short-term from long-term interests, nor does it use a recurrent or transformer history encoder over the watch sequence itself. Instead, it relies on mean-pooled history embeddings and a lightweight conditioning pathway.

5. Loss function and training objective

HiPHer is trained with a contrastive saliency loss formulated as a pairwise hinge ranking objective. Let $s_k$1 and $s_k$2 denote positive and negative segment feature vectors, with predicted scores $s_k$3 and $s_k$4, respectively. The loss is

$s_k$5

If the positive segment already scores at least $s_k$6 higher than the negative segment, the loss is zero; otherwise the model is penalized in proportion to the violation (Lee et al., 22 Jul 2025).

The paper states that prior work often uses $s_k$7, but HiPHer ablates the margin and finds that smaller values in the range $s_k$8 work better, because they allow more fine-grained ranking among segments (Lee et al., 22 Jul 2025). This is a salient design choice because the saliency labels lie on a $s_k$9–$\mathcal{H}$0 scale rather than a binary highlight/non-highlight axis. A plausible implication is that coarse large-margin ranking can blur distinctions among moderately relevant segments, whereas a smaller margin better fits gradual preference-conditioned variation.

Training uses the same scene-based segmentation regime as the dataset construction pipeline. Inputs are CLIP-derived multimodal features, and the trainable components are the projectors, cross-attention layer, transformer encoder, and scoring head (Lee et al., 22 Jul 2025).

6. Evaluation, baselines, and empirical findings

HiPHer is evaluated on HiPpo-Video and, for baseline context, on QVHighlights. The evaluation covers highlight detection (HD), moment retrieval (MR), video summarization (VS), and the paper’s personalized highlighting setting, using common metrics such as mAP, Hit@1, Recall@1@IoU, F1, and RMSE (Lee et al., 22 Jul 2025).

For HD, positives are defined by thresholding segment saliency scores, and Hit@1 is reported for thresholds $\mathcal{H}$1. For MR, Recall@1 is computed at IoU thresholds 0.5 and 0.7. For VS, F1 is computed against ground-truth salient segments. Score prediction quality is measured using

$\mathcal{H}$2

These metrics allow HiPHer to be compared directly with both generic and query-based baselines (Lee et al., 22 Jul 2025).

The baselines include SL-Module, Moment-DETR, UMT, QD-DETR, UVCOM, TR-DETR, CLIP-It, and VSL. Query-based baselines are adapted to HiPpo-Video by generating queries from watch history using key phrases that summarize user history (Lee et al., 22 Jul 2025).

On the simulated HiPpo-Video benchmark, the reported results are:

Method	RMSE	mAP	Hit1@7	Hit1@9	[email protected]	[email protected]
Best baseline noted in text	—	0.732	—	—	0.398	0.210
HiPHer	0.301	0.766	0.507	0.166	0.452	0.245

The paper specifies that UMT is the best baseline for mAP at 0.732, Moment-DETR reaches 0.398 on [email protected], and UMT reaches 0.210 on [email protected], so HiPHer improves by +0.034, +0.039, and +0.035 on those three metrics, respectively (Lee et al., 22 Jul 2025).

On the real-history subset of 40 histories, HiPHer is compared to Moment-DETR, QD-DETR, and TR-DETR. The reported results are:

Method	RMSE	H1@7	H1@9	[email protected]
Moment-DETR	0.419	0.472	0.389	0.417
QD-DETR	0.446	0.444	0.361	0.385
TR-DETR	0.443	0.306	0.250	0.429
HiPHer	0.427	0.486	0.400	0.624

Although the RMSE is not the lowest in this table, HiPHer achieves the best H1@7, H1@9, and especially [email protected] = 0.624, which the paper interprets as much better personalized highlight segmentation under real user behavior (Lee et al., 22 Jul 2025).

In summarization ablations, CLIP-It without a query obtains F1@5 = 0.564 and F1@7 = 0.211, phrase-query CLIP-It obtains 0.566 and 0.230, sentence-query CLIP-It obtains 0.658 and 0.234, VSL obtains 0.466 and 0.187, while HiPHer reaches 0.726 and 0.486 (Lee et al., 22 Jul 2025). The relative margin over the best CLIP-It setting is especially large at the stricter $\mathcal{H}$3 threshold.

7. Ablations, interpretation, and limitations

Ablation studies isolate the role of history length and modality. When the preference embedding is constructed from only the last $\mathcal{H}$4 history videos, performance improves monotonically as more history is included (Lee et al., 22 Jul 2025). This is one of the clearest empirical findings: longer histories reveal more stable preferences, even though HiPHer encodes them with simple mean pooling.

The modality ablation compares HiPHer-V (vision only), HiPHer-T (text only), and full HiPHer. The reported values are:

Variant	mAP	H1@7	[email protected]
HiPHer-V	0.67	0.12	0.32
HiPHer-T	0.74	0.15	0.39
HiPHer	0.77	0.17	0.45

The text-only variant outperforms the vision-only variant, and the multimodal model is best overall (Lee et al., 22 Jul 2025). This suggests that ASR transcripts are particularly informative for preference modeling, especially when user interests are semantic or topical rather than primarily visual.

Qualitative case studies show that for the same target video, different watch histories lead HiPHer to generate different highlight profiles (Lee et al., 22 Jul 2025). This behavior is central to the method’s claim: the model is not simply learning generic highlight priors but using history-derived preferences to modulate saliency.

The paper also identifies several limitations. First, the dataset is simulated using an LLM-based user model, so biases in the simulator can propagate to both labels and learned behavior. Second, HiPHer’s preference modeling is intentionally simple: it uses mean pooling over segments and videos, does not distinguish short- and long-term interests explicitly, and does not model users across multiple sessions (Lee et al., 22 Jul 2025). Third, the framework is currently tied to YouTube-derived content distributions. A plausible implication is that deployment on other domains, such as short-form video or educational platforms, would require revalidation or adaptation.

Future directions proposed or implied by the paper include non-uniform history aggregation, explicit modeling of short-term versus long-term preferences, sequence models over history, improved simulators, multi-session personalization, and integration with recommender systems that jointly predict the next video and highlight segments within it (Lee et al., 22 Jul 2025).

Taken together, HiPHer establishes a formulation in which personalization is encoded not by a single query but by a multimodal history-derived preference embedding. Its technical contribution is comparatively modest in architectural complexity, but its significance lies in defining a new supervision regime for history-driven, preference-aware, segment-level video understanding (Lee et al., 22 Jul 2025).