---
title: 'HiPHer: History-Driven Video Highlighter'
url: https://www.emergentmind.com/topics/hipher
type: topic
---

# HiPHer: History-Driven Video Highlighter

Searching arXiv for “HiPHer” and closely related papers to ground the article.
HiPHer is a **History-driven Preference-aware Video Highlighter**, introduced as the core method in the HIPPO-Video work on personalized video highlighting. It is designed to predict **segment-wise saliency scores** for a target video conditioned on a user’s **watch history**, rather than producing generic highlights or conditioning only on a short natural-language query. In the formulation used by the paper, a target video \(V\) is divided into segments \(s_k\), a session history is represented as \(\mathcal{H}=\{H_1,\dots,H_m\}\), and the task is to predict \(Y=\{y_1,\dots,y_n\}\), where each \(y_k\) reflects the relevance of segment \(s_k\) to the specific user inferred from \(\mathcal{H}\) [2507.16873].

## 1. Definition and task setting

HiPHer is defined within the broader task of **personalized video highlighting**, which the paper distinguishes from generic highlight detection, video summarization, moment retrieval, and query-focused summarization. Generic highlight detection and video summarization assign importance scores or summaries that are intended to reflect what is salient on average. Query-based approaches condition on a single phrase or sentence. HiPHer instead treats saliency as **preference-conditioned**: a segment is important if it is salient to a particular viewer, given that viewer’s recent watch history and inferred interests [2507.16873].

The task is formalized as follows. If a target video \(V\) is uniformly divided into \(n\) segments,
\[
V \rightarrow \{s_1,\dots,s_n\},
\]
and the watch history contains \(m\) videos,
\[
\mathcal{H}=\{H_1,\dots,H_m\},
\]
the goal is to predict
\[
Y = \{y_1,\dots,y_n\},
\]
where each \(y_k\) depends jointly on segment content and user preferences derived from \(\mathcal{H}\) [2507.16873].

This formulation positions HiPHer at the intersection of user modeling and fine-grained video understanding. A plausible implication is that it extends personalization from the item level, which is common in recommender systems, to the **intra-video** level, where the objective is not which video to recommend next but which parts of a video are most relevant to a specific viewer.

## 2. Relation to HIPPO-Video

HiPHer was introduced together with **HIPPO-Video**, a dataset constructed specifically for preference-conditioned highlighting. The dataset contains **2,040** watch-history / saliency-score pairs, corresponding to **20,400 videos** across **170 semantic categories**. Each watch history contains **10 videos**, and the **last video** in each history is used as the target video for segment-wise saliency annotation [2507.16873].

Each training instance therefore includes a watch history
\[
\mathcal{H} = \{H_1,\dots,H_{10}\},
\]
together with a segmentation of the tenth video into clips \(s_k\) and a saliency score
\[
y_k \in \{1,\dots,10\}
\]
for each segment. These scores are not generic annotations. They are assigned using the simulator’s **final long-term preferences** \(p_{10}\) and the personal review written after watching the target video, so the labels explicitly encode user-conditioned relevance [2507.16873].

The dataset uses an **LLM-based user simulator** that iteratively retrieves candidate videos, selects a “most wanted” and “least wanted” video, produces summaries and personal opinions, and updates preferences in an **ADD / REFINEMENT / REMOVAL** format over ten turns. This process yields session-level histories with evolving preferences, rather than isolated query–video pairs [2507.16873]. Human verification reported that **97.56%** of simulator queries were judged reasonable with **Fleiss’ \(\kappa = 0.85\)**, the simulator’s most-wanted video matched human majority **71.42%** of the time, and approximately **98%** of saliency pairs were considered reasonable by majority agreement [2507.16873].

HiPHer is therefore best understood not as a standalone highlighting heuristic but as the model counterpart to a new data formulation. The method derives its input signal from full multimodal watch histories rather than from a single query string.

## 3. Input representation and preference modeling

HiPHer takes as input a target video \(V\), segmented into \(n\) segments, together with a watch history \(\mathcal{H}\). For each target segment \(k\), the representation consists of a **representative frame** \(f_k\) and a transcript \(t_k\) obtained via ASR. Feature extraction is performed with **CLIP**: a pre-trained CLIP image encoder produces a visual feature
\[
s^f_k = \text{CLIP\_img}(f_k),
\]
and a CLIP text encoder produces a textual feature
\[
s^t_k = \text{CLIP\_text}(t_k).
\]
These are concatenated to form the segment representation
\[
s_k = s_k^f \oplus s_k^t.
\]
The paper notes that CLIP dimensions are typically 512, giving a concatenated dimension of 1024 [2507.16873].

History videos are encoded in the same manner. For each history video \(H_i\), whose segments are \(s^{(i)}_1,\dots,s^{(i)}_{n_i}\), HiPHer constructs a single video-level embedding by mean pooling over segments:
\[
h^{(i)} = \mathcal{A}gg_s \big( s^{(i)}_1,\dots,s^{(i)}_{n_i} \big),
\]
with
\[
h^{(i)} = \frac{1}{n_i}\sum_{k=1}^{n_i} s^{(i)}_k.
\]

The global user preference embedding is then formed by mean pooling across history videos:
\[
e_p = \mathcal{A}gg_h\left( \{h^{(i)} : h^{(i)} = \mathcal{A}gg_s(s_1^{(i)},\dots,s_{n_i}^{(i)})\}_{i=1}^m \right),
\]
and concretely
\[
e_p = \frac{1}{m} \sum_{i=1}^{m} h^{(i)}.
\]
Thus, HiPHer represents user preference as a **single vector** summarizing the average multimodal content of the watch history [2507.16873].

The authors explicitly note that more advanced aggregation schemes could weight videos differently, but HiPHer uses simple mean pooling. This suggests that the method’s empirical performance derives less from a sophisticated history encoder than from the introduction of history-conditioned supervision and cross-modal conditioning.

## 4. Architecture and conditioning mechanism

Before preference conditioning, both the segment representations \(s_k\) and the preference embedding \(e_p\) are passed through **projection layers** to align them in a shared embedding space. The projection layers are described as three sequential layers of **LayerNorm** and **Dropout**, with an implied standard linear projection structure. This yields projected segment representations \(\tilde{s}_k \in \mathbb{R}^d\) and a projected preference embedding \(\tilde{e}_p \in \mathbb{R}^d\) [2507.16873].

The core personalization mechanism is a **cross-attention** layer in which the target segments act as queries and the preference embedding provides keys and values. In the paper’s description, segment representations query the preference vector to obtain preference-conditioned segment features:
\[
\text{Attn}(\tilde{s}_k, \tilde{e}_p) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V.
\]
Denoting the resulting representation by
\[
\hat{s}_k = \text{CrossAttn}(Q=s_k, K=e_p, V=e_p),
\]
the intent is that each segment representation becomes conditioned on the global preference vector rather than being scored in isolation [2507.16873].

After cross-attention, the sequence of conditioned segment embeddings \(\{\hat{s}_1,\dots,\hat{s}_n\}\) is fed into a **Transformer encoder** with multi-head self-attention and feed-forward layers. This models contextual dependencies among segments, analogous to transformer-based video understanding models. If the encoder outputs \(z_k\) for segment \(k\),
\[
z_k = \text{TransformerEncoder}(\hat{s}_1,\dots,\hat{s}_n)_k,
\]
then a prediction head maps each \(z_k\) to a scalar saliency score,
\[
y_k = f_{\text{score}}(z_k).
\]
The paper does not specify a more elaborate scoring head than a standard regression mapping [2507.16873].

A concise summary of the architecture is given below.

| Component | Function | Formulation |
|---|---|---|
| Segment encoding | Encode target segments from frame and transcript | \(s_k = s_k^f \oplus s_k^t\) |
| History aggregation | Build user preference vector from watched videos | \(e_p = \frac{1}{m}\sum_{i=1}^{m} h^{(i)}\) |
| Preference conditioning | Inject user preference into segment representations | \(\hat{s}_k = \text{CrossAttn}(Q=s_k, K=e_p, V=e_p)\) |
| Context modeling | Model inter-segment structure | \(z_k = \text{TransformerEncoder}(\hat{s}_1,\dots,\hat{s}_n)_k\) |

From an architectural perspective, HiPHer is comparatively simple. It does not explicitly separate short-term from long-term interests, nor does it use a recurrent or transformer history encoder over the watch sequence itself. Instead, it relies on mean-pooled history embeddings and a lightweight conditioning pathway.

## 5. Loss function and training objective

HiPHer is trained with a **contrastive saliency loss** formulated as a pairwise hinge ranking objective. Let \(v^+\) and \(v^-\) denote positive and negative segment feature vectors, with predicted scores \(y^+\) and \(y^-\), respectively. The loss is
\[
\mathcal{L}_{\text{saliency}} = \sum_{(v^+, v^-)} \max(0, \gamma - (y^+ - y^-)).
\]
If the positive segment already scores at least \(\gamma\) higher than the negative segment, the loss is zero; otherwise the model is penalized in proportion to the violation [2507.16873].

The paper states that prior work often uses \(\gamma = 1\), but HiPHer ablates the margin and finds that smaller values in the range \([0.1, 0.2]\) work better, because they allow more fine-grained ranking among segments [2507.16873]. This is a salient design choice because the saliency labels lie on a \(1\)–\(10\) scale rather than a binary highlight/non-highlight axis. A plausible implication is that coarse large-margin ranking can blur distinctions among moderately relevant segments, whereas a smaller margin better fits gradual preference-conditioned variation.

Training uses the same scene-based segmentation regime as the dataset construction pipeline. Inputs are CLIP-derived multimodal features, and the trainable components are the projectors, cross-attention layer, transformer encoder, and scoring head [2507.16873].

## 6. Evaluation, baselines, and empirical findings

HiPHer is evaluated on HiPpo-Video and, for baseline context, on QVHighlights. The evaluation covers **highlight detection (HD)**, **moment retrieval (MR)**, **video summarization (VS)**, and the paper’s personalized highlighting setting, using common metrics such as **mAP**, **Hit@1**, **Recall@1@IoU**, **F1**, and **RMSE** [2507.16873].

For HD, positives are defined by thresholding segment saliency scores, and Hit@1 is reported for thresholds \(k \in \{7,9\}\). For MR, Recall@1 is computed at IoU thresholds 0.5 and 0.7. For VS, F1 is computed against ground-truth salient segments. Score prediction quality is measured using
\[
\text{RMSE} = \sqrt{\frac{1}{N}\sum_{k=1}^{N} (y_k - \hat{y}_k)^2}.
\]
These metrics allow HiPHer to be compared directly with both generic and query-based baselines [2507.16873].

The baselines include **SL-Module**, **Moment-DETR**, **UMT**, **QD-DETR**, **UVCOM**, **TR-DETR**, **CLIP-It**, and **VSL**. Query-based baselines are adapted to HiPpo-Video by generating queries from watch history using key phrases that summarize user history [2507.16873].

On the simulated HiPpo-Video benchmark, the reported results are:

| Method | RMSE | mAP | Hit1@7 | Hit1@9 | Recall1@0.5 | Recall1@0.7 |
|---|---:|---:|---:|---:|---:|---:|
| Best baseline noted in text | — | 0.732 | — | — | 0.398 | 0.210 |
| **HiPHer** | **0.301** | **0.766** | **0.507** | **0.166** | **0.452** | **0.245** |

The paper specifies that **UMT** is the best baseline for mAP at **0.732**, **Moment-DETR** reaches **0.398** on Recall@0.5, and **UMT** reaches **0.210** on Recall@0.7, so HiPHer improves by **+0.034**, **+0.039**, and **+0.035** on those three metrics, respectively [2507.16873].

On the **real-history** subset of **40 histories**, HiPHer is compared to Moment-DETR, QD-DETR, and TR-DETR. The reported results are:

| Method | RMSE | H1@7 | H1@9 | F1@0.5 |
|---|---:|---:|---:|---:|
| Moment-DETR | 0.419 | 0.472 | 0.389 | 0.417 |
| QD-DETR | 0.446 | 0.444 | 0.361 | 0.385 |
| TR-DETR | 0.443 | 0.306 | 0.250 | 0.429 |
| **HiPHer** | 0.427 | **0.486** | **0.400** | **0.624** |

Although the RMSE is not the lowest in this table, HiPHer achieves the best H1@7, H1@9, and especially **F1@0.5 = 0.624**, which the paper interprets as much better personalized highlight segmentation under real user behavior [2507.16873].

In summarization ablations, **CLIP-It** without a query obtains **F1@5 = 0.564** and **F1@7 = 0.211**, phrase-query CLIP-It obtains **0.566** and **0.230**, sentence-query CLIP-It obtains **0.658** and **0.234**, **VSL** obtains **0.466** and **0.187**, while **HiPHer** reaches **0.726** and **0.486** [2507.16873]. The relative margin over the best CLIP-It setting is especially large at the stricter \(F1@7\) threshold.

## 7. Ablations, interpretation, and limitations

Ablation studies isolate the role of history length and modality. When the preference embedding is constructed from only the last \(k \in \{1,2,5,10\}\) history videos, performance improves monotonically as more history is included [2507.16873]. This is one of the clearest empirical findings: longer histories reveal more stable preferences, even though HiPHer encodes them with simple mean pooling.

The modality ablation compares **HiPHer-V** (vision only), **HiPHer-T** (text only), and full HiPHer. The reported values are:

| Variant | mAP | H1@7 | R1@0.5 |
|---|---:|---:|---:|
| HiPHer-V | 0.67 | 0.12 | 0.32 |
| HiPHer-T | 0.74 | 0.15 | 0.39 |
| HiPHer | **0.77** | **0.17** | **0.45** |

The text-only variant outperforms the vision-only variant, and the multimodal model is best overall [2507.16873]. This suggests that ASR transcripts are particularly informative for preference modeling, especially when user interests are semantic or topical rather than primarily visual.

Qualitative case studies show that for the same target video, different watch histories lead HiPHer to generate different highlight profiles [2507.16873]. This behavior is central to the method’s claim: the model is not simply learning generic highlight priors but using history-derived preferences to modulate saliency.

The paper also identifies several limitations. First, the dataset is **simulated** using an LLM-based user model, so biases in the simulator can propagate to both labels and learned behavior. Second, HiPHer’s preference modeling is intentionally simple: it uses mean pooling over segments and videos, does not distinguish short- and long-term interests explicitly, and does not model users across multiple sessions [2507.16873]. Third, the framework is currently tied to YouTube-derived content distributions. A plausible implication is that deployment on other domains, such as short-form video or educational platforms, would require revalidation or adaptation.

Future directions proposed or implied by the paper include non-uniform history aggregation, explicit modeling of short-term versus long-term preferences, sequence models over history, improved simulators, multi-session personalization, and integration with recommender systems that jointly predict the next video and highlight segments within it [2507.16873].

Taken together, HiPHer establishes a formulation in which personalization is encoded not by a single query but by a multimodal history-derived preference embedding. Its technical contribution is comparatively modest in architectural complexity, but its significance lies in defining a new supervision regime for **history-driven, preference-aware, segment-level video understanding** [2507.16873].

Source: https://www.emergentmind.com/topics/hipher