QVHighlights-Unify: Online Multimodal Video Grounding

• QVHighlights-Unify is a benchmark for online video grounding using multiple modalities including text, images, and generated video segments.
• It employs a construction pipeline that integrates retrieved and generated queries validated by CLIP features and human evaluation.
• The benchmark introduces online evaluation metrics with timeliness penalties to capture accuracy and latency in streaming video scenarios.

Searching arXiv for papers on QVHighlights-Unify, OVG-HQ, and related QVHighlights context. QVHighlights-Unify is an expanded benchmark derived from QVHighlights and introduced for Online Video Grounding with Hybrid-modal Queries (OVG-HQ), a setting in which target segments must be localized in streaming video from text, images, video segments, and their combinations (Zeng et al., 16 Aug 2025). Its primary role is to remove two limitations of traditional video grounding corpora: the assumption of text-only queries and the assumption of offline access to the full video. In that respect, QVHighlights-Unify extends the scope of the original QVHighlights dataset, which enabled unified study of query-based moment retrieval and highlight detection through free-form natural-language queries, relevant moments, and clip-level saliency annotations (Lei et al., 2021).

1. Origin in online, hybrid-modal video grounding

QVHighlights-Unify was introduced because traditional video grounding datasets do not support evaluation under streaming or online video conditions and do not accommodate visual-query inputs. The associated task, OVG-HQ, is defined as online segment localization using text, images, video segments, and their combinations. The paper identifies two central difficulties in this setting: limited context in online settings and modality imbalance during training, where dominant modalities overshadow weaker ones (Zeng et al., 16 Aug 2025).

The dataset therefore serves a dual purpose. First, it operationalizes online grounding, where predictions must be produced as video content arrives rather than after exhaustive access to the full timeline. Second, it broadens the notion of a query from a sentence to a hybrid object that can include retrieved images, generated images, and generated video exemplars. This design repositions query specification as a multimodal interaction problem rather than a purely linguistic one.

2. Dataset construction and query synthesis

QVHighlights-Unify is constructed by expanding the existing QVHighlights dataset through several query-generation procedures (Zeng et al., 16 Aug 2025). The construction pipeline includes four main components.

Image-R (retrieved images): for each text query, ten semantically relevant images are retrieved via web search. The InternVL vision-LLM ranks them by semantic similarity to the query, and the highest-scoring image is selected as the retrieved-image query.

Text-C + Image-C (complementary text/image pairs): for queries that are difficult to express, a complementary image is generated using stable diffusion generators according to a modified text. The corresponding complementary text explicitly contrasts the modified and original semantics, exemplified by formulations such as “The action is swimming, not dancing.” The paper states that this step follows the annotation approach of ICQ-Highlight.

Image-G and Segment-G (generated visual queries): Image-G samples are generated in four visual styles—scribble, cartoon, cinematic, realistic— from text prompts using Stable Diffusion. Segment-G samples are six-second video clips generated for each text query using CogVideoX-5B, with GPT-4o used to enrich the query text prior to generation. The paper further states that all generated samples are subject to manual filtering and refinement for quality and relevance.

Validation: semantic alignment among textual queries, images, and generated segments is cross-validated using CLIP feature space overlap and human evaluation. The reported agreement is that InternVL-based retrieval correlates well with human preference, with a Pearson correlation of 0.86 on a random subset.

3. Supported query modalities and benchmark scale

QVHighlights-Unify supports text, image, video segment, and hybrid queries, including arbitrary combinations, and the paper enumerates eight distinct query configurations (Zeng et al., 16 Aug 2025). These are divided into single-modal and dual/multi-modal settings.

Single-modal configurations: Text, Image-R, Image-G, Segment-G.

Dual/multi-modal configurations: Image-R + Text, Image-G + Text, Segment-G + Text, Complementary Text-C + Image-C.

This modality design distinguishes QVHighlights-Unify from earlier datasets that were either text-only or more limited in multimodal coverage. The benchmark comparison reported in the paper is summarized below.

Dataset	Text Queries	Image Queries	Segment Queries
QVHighlights	10.3K	0	0
ICQ-Highlight	1.5K	6.2K	0
QVHighlights-Unify	19.0K	26.3K	8.8K

The authors describe QVHighlights-Unify as significantly larger and richer in modalities than previous datasets. They also report that examples and t-SNE analysis confirm semantic overlap and diversity among modalities. In practical terms, the dataset is intended to support user-simulation scenarios in which a query may be easier to specify visually, textually, or through their combination.

4. Online metrics and timeliness-sensitive evaluation

A central contribution tied to QVHighlights-Unify is the introduction of online evaluation metrics, motivated by the claim that traditional offline metrics do not penalize late predictions and therefore fail to capture time-sensitive requirements in streaming settings (Zeng et al., 16 Aug 2025).

The first metric family is oR@$$, IoU=$$, an online variant of recall:

$$\text{oR@}n,\text{IoU@}m = \frac{1}{N_q} \sum_{i=1}^{N_q} \beta_i \cdot r(n, m, q_i)$$

Here, $r(n,m,q_i)=1$ if at least one of the top-$n$ predictions for query $q_i$ has IoU above $m$, and $0$ otherwise. The factor $\beta_i \in (0,1]$ introduces a timeliness penalty and decays linearly if prediction is made after ground-truth endpoint, with thresholds given as 1s/3s/5s after end.

The second metric is online mean Average Precision, denoted omAP$_m$:

$$0

with per-query online AP defined as

$$1

where $$2 is the number of true positive hits for query $$3, $$4 is the timeliness decay for the $$5-th prediction, and $$6 are precision and recall at cutoff $$7.

These metrics are described as averaging over multiple latency thresholds and jointly capturing accuracy and detection latency. A common misunderstanding is to treat them as mere renamings of offline recall and mAP; the distinguishing feature is precisely the explicit incorporation of delayed-decision penalties.

5. Relation to QVHighlights and the broader unification agenda

QVHighlights-Unify inherits its foundation from QVHighlights, which introduced a dataset of 10,148 videos and 10,310 queries, each annotated with a human-written free-form natural-language query, relevant moments, and five-point saliency scores for all query-relevant clips (Lei et al., 2021). That earlier benchmark unified query-based video moment retrieval and query-based highlight detection, and it was accompanied by Moment-DETR, an end-to-end transformer encoder-decoder model for joint moment localization and saliency prediction.

Subsequent work on QVHighlights pursued several different notions of unification. UniVTG proposed a unified formulation spanning moment retrieval, highlight detection, and video summarization through clip-wise foreground indicators, boundary offsets, and saliency scores (Lin et al., 2023). DiffusionVMR recast moment retrieval and highlight detection as a conditional denoising generation process (Zhao et al., 2023). UVCOM argued that moment retrieval and highlight detection require different emphases—local relation activation for the former and global knowledge accumulation for the latter—while still solving them jointly (Xiao et al., 2023). Later, Moment and Highlight Detection via MLLM Frame Segmentation reformulated both tasks through frame-wise binary mask generation from a multimodal LLM (Jiwanta et al., 13 Dec 2025).

Against that background, QVHighlights-Unify adds a different axis of unification. Rather than only unifying label types or learning objectives, it introduces unification at the level of query modality and temporal access regime. This suggests that the dataset marks a shift from unifying downstream temporal grounding tasks to unifying the forms in which user intent is expressed and evaluated.

6. Conceptual significance and interpretive boundaries

QVHighlights-Unify should not be reduced to a text dataset with auxiliary images appended to it. The benchmark explicitly supports standalone image queries, standalone segment queries, and hybrid combinations, and its construction pipeline includes both retrieved and generated visual queries (Zeng et al., 16 Aug 2025). Nor is it merely an offline benchmark with renamed metrics: the evaluation protocol is designed specifically to measure prediction timeliness under online constraints.

Its significance lies in the way it reframes video grounding. In earlier formulations, the query was typically a sentence and the principal modeling challenge was cross-modal alignment between text and video. In QVHighlights-Unify, the query space itself becomes heterogeneous, and the benchmark emphasizes the operational constraint that decisions must be made with limited context as the stream unfolds. A plausible implication is that future grounding systems will be evaluated less by their performance under static, text-only assumptions and more by their robustness to hybrid-modal intent specification and latency-sensitive deployment.

Within the literature centered on QVHighlights, QVHighlights-Unify therefore occupies a specific position: it does not replace the original benchmark’s role in unified moment retrieval and highlight detection, but extends the benchmark lineage toward online, multimodal, and interaction-oriented grounding.