OVG-HQ-Unify: Unified Online Video Grounding

• OVG-HQ-Unify is a unified online video grounding framework that localizes relevant video moments using text, image, and segment queries in real-time streaming settings.
• It employs a parametric memory block and Transformer-based fusion to integrate multimodal features while preserving modality-specific information.
• The framework uses hybrid cross-modal distillation to address modality imbalance and achieves significant improvements in online recall and mean average precision.

OVG-HQ-Unify is primarily the unified framework introduced for Online Video Grounding with Hybrid-modal Queries, a setting in which a model must localize relevant video moments under streaming constraints from queries that may be text, image, video segment, or their combinations (Zeng et al., 16 Aug 2025). In the surrounding literature, the same phrase also functions as a broader unification motif: high-quality multimodal systems are organized around shared backbones, modality-specific interfaces, temporally or structurally constrained fusion, and evaluation protocols that test whether understanding, grounding, generation, or localization remain coherent across modalities and tasks (Fang et al., 2024, Hossain et al., 29 Apr 2026, Song et al., 12 May 2025, Wang et al., 2024, Ning et al., 17 Oct 2025, Xie et al., 4 Apr 2025). This suggests that OVG-HQ-Unify is best understood not only as a named online grounding architecture, but also as a research program centered on unified, high-quality multimodal reasoning under realistic constraints.

1. Task formulation and conceptual boundaries

In its explicit formulation, OVG-HQ generalizes classical video grounding along two axes: online access and hybrid-modal queries. Classical temporal grounding assumes an untrimmed video $V=\{x_i\}_{i=1}^{T}$, a text query $\mathcal{Q}$, and full access to the entire video before predicting moments $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$ (Zeng et al., 16 Aug 2025). OVG-HQ instead restricts the query to any subset $\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$, where $q_t$ is text, $q_i$ is an image, and $q_s$ is a video segment, while limiting visual context at timestamp $t$ to a sliding window $V_{t-k+1:t}=\{x_i\}_{i=t-k+1}^{t}$ (Zeng et al., 16 Aug 2025). Predictions are online and irrevocable: once a moment is emitted at time $t$, it is not revised by future frames.

This online definition matters because it changes both model design and evaluation. Offline video grounding can exploit future context, globally optimized proposal sets, and post-hoc refinement. OVG-HQ forbids those conveniences and therefore emphasizes temporally local evidence, historical memory, and decision timeliness (Zeng et al., 16 Aug 2025). The proposal parameterization reflects this: each anchor ends at the current time, $\mathcal{Q}$0 with $\mathcal{Q}$1, so grounding is anchored in streaming time rather than retrospective search (Zeng et al., 16 Aug 2025).

A common misconception is that “unify” denotes a single undifferentiated modality space. Across the cited systems, unification is more structured. OVFormer aligns video-level instance queries with CLIP image embeddings through a dedicated cross-attention module rather than collapsing them into one raw feature space (Fang et al., 2024). HQ-UNet inserts a compact quantum bottleneck into an otherwise classical U-Net rather than replacing the entire network with a quantum stack (Hossain et al., 29 Apr 2026). UnifyGeo shares an encoder but retains separate semantic and fine-grained heads for retrieval and metric localization (Song et al., 12 May 2025). OV-Uni3DETR fuses projected image features and voxelized point-cloud features, yet still trains with modality switching and dual branches to isolate noisy supervision (Wang et al., 2024). The literature therefore treats unification as controlled interoperability, not modality erasure.

2. Core architecture of the named OVG-HQ-Unify framework

OVG-HQ-Unify consists of three parts: feature extraction, memory-guided multi-modal fusion, and memory-guided moment prediction (Zeng et al., 16 Aug 2025). The video encoder is CLIP’s image encoder; the text query uses the CLIP text encoder; image queries use the CLIP image encoder; and segment queries are processed with the same video feature extractor under temporal sampling (Zeng et al., 16 Aug 2025). All modality features are projected into a common dimension $\mathcal{Q}$2, and each query modality is prefixed with a learnable modality token so that a hybrid query can be represented as a single structured sequence, such as $\mathcal{Q}$3 for text-plus-image input (Zeng et al., 16 Aug 2025).

Cross-modal fusion is implemented with a Transformer decoder in which video-window features act as queries and hybrid-query features act as keys and values, producing a query-aware video representation $\mathcal{Q}$4 (Zeng et al., 16 Aug 2025). The distinctive component is the Parametric Memory Block (PMB), which stores historical information in parameters rather than in an explicit external memory bank. Given input $\mathcal{Q}$5, PMB first updates its memory parameters $\mathcal{Q}$6 through the reconstruction objective

$\mathcal{Q}$7

with adaptive update

$\mathcal{Q}$8

and then emits the memory-augmented representation

$\mathcal{Q}$9

PMB is used twice: once after fusion to inject longer-range temporal context into $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$0, and again in the prediction refinement stage to retain information from earlier predictions (Zeng et al., 16 Aug 2025).

Moment prediction is anchor-based. A second Transformer decoder maps anchor embeddings and memory-guided video features to anchor features $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$1, after which a classification head predicts foreground and background scores and a regression head predicts length and offset parameters (Zeng et al., 16 Aug 2025). Boundaries are updated by

$\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$2

A Prediction Refinement Module feeds concatenated anchor features and prediction features into another PMB instance, yielding refined classification and regression outputs before thresholding by foreground score (Zeng et al., 16 Aug 2025).

The training procedure addresses modality imbalance through hybrid cross-modal distillation. A teacher expert trained on the informative text-plus-segment setting supervises the unified student through KL divergence on anchor features and MSE losses on classification and regression outputs: $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$3 with total objective

$\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$4

This design directly targets the reported failure mode in naïve joint training, where text dominates and weaker modalities degrade (Zeng et al., 16 Aug 2025).

3. Data construction and evaluation regimes

OVG-HQ-Unify is paired with QVHighlights-Unify, an extension of QVHighlights that introduces hybrid-modal queries at scale (Zeng et al., 16 Aug 2025). The original QVHighlights provides more than 10,000 daily-vlog and news videos with free-form text queries and annotated moments. QVHighlights-Unify augments this with retrieved images (Image-R), generated images (Image-G), generated query segments (Segment-G), and complementary text-image pairs (Image-C + Text-C) (Zeng et al., 16 Aug 2025). Its reported scale is 19.0K text queries, 26.3K image queries, and 8.8K segment queries, compared with 10.3K text-only queries in the original QVHighlights and 1.5K text plus 6.2K image queries in ICQ-Highlight (Zeng et al., 16 Aug 2025). For Image-R, a human study over 200 queries reports a Pearson correlation of 0.86 between InternVL ranking and human ranking, with 96% top-3 exact match (Zeng et al., 16 Aug 2025).

Because offline metrics ignore latency, OVG-HQ introduces online variants of recall and average precision. For a successful prediction, a decay factor $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$5 equals 1 when the prediction is made exactly at the ground-truth end time and decays linearly to 0 within a threshold $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$6 (Zeng et al., 16 Aug 2025). The online recall metric is

$\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$7

and online mean average precision is defined through decay-weighted precision and recall contributions: $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$8 These metrics make timeliness part of correctness rather than a secondary runtime statistic (Zeng et al., 16 Aug 2025).

The broader OVG-HQ-Unify ecosystem is evaluated under similarly task-specific but increasingly unified protocols. OVFormer uses LV-VIS and zero-shot transfer to YouTube-VIS 2019/2021 and OVIS for open-vocabulary video instance segmentation (Fang et al., 2024). HQ-UNet evaluates dense prediction on LandCover.ai after preprocessing GeoTIFF orthophotos into $\mathcal{M}=\{\mathcal{M}_m=(s_m,e_m)\}_{m=1}^{M}$9 tiles and normalized $\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$0 patches (Hossain et al., 29 Apr 2026). UnifyGeo targets large-scale fine-grained cross-view geo-localization on VIGOR, CVUSA, and CVACT, jointly assessing retrieval and meter-level localization (Song et al., 12 May 2025). UniMedVL is trained through the Observation-Knowledge-Analysis paradigm on UniMed-5M, a dataset of over 5.6M samples spanning nine imaging modalities (Ning et al., 17 Oct 2025). MME-Unify provides a cross-model benchmark with 4,104 samples over understanding, generation, and mixed-modality “unify” tasks, and defines the aggregate score

$\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$1

for unified multimodal models (Xie et al., 4 Apr 2025).

4. Representative unification patterns across adjacent literature

Beyond the named online grounding model, several works instantiate closely related principles of unification. OVFormer addresses Open-Vocabulary Video Instance Segmentation by generating video-level instance queries, aligning them with CLIP image features through cross-attention, and classifying them against CLIP text embeddings in a unified CLIP space (Fang et al., 2024). Its alignment step is

$\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$2

followed by classification

$\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$3

The design explicitly addresses the domain gap between instance queries and VLM features and the underutilization of temporal consistency (Fang et al., 2024).

HQ-UNet provides a distinct but structurally analogous example in remote sensing. It inserts a compact parameterized quantum circuit at the bottleneck of a separable-convolution U-Net, using 16 qubits on a conceptual $\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$4 grid, angle encoding with $\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$5, $\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$6, and $\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$7, and a non-pooling QCNN based on a 2-qubit filter

$\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$8

Its quantum block is explicitly shallow and parameter-shared, so the quantum component functions as a latent feature transform rather than a new downsampling stage (Hossain et al., 29 Apr 2026).

UnifyGeo integrates cross-view retrieval and metric localization in one hierarchical network with shared ConvNeXt-Tiny stages, a semantic branch for global descriptors, a fine-grained branch for spatial matching, and a re-ranking mechanism combining retrieval scores with detailed match peaks (Song et al., 12 May 2025). OV-Uni3DETR similarly unifies modality, scene, and category vocabulary in 3D detection by projecting image features into voxel space, fusing them with point-cloud features,

$\mathcal{Q}\subseteq\{q_t,q_i,q_s\}$9

and propagating knowledge between 2D and 3D through cycle-modality propagation (Wang et al., 2024).

UniMedVL and OpenUni extend the same unification logic into multimodal understanding-generation systems. UniMedVL combines a ViT-based understanding encoder, a FLUX VAE generation encoder, and a mixture-of-transformer-experts under a unified loss

$q_t$0

while OpenUni bridges frozen InternVL3 backbones and SANA diffusion models through 256 learnable queries and a 6-layer transformer connector (Ning et al., 17 Oct 2025, Wu et al., 29 May 2025). Uni-Mapper translates the same “unify” imperative into multi-session, multi-robot, multi-LiDAR mapping through dynamic object removal, dynamic-aware loop closure, and anchor-node-based pose graph optimization (Kang et al., 28 Jul 2025).

System	Domain	Defining unification mechanism
OVG-HQ-Unify	Online video grounding	PMB memory + hybrid-query distillation
OVFormer	Open-vocabulary VIS	Unified embedding alignment with CLIP
HQ-UNet	Remote-sensing segmentation	Quantum bottleneck inside classical U-Net
UnifyGeo	Cross-view geo-localization	Shared encoder for retrieval and metric localization
OV-Uni3DETR	Open-vocabulary 3D detection	Cycle-modality propagation
UniMedVL	Medical multimodal understanding/generation	OKA curriculum + dual visual encoders
OpenUni	Unified understanding/generation baseline	Learnable queries + light connector
Uni-Mapper	Multi-modal LiDAR mapping	Dynamic-aware loop closure + anchor-node merging

This comparison suggests that OVG-HQ-Unify is less a single architectural template than a recurring systems pattern: a high-capacity classical backbone is preserved, modality interfaces are explicitly engineered, and the “unified” component is localized in a bottleneck, alignment layer, memory module, or cross-task training objective.

5. Empirical profile and reported results

On QVHighlights-Unify, the named OVG-HQ-Unify model reports 23.26 for $q_t$1 and 23.09 for $q_t$2 on text queries, exceeding TwinNet’s 20.78 and 19.73 and substantially surpassing offline methods adapted to online evaluation such as TaskWeave, TR-DETR, and R$q_t$3-Tuning (Zeng et al., 16 Aug 2025). In the hybrid-query setting, expert models show that Segment-G outperforms image-only queries and that Segment-G plus text outperforms either modality alone, while naïve unified training collapses visual-only performance; hybrid distillation then restores much of that loss, with Image-R improving from 11.43% to 20.41% in $q_t$4 (Zeng et al., 16 Aug 2025). Runtime analysis on RTX 4090 reports 21.76 ms latency for the overall model, or 45.95 FPS, while PMB adds 2.20 ms and dynamic memory update adds 0.30 ms, indicating that the memory mechanism is not the dominant computational cost (Zeng et al., 16 Aug 2025).

The broader literature shows comparable gains when unification is done with explicit architectural structure. OVFormer reaches 21.9 mAP with a ResNet-50 backbone on LV-VIS, compared with 14.5 for OV2Seg* on the same validation setting, and reports nearly equal base and novel performance at 22.1 mAP$q_t$5 and 21.8 mAP$q_t$6 (Fang et al., 2024). It also reports zero-shot gains of +7.6 mAP on YouTube-VIS 2019 and +3.9 mAP on OVIS (Fang et al., 2024). HQ-UNet reaches mIoU 0.8050 and overall accuracy 94.76% on LandCover.ai, improving over the classical U-Net baseline of 0.6451 mIoU and 82.43% OA while using a compact quantum bottleneck under NISQ-oriented constraints (Hossain et al., 29 Apr 2026). UnifyGeo reports a rise in 1-meter-level localization recall on VIGOR from 1.53\% to 39.64\% in same-area evaluation and from 0.43\% to 25.58\% in cross-area evaluation (Song et al., 12 May 2025). OV-Uni3DETR reports that it surpasses existing methods by more than 6\% on average, and on indoor benchmarks its multi-modal configuration reaches 20.85 AP$q_t$7 on SUN RGB-D and 17.99 AP$q_t$8 on ScanNet in the reported open-vocabulary settings (Wang et al., 2024).

In unified understanding-generation systems, UniMedVL reports SLAKE 75.4, PathVQA 53.5, OmniMedVQA 85.8, average FID 96.29, and BiomedCLIP 0.706, while emphasizing bidirectional knowledge sharing between generation and understanding (Ning et al., 17 Oct 2025). OpenUni reports GenEval 0.86 for OpenUni-L-1024 and DPG-Bench 83.08, using only 3.1B activated parameters in the large variant (Wu et al., 29 May 2025). MME-Unify’s benchmarking results then show the remaining gap: the best overall evaluated U-MLLM, Gemini 2.0 Flash-exp, attains MME-U 45.57, indicating that current “unified” models still underperform on mixed-modality tasks relative to the ambition implied by the term (Xie et al., 4 Apr 2025).

6. Limitations, misconceptions, and open research directions

The most immediate limitation of the named OVG-HQ-Unify system is the one it explicitly identifies: limited context in online settings and modality imbalance during training (Zeng et al., 16 Aug 2025). PMB alleviates but does not remove the first problem, because the streaming model still operates on a fixed sliding window and a compact parameterized memory rather than on unrestricted history. The second problem is reduced by hybrid distillation, but the need for a privileged teacher trained on text-plus-segment queries suggests that unified multimodal learning remains asymmetric (Zeng et al., 16 Aug 2025). QVHighlights-Unify is also built from vlogs and news video with retrieved and generated visual queries; this supports scale, but it implies a domain bias and dependence on CLIP, InternVL, Stable Diffusion, and CogVideoX for feature quality and synthetic-query realism (Zeng et al., 16 Aug 2025).

Across adjacent systems, a second misconception is that performance gains follow merely from adding modalities. The cited evidence does not support that view. Existing quantum-classical baselines in remote sensing perform poorly when quantum modules are added naïvely, whereas HQ-UNet improves only after placing a shallow non-pooling QCNN at the bottleneck (Hossain et al., 29 Apr 2026). Existing open-vocabulary VIS systems perform poorly on novel categories when they compare raw instance queries directly with text embeddings, whereas OVFormer improves only after explicit video-level query–image alignment and semi-online inference (Fang et al., 2024). UnifyGeo ablations show that unified learning without its feature aggregator can worsen metric localization before re-ranking and multi-granularity design resolve the conflict (Song et al., 12 May 2025). The repeated implication is that unification requires carefully engineered interfaces and loss couplings, not additive multimodality.

Open directions follow the same pattern. In online grounding, likely extensions include longer-range or hierarchical memories, additional query modalities, dynamic queries, and broader domains such as surveillance, sports, or tutorials (Zeng et al., 16 Aug 2025). In open-vocabulary video segmentation, explicit contrastive or distillation objectives between queries and CLIP features are proposed as likely improvements (Fang et al., 2024). In hybrid quantum-classical vision, the next steps are additional datasets, ablation studies on qubit count and circuit depth, and deployment beyond noiseless simulators (Hossain et al., 29 Apr 2026). In unified understanding-generation, both UniMedVL and MME-Unify point to the need for stronger task coupling, safer and more faithful generation, and better mixed-modality evaluation than CLIP-based similarity alone can provide (Ning et al., 17 Oct 2025, Xie et al., 4 Apr 2025). In 3D and geo-localization settings, the unresolved issues are noisy pseudo-labels, scene-dependent calibration assumptions, and the extension from boxes or metric points to richer world representations (Wang et al., 2024, Song et al., 12 May 2025).

Taken together, the literature presents OVG-HQ-Unify as a technically specific online grounding framework and, at the same time, as a wider design doctrine: unify only where representations can be reliably aligned, preserve task-specific structure where modalities conflict, and evaluate “high quality” with metrics that reflect not only accuracy but also timeliness, calibration, or joint multimodal consistency.