---
title: 'ChronusOmni: DualGround for Temporal Grounding'
url: https://www.emergentmind.com/topics/chronusomni
type: topic
---

# ChronusOmni: DualGround for Temporal Grounding

ChronusOmni is not referenced in the provided source [2510.20244]. The central subject described in this work is DualGround, a dual-branch architecture for structured phrase and sentence-level temporal grounding within the domain of Video Temporal Grounding (VTG). All information below concerns DualGround as defined and examined in the source.

## 1. Formalization of Audiovisual Temporal Grounding

Audiovisual temporal grounding involves localizing segments in long, untrimmed videos corresponding to natural language queries. This is instantiated by two primary subtasks: Moment Retrieval (MR) and Highlight Detection (HD). Formally, a video is represented by $T$ clip-level features $V = \{v_1,\ldots,v_T\}, v_i \in \mathbb{R}^d$ and a tokenized query yields $L-1$ word embeddings $\{e_1,\ldots,e_{L-1}\}$ plus a special [EOS] token $e_{[EOS]} \in \mathbb{R}^d$. The objective for MR is to predict $(\hat{s}, \hat{e})$ (start, end indices) maximizing overlap with the annotated segment, while HD assigns each clip $i$ a saliency score $s_i \in \mathbb{R}$ denoting its relevance. The function modeled is
$$
f : (V, \{e_{l}\}, e_{[EOS]}) \rightarrow ((\hat{s}, \hat{e}), \{s_i\})
$$
MR is evaluated by Recall@1@IoU={0.5,0.7}; HD by mAP and HIT@1.

## 2. DualGround Architecture: Sentence-Level and Phrase-Level Branches

### 2.1 Sentence-Level Branch (Global Semantics)

The sentence-level branch isolates the [EOS] embedding and augments it with $L_d$ trainable dummy tokens $D = \{d_1, ..., d_{L_d}\}$, constructing $E = [D; e_{[EOS]}] \in \mathbb{R}^{(L_d + 1) \times d}$. This sequence is refined by a lightweight Transformer encoder to produce $[D'; e_{[EOS]}]$. Video features $V$ are projected to queries $Q = \{q_i\}$, while $E'$ gives keys $K = \{k_j\}$ and values $U = \{u_j\}$. Cross-modal attention exclusively attends to the [EOS] slot ($j = L_d + 1$):
$$
\alpha_i = \text{softmax}\left(\frac{q_i \cdot k_j}{\sqrt{d}}\right)\Big|_{j=L_d+1}, \qquad v_i^s = \alpha_i \cdot u_{L_d+1}
$$
Temporal self-attention layers are stacked over $\{v_i^s\}$ to yield $V^s \in \mathbb{R}^{T \times d}$.

### 2.2 Phrase-Level Branch (Local Semantics)

Word tokens are clustered into $N$ semantically coherent phrases for localized alignment. Generation involves Recurrent Phrase Generation (RPG), slot-attention refinement, and phrase-clip context embedding:
- RPG recursively generates phrases $p^{(n)}$ for $n = 1, ..., N$:
  $$
  g^{(1)} = \varphi(W_q^{(1)} e_{[EOS]}, 0),\quad g^{(n)} = \varphi(W_q^{(n)} e_{[EOS]}, p^{(n-1)})
  $$
  $$
  p^{(n)} = \sum_{l=1}^{L-1} \text{softmax}\left(\frac{g^{(n)} \cdot e_l}{\sqrt{d}}\right) e_l
  $$
- Initial phrase set $P_i$ is refined through slot-attention and augmented with a learnable $P_{[EOS]}$, allowing global context propagation.
- Phrase-clip context is captured by projecting and interacting features with Hadamard product:
  $$
  C_{n, t} = f_{ctx}(f_p(p^{(n)}) \odot f_v(v_t)),\quad (f_p, f_v, f_{ctx}\text{: MLPs with GELU})
  $$
- Aggregation is guided by $P_{[EOS]}$ attending to refined phrases for temporal fusion:
  $$
  v_{p,t} = \sum_{n=1}^N \text{softmax}\left(\frac{\langle W_q P_{[EOS]}, W_k p^{(n)} \rangle}{\sqrt{d}}\right) C_{n, t}
  $$
Final output $V^p \in \mathbb{R}^{T \times d}$ is produced with temporal self-attention.

### 2.3 Fusion, Temporal Pyramid, and Decoding

The output streams are merged via addition: $F = V^s + V^p$. Prediction heads use a multi-scale temporal pyramid (1D convolution at several resolutions), sharing heads for moment confidence, normalized start/end regression (MR), and saliency (HD).

## 3. Token-Role Aware Interaction and Objective Functions

Role-aware attention mechanisms avoid the documented over-reliance on [EOS] found in previous VTG frameworks, enabling more granular word-level and phrase-level grounding. Core equations:
- Sentence-Level ACA:
  $$
  \alpha_i = \text{softmax}\left(\frac{q_i\cdot k_j}{\sqrt{d}}\right)_{j=L_d+1},\quad ACA(v_i) = \alpha_i u_{L_d+1}
  $$
- Phrase-Clip context:
  $$
  C = f_{ctx}(f_p(P) \odot f_v(V))
  $$
- Phrase-Guided Aggregation:
  $$
  v_{p,t} = \sum_{n=1}^N \text{softmax}\left(\frac{\langle W_q P_{[EOS]}, W_k p^{(n)} \rangle}{\sqrt{d}}\right) C_{n,t}
  $$

The total training loss is
$$
\mathcal{L}_\text{total} = \lambda_\text{mr} \mathcal{L}_\text{mr} + \lambda_\text{hd} \mathcal{L}_\text{hd} + \lambda_\text{phrase} ( \mathcal{L}_\text{DQA} + \mathcal{L}_\text{EOS})
$$
with MR loss comprising focal classification and $L_1$ regression, HD loss combining ranking and contrastive terms for both saliency scores and sentence attention, DQA loss enforcing phrase orthogonality, and [EOS] reconstruction loss aligning phrase-level and global representations.

## 4. Feature Extraction, Implementation, and Evaluation

Experiments utilize QVHighlights and Charades-STA datasets with CLIP + SlowFast or InternVideo2 backbones. The pipeline caches pretrained features; no fine-tuning on extractors is performed. Architecturally, DualGround operates at $d = 256$ hidden size, with post-norm Transformers, AdamW optimization, and 8 attention heads. For moment proposal post-processing, non-maximum suppression is applied at IoU = 0.7.

Benchmark evaluations include Recall@1@IoU=0.5/0.7, mAP, VG-Hit@1, VG-mAP. DualGround demonstrates superior performance on QVHighlights (InternVideo2): R1@0.5 = 71.87%, R1@0.7 = 56.94%, mAP = 52.73%. On Charades-STA (InternVideo2) R1@0.5 = 70.67%, R1@0.7 = 50.33% [2510.20244].

| Dataset         | R1@0.5 (%) | R1@0.7 (%) | mAP (%) | VG-Hit@1 (%) | VG-mAP (%) |
|-----------------|------------|------------|---------|--------------|------------|
| QVHighlights    | 71.87      | 56.94      | 52.73   | 70.80        | 44.02      |
| Charades-STA    | 70.67      | 50.33      |   —     |      —       |    —       |
| FlashVTG (base) | 70.69      | 53.96      | 52.00   | 71.00        | 44.09      |

## 5. Ablation Studies and Qualitative Findings

Ablation reveals optimal phrase count ($N = 4$ for QVHighlights, $N = 3$ for Charades-STA), with degraded performance at extremes. RPG yields +1.7% R1@0.7; slot-attention and DQA further boost metrics (+0.8%, +1.2% respectively). Prior approaches show strong attention correlation to [EOS] ($\approx 0.97$ Pearson); disabling word tokens is harmful for CLIP features but variably impactful with InternVideo2, highlighting the importance of role-aware separation.

Qualitative examples underscore DualGround's granularity: queries like "the lady in red jacket comes into the room" are localized precisely—baseline models disproportionately predict broader segments. Visualization of phrase-clip activation norms reveals sharp semantic alignment corresponding to phrase boundaries.

## 6. Limitations and Prospects

DualGround uses a fixed phrase count $N$, which must be empirically tuned per dataset; learning $N$ dynamically is an open direction. The current approach does not utilize audio features; future work may incorporate audio via cross-modal ACA over spectrogram tokens to improve multimodal event grounding. While computational overhead of the dual-branch approach is modest, efficiency may improve with learned phrase boundaries. As vision-language encoders strengthen global [EOS] signals, the necessity of disentangled semantic modeling increases.

*This suggests that further decoupling of local/global representations and adaptive phrase grouping will be vital for continued progress as video-language models evolve.* [2510.20244]

Source: https://www.emergentmind.com/topics/chronusomni