Clarify: Refining Video Grounding Boundaries

• The paper introduces EtC, a pseudo-boundary refinement pipeline that first expands temporal labels with MLLM-generated semantic descriptions before clarifying them via mutual learning.
• It preserves temporal continuity and enriches video content by using detailed frame descriptions to mitigate noise from imperfect pseudo labels.
• Empirical results on Charades-STA and ActivityNet Captions show significant improvements in temporal localization accuracy compared to previous methods.

Searching arXiv for the primary paper and closely related WSVG/background papers. First, locating the primary paper. Searching for related weakly supervised video grounding papers mentioned in the source material. EtC (“Expand then Clarify”) is a weakly supervised video grounding framework for mapping a natural-language query $q$ to a temporal segment in an untrimmed video $V$ without access to ground-truth temporal boundaries during training. It addresses the supervision gap between video-level labels and boundary-level prediction by first expanding incomplete pseudo boundaries with multimodal LLM (MLLM) descriptions and then clarifying those expanded boundaries through mutual learning and a proposal-level contrastive objective, with the aim of producing more precise refined boundaries $B^*$ (Li et al., 2023).

1. Problem setting and motivation

Weakly supervised video grounding (WSVG) trains on matched video–query pairs but must output start and end times. Early explicit-supervision approaches attempt to bridge this gap by generating pseudo temporal boundaries through data augmentations and teacher–student consistency. The central difficulty identified by EtC is that such augmentations can disrupt temporal continuity in video or alter crucial tokens in text, so noise in imperfect pseudo labels is propagated and amplified during training (Li et al., 2023).

EtC formulates a different response to this problem. Rather than randomly augmenting the original video–text pair, it preserves the original temporal content and introduces additional semantic detail through MLLM-based frame descriptions. The method is organized around two complementary observations stated explicitly in the formulation. First, frames within an initial pseudo boundary $B_{\text{init}}$ are near ground-truth-relevant content and share visual continuity with ground-truth frames. Second, the signals available during refinement are asymmetric: initial boundaries are typically incomplete but relatively clean, whereas expanded boundaries are typically comprehensive but noisier. The framework is therefore designed to exploit both signals rather than choosing one over the other.

A plausible implication is that EtC should be understood less as a pseudo-label generator in the conventional augmentation sense than as a pseudo-boundary refinement pipeline that converts semantic densification into a denoising problem. That interpretation is consistent with the paper’s emphasis on “Expand” followed by “Clarify,” rather than direct relabeling.

2. Formalization and architectural structure

The method uses the following notation. A video is $V=\{x_t\}_{t=1}^{T}$ with $T$ frames; a query is $q$, a sequence of $L$ tokens; initial pseudo boundaries are $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$; expanded pseudo boundaries are $$B_{\text{exp}}=[t_s^{\text{exp}}, t_e^{\text{exp}}]$$; and refined boundaries are $V$0. Candidate proposals are $V$1 with $V$2. Visual features are framewise vectors $V$3, extracted with I3D on Charades-STA or C3D on ActivityNet Captions; videos are downsampled by $V$4 and the maximum temporal length is $V$5. Text features are $V$6 from GloVe word2vec with maximum query length $V$7 (Li et al., 2023).

A transformer fuses video and text. A $V$8 token is appended to the video features to obtain $V$9, and a fully connected head with sigmoid predicts learnable Gaussian pseudo-boundary parameters:

$B^*$0

where $B^*$1 and $B^*$2 are normalized center and width. Proposal features $B^*$3 are pooled from $B^*$4 according to $B^*$5. These proposal features are then consumed by a baseline grounding head, instantiated either with a query reconstruction branch or a multiple-instance learning branch.

The query reconstruction branch, exemplified by CPL, masks about one third of the query tokens and reconstructs the query conditioned on $B^*$6 and previously generated tokens:

$B^*$7

The MIL branch instead computes positive and negative video–query matching scores using top-$B^*$8 proposal averaging and applies a margin loss:

$B^*$9

EtC itself does not introduce additional regression heads. Its architectural intervention is concentrated in the two-stage procedure and in the refinement of the learnable Gaussian boundary parameters through mutual learning and proposal-level contrast.

3. Expand: boundary expansion by MLLM-enriched descriptions

In the Expand stage, a baseline WSVG model first produces $B_{\text{init}}$0. EtC then uses BLIP-2 to generate multiple detailed descriptions for frames inside the initial region. The training procedure precomputes descriptions for each frame using $B_{\text{init}}$1 prompts, generating $B_{\text{init}}$2 descriptions per frame and storing them in a per-frame dictionary. During training, $B_{\text{init}}$3 frames are randomly sampled within the current initial region, and one description is sampled from their descriptions to form an expanded description $B_{\text{init}}$4 (Li et al., 2023).

This design is motivated by video continuity: frames adjacent to relevant content are visually similar, so framewise language can expose objects, actions, and context omitted by the original query. The immediate effect is expansive rather than corrective. The richer text drives re-localization toward a broader temporal region $B_{\text{init}}$5 that more comprehensively covers the event but may include irrelevant content. The paper characterizes this as a shift from “incomplete yet clean” boundaries to “comprehensive yet noisy” ones.

The Expand stage is therefore intentionally permissive. It does not attempt to produce the final boundary directly. Instead, it supplies the second stage with a semantically enriched but potentially over-inclusive proposal. The paper’s ablations further report that BLIP-2 outperforms alternatives such as LLaVA for this task, which is consistent with the framework’s dependence on frame-level descriptions that remain useful under later denoising (Li et al., 2023).

A plausible implication is that Expand is best viewed as semantic recall maximization: it sacrifices precision to reduce the risk of omitting relevant temporal content that the weakly supervised baseline failed to capture initially.

4. Clarify: mutual learning and proposal-level contrastive refinement

The Clarify stage explicitly optimizes the balance between the two boundary sources. Let the original-query branch output $B_{\text{init}}$6 and the expanded-description branch output $B_{\text{init}}$7. Mutual learning enforces consistency between the two parameterizations with stop-gradient:

$B_{\text{init}}$8

where $B_{\text{init}}$9 truncates the input gradient (Li et al., 2023).

The role of stop-gradient is not incidental. It prevents degenerate collapse and allows each branch to move toward the other’s current estimate without jointly over-constraining both. In the paper’s interpretation, this stabilizes training and lets the cleaner branch regularize the noisier branch while allowing the noisier branch to broaden the cleaner one.

Clarify then adds a proposal-level contrastive loss (PCL) based on two alignment signals. The first is query–description matching (QDM), where $V=\{x_t\}_{t=1}^{T}$0 scores similarity between the original query and the MLLM description for frame $V=\{x_t\}_{t=1}^{T}$1, computed with a text model such as BERT. The second is query–frame matching (QFM), where $V=\{x_t\}_{t=1}^{T}$2 measures how well the MLLM says frame content matches the query. For a boundary $V=\{x_t\}_{t=1}^{T}$3 with $V=\{x_t\}_{t=1}^{T}$4 and $V=\{x_t\}_{t=1}^{T}$5, and a temporal margin $V=\{x_t\}_{t=1}^{T}$6, EtC defines inside and outside averages:

$V=\{x_t\}_{t=1}^{T}$7

$V=\{x_t\}_{t=1}^{T}$8

$V=\{x_t\}_{t=1}^{T}$9

The hinge-style contrastive objective with margin $T$0 is

$T$1

This loss is applied twice, once with $T$2 and once with $T$3, giving

$T$4

The full Clarify objective for the two branches is

$T$5

The balancing coefficients are dataset-specific: on Charades-STA, $T$6 and $T$7; on ActivityNet Captions, $T$8 and $T$9.

The paper notes that a proposal-level InfoNCE formulation is possible in principle, but the reported experiments found the hinge-style objective more stable and efficient. This suggests that Clarify is designed around local boundary correction rather than large negative-set discrimination.

5. Optimization procedure, inference path, and empirical performance

Training proceeds in a fixed sequence. Framewise BLIP-2 descriptions are precomputed. For each video–query pair, the model performs a forward pass through the original-query branch and the expanded-description branch, computes the baseline grounding losses $q$0 and $q$1, applies mutual learning, computes the QDM and QFM sequences, evaluates the proposal-level contrastive losses for both branches, and backpropagates $q$2. Optimization uses Adam with learning rate $q$3 and an inverse square-root scheduler. The baseline is warmed up for $q$4 epochs on Charades-STA and $q$5 epochs on ActivityNet Captions, and training continues for $q$6 epochs. The Clarify process always uses the top-1 pseudo boundary from the base model (Li et al., 2023).

Inference is considerably simpler. Only the original query $q$7 and video $q$8 are used. The trained baseline outputs the top-1 boundary $q$9. No MLLM descriptions are needed at test time, and no NMS is required under the top-1 setting.

The reported empirical results show improvements both in ablation studies and in comparison with the prior state of the art.

Dataset	Base $L$0 Final EtC	State-of-the-art comparison
Charades-STA	$L$1	surpasses UGS by $L$2
ActivityNet Captions	$L$3	exceeds UGS by $L$4

On ActivityNet Captions, component ablations isolate the contributions of the two refinement mechanisms. Starting from the replicated CPL base at $L$5 for $L$6, adding mutual learning gives $L$7, adding PCL gives $L$8, and the final EtC model reaches $L$9 (Li et al., 2023).

The reliability analysis further reports a reduction in completely wrong pseudo boundaries ($$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$0), from $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$1 to $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$2 on Charades-STA and from $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$3 to $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$4 on ActivityNet Captions. The paper interprets this as effective noise mitigation. It also reports that PCL contributes more when the initial pseudo boundaries are lower quality, which is consistent with Clarify’s stated role as a denoising mechanism.

6. Position within WSVG research, adaptability, and limitations

EtC is positioned against explicit-supervision pseudo-boundary methods that rely on random augmentations. Its novelty lies in preserving temporal integrity and enriching semantics through MLLM-generated descriptions before denoising them with learnable refinement. In that sense, its contribution is not merely the use of an MLLM, but the specific coupling of semantic expansion with a refinement objective that contrasts inside-boundary and outside-boundary alignment signals (Li et al., 2023).

The framework is designed to be adaptable to multiple baseline localizers. The paper explicitly discusses compatibility with CNM and CPL and notes more limited integration with RSTPN because RSTPN uses fixed proposals. This boundary-parameter-centric design also explains why EtC does not directly improve rank-5 recall: it refines the most confident boundary rather than introducing multi-boundary selection or NMS-like reasoning.

The stated limitations are correspondingly concrete. The method depends on expanded-boundary quality; if MLLM descriptions mischaracterize frames, the expanded boundaries can drift. It is sensitive to hyperparameters such as $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$5, $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$6, $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$7, and $$B_{\text{init}}=[t_s^{\text{init}}, t_e^{\text{init}}]$$8. It incurs additional preprocessing and training cost because BLIP-2, BERT-based QDM scoring, and MLLM-based QFM scoring must all be computed. Its focus remains top-1 boundary refinement, leaving multi-boundary reasoning as future work. The paper also suggests further study of PCL integration with manual proposal generators or diffusion-based localizers.

A plausible broader implication is that EtC exemplifies a general pattern in weak supervision: semantic enrichment can improve recall only if accompanied by a mechanism that explicitly models the difference between useful expansion and harmful over-inclusion. Within EtC, that mechanism is the Clarify stage, and the empirical results indicate that this stage is not auxiliary but structurally central to the method’s performance.