DRNet: Denoise-then-Retrieve Network

• The paper introduces DRNet, which first denoises text-irrelevant video clips to improve multimodal alignment in video moment retrieval.
• DRNet uses a three-stage architecture—text-conditioned denoising, text-reconstruction feedback, and conditional retrieval—to accurately predict target moments.
• Empirical results on benchmarks like QVHighlights and Charades-STA show that DRNet outperforms prior methods by effectively filtering noise and enhancing retrieval precision.

Denoise-then-Retrieve Network (DRNet) is a framework for text-driven Video Moment Retrieval (VMR), defined as the task of locating, in an untrimmed video, the temporal segment that best matches a free-form natural-language query. In the formulation used by the method, a video with $L_v$ clips and a query with $L_t$ words is mapped to a target moment represented as a center-span pair $(m_c, m_\sigma)$. DRNet is introduced with the central claim that conventional transformer-style VMR pipelines encode all clips jointly, including many text-irrelevant ones, and thereby degrade multimodal alignment. Its proposed remedy is a denoise-then-retrieve paradigm: first suppress text-irrelevant clips to produce purified multimodal representations, then perform retrieval on the purified sequence rather than over the full video (Liu et al., 15 Aug 2025).

1. Problem formulation and motivating observation

The method is motivated by the observation that in many VMR settings only a small fraction of the video is relevant to the query, while a large fraction acts as distractor content. On QVHighlights, the reported analysis shows that ground-truth clips often occupy less than 30% of the video, while noisy clips account for over 60% in most videos. Under a retrieve-over-all-clips pipeline, the model is required to force relevant and irrelevant clips into the same multimodal interaction process, which is described as contaminating the multimodal representation, weakening text-clip alignment, and making optimization harder because target evidence must be disentangled from overwhelming distractors (Liu et al., 15 Aug 2025).

This leads to a conceptual reformulation of VMR. Rather than treating denoising as an implicit byproduct of a localization head, DRNet narrows the search space before final retrieval. The shift is architectural as well as algorithmic: the method explicitly identifies text-irrelevant clips, constructs a clip-level noise mask, and passes only purified visual representations into the final retrieval stage. A plausible implication is that the method treats query-conditioned filtering as a first-class intermediate task rather than as an emergent property of localization training.

The target of denoising is also defined carefully. The method does not equate denoising with removing every non-ground-truth clip. It explicitly distinguishes ground-truth clips, challenging non-target frames that are partially aligned and can still provide useful context, and irrelevant or noisy clips that exhibit little or no alignment. This distinction is important because the denoising stage is intended to preserve semantically relevant temporal context around the target while suppressing unrelated clips.

2. Overall architecture and representational pipeline

DRNet consists of three stages: Text-Conditioned Denoising (TCD), Text-Reconstruction Feedback (TRF), and conditional retrieval on purified representations. The model begins with frozen pretrained feature extractors. Video features are the concatenation of CLIP and SlowFast features, while text features come from the CLIP text encoder. Both are projected into a common hidden dimension $D$ through FFNs, yielding

$$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$

TCD first injects textual information into video clips, performs multimodal context interaction, estimates clip relevance, and generates a noise mask. Its output is a purified visual sequence $\hat{\boldsymbol f_v}$. TRF is a training-time auxiliary path that reconstructs a sentence-level query embedding from the purified video representations and aligns it with the original text embedding. The final retrieval stage then operates on the denoised multimodal sequence and predicts the target moment through global retrieval, boundary prediction, text-conditioned foreground classification, and contrastive learning (Liu et al., 15 Aug 2025).

All CIO-based encoders contain three CIO layers, and the hidden dimension is

$D = 1024.$

The model is reported to run on a single RTX 3090 GPU. For Charades-STA, the work also reports results with VGG features in addition to SlowFast+CLIP, following prior settings. The features are pre-extracted and matched to those used by UniVTG for fair comparison. The paper does not provide explicit clip segmentation length or frame sampling details beyond inheriting pre-extracted features from prior work, so part of the reproduction protocol remains external to the DRNet description.

3. Text-Conditioned Denoising (TCD)

TCD is the defining component of DRNet. Its objective is to decide which clips are semantically irrelevant to the query and should be masked out before retrieval. It proceeds in three steps: cross-attention-based text injection, Context Interaction Operator (CIO)-based context modeling, and dynamic denoising through text-conditioned convolution (Liu et al., 15 Aug 2025).

The first operation is cross-attention from video to text, with video features acting as queries and text features as keys and values: $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$ This produces text-aware video features

$$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$

The weighted text features are projected through an MLP and integrated into the original video representations, although the update equation is not given explicitly. Functionally, this step makes each clip representation query-aware before the model estimates whether it is noise.

After cross-attention, DRNet applies the Context Interaction Operator, which is built from bi-directional Mamba blocks. The method emphasizes the linear complexity and selective information propagation of state space models for long-sequence processing. Position and modality embeddings are added to text, video, and learnable global tokens: $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$

$L_t$0

The paper contains a notation inconsistency, writing the global tokens as $L_t$1 while using $L_t$2 for sequence length; the intended meaning is a set of $L_t$3 learnable global tokens. Each CIO contains separate forward and backward Mamba blocks, with the backward output flipped and added to the forward output. Stacking three CIOs yields intra-modal context interaction, cross-modal context interaction, and global context integration. The outputs are context-aware textual, visual, and global features $L_t$4, $L_t$5, and $L_t$6.

The denoising step then estimates clip-level relevance using text-conditioned dynamic convolution. The word-level text feature $L_t$7 is pooled to a fixed length $L_t$8, and a fully connected layer generates dynamic kernels

$L_t$9

These are applied to the visual features: $(m_c, m_\sigma)$0 where $(m_c, m_\sigma)$1 denotes concatenation along the channel dimension and $(m_c, m_\sigma)$2 is a $(m_c, m_\sigma)$3 convolution for dimensionality reduction. A sigmoid then produces text-clip alignment scores,

$(m_c, m_\sigma)$4

and thresholding at $(m_c, m_\sigma)$5 yields a binary noise mask: $(m_c, m_\sigma)$6 Here $(m_c, m_\sigma)$7 denotes a relevant clip and $(m_c, m_\sigma)$8 a noisy clip. The notation uses $(m_c, m_\sigma)$9, but from context this corresponds to $D$0. The mask is therefore clip-wise rather than frame-wise. The masking stage is hard rather than soft, although the pre-threshold alignment scores remain continuous. The purified sequence passed onward is

$D$1

The exact masking operator is not written explicitly. The description states that $D$2 is applied to mask out noisy clips and generate purified visual representations, but it does not specify whether the implementation uses zeroing, elementwise multiplication, or another equivalent mechanism. This omission is one of the architectural details left unspecified.

4. Text-Reconstruction Feedback (TRF)

TRF is an auxiliary training-time supervision path designed to improve denoising quality. Its premise is that if the purified video representations truly retain query-relevant information, then they should support reconstruction of a sentence-level query embedding. TRF introduces a learnable query embedding $D$3 and a mapping network $D$4 built by stacking three CIOs. This token interacts with the purified visual features to generate a reconstructed sentence embedding: $D$5

The original sentence-level text embedding $D$6 is computed by average-pooling the word-level text embeddings $D$7. TRF then applies a cosine-similarity-based semantic consistency loss,

$D$8

The paper notes a formatting issue in the displayed equation, but its intended form is cosine distance scaled by $D$9.

The role of TRF is supervisory rather than generative in the broader sense. If TCD removes useful clips or retains too much noise, then the reconstructed query embedding drifts away from the original text embedding and the semantic consistency loss increases. In that way, TRF supplies auxiliary supervision specifically to the denoising stage. The ablation evidence supports this interpretation: removing TRF degrades retrieval, and removing both TRF and the decoder hurts more than removing the decoder alone, which the authors attribute to the semantic consistency feedback producing cleaner features (Liu et al., 15 Aug 2025).

The figure description indicates that TRF belongs to the training-only path, with dashed lines denoting components absent from inference. A practical implication is that TRF adds training complexity but does not necessarily impose the same cost at test time.

5. Retrieval heads, prediction targets, and optimization

After denoising, DRNet performs conditional retrieval on the purified multimodal sequence. A multimodal encoder $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$0, again built from three CIO layers, decodes the denoised input: $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$1 The resulting output sequence contains global, textual, and visual features $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$2, $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$3, and $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$4, which feed several prediction heads (Liu et al., 15 Aug 2025).

Global retrieval uses the global features $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$5 to regress the target moment directly as center and span. An MLP predicts the center coordinate $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$6, and a fully connected layer predicts the span $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$7. The loss is

$$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$8

The prose refers to generalized IoU, while the equation uses $$\boldsymbol V = [v_1, v_2, \dots, v_{L_v}], \qquad \boldsymbol T = [t_1, t_2, \dots, t_{L_t}].$$9; this is a notation inconsistency rather than a stated algorithmic difference.

Boundary prediction follows UniVTG. Three $\hat{\boldsymbol f_v}$0 convolution layers with $\hat{\boldsymbol f_v}$1 filters and ReLU operate on $\hat{\boldsymbol f_v}$2. The final layer outputs two channels corresponding to left and right offsets $\hat{\boldsymbol f_v}$3, which define predicted boundaries $\hat{\boldsymbol f_v}$4. Supervision is

$\hat{\boldsymbol f_v}$5

Text-conditioned foreground classification predicts whether each clip is foreground or background relative to the query. The text features $\hat{\boldsymbol f_v}$6 are pooled to fixed length $\hat{\boldsymbol f_v}$7, then mapped to convolution kernels

$\hat{\boldsymbol f_v}$8

This head shares the same structure as the boundary head, except that $\hat{\boldsymbol f_v}$9 is used as the first-layer convolution kernel and the output is a single foreground score $D = 1024.$0 per clip. The loss is binary cross-entropy: $D = 1024.$1

The model also includes intra-video and inter-video contrastive learning. Relevance between a clip embedding and a sentence-level text embedding $D = 1024.$2 is measured by cosine similarity $D = 1024.$3. The losses are

$D = 1024.$4

$D = 1024.$5

and the combined contrastive objective is

$D = 1024.$6

The full training objective combines semantic consistency, global localization, boundary regression, classification, and contrastive terms: $D = 1024.$7 The reported coefficients are

$D = 1024.$8

For contrastive learning, $D = 1024.$9 and $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$0 on QVHighlights, while $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$1 and $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$2 on Charades-STA. No curriculum learning is described. The paper also does not specify the exact test-time fusion rule among global retrieval, boundary prediction, and classification beyond stating that the purified representations are decoded for accurate retrieval.

6. Empirical results, transferability, and limitations

DRNet is evaluated on QVHighlights and Charades-STA. On the official QVHighlights test split, the reported results are $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$3 for [email protected], $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$4 for [email protected], $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$5 for [email protected], $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$6 for [email protected], and $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$7 for mAP@Avg. The method is reported to outperform prior methods including UVCOM, TR-DETR, MESM, UniVTG, QD-DETR, UMT, and MomentDiff. Against the strongest prior entries in the table, the reported margins are $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$8 on [email protected], $${\rm Attn}(Q_{\bold V}, K_{\bold T}, V_{\bold T}) = {\rm softmax}\!\left(\frac{Q_{\bold V}(K_{\bold T})^T}{\sqrt{D}}\right)V_{\bold T}.$$9 on [email protected], $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$0 on [email protected], and $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$1 on mAP@Avg versus TR-DETR, and $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$2, $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$3, $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$4, and $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$5 on the same metrics versus UVCOM (Liu et al., 15 Aug 2025).

On Charades-STA with VGG features, DRNet reports [email protected] of $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$6, [email protected] of $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$7, and a table entry for mIoU displayed as $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$8, with the table formatting noted as corrupted. The narrative nevertheless states clearly that DRNet surpasses MESM by $$\hat{\bold V} = [\hat v_1, \hat v_2, \dots, \hat v_{L_v}].$$9 percentage points on [email protected]. With SlowFast+CLIP features, the method is also reported to outperform TR-DETR, LLMEPET, and UVCOM, with emphasis on gains under stricter metrics such as [email protected].

The ablation results identify TCD as the principal source of improvement. On QVHighlights val, removing TCD reduces mAP@Avg from $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$0 to $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$1, described as an average drop of $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$2. Removing TRF also lowers performance. Replacing Mamba with Transformer in CIO reduces mAP@Avg from $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$3 to $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$4, a drop of about $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$5 points, suggesting that the state-space formulation is better suited in this setting than standard self-attention. Additional ablations show that removing cross-attention in TCD, replacing text-conditioned dynamic kernels with standard convolutions, or removing learnable global tokens all degrade performance. An ablation over depth reports that three CIO layers work best for TCD, TRF, and the decoder.

The denoise-then-retrieve paradigm is also presented as plug-and-play. Applying generated noise masks to other VMR models without modifying their architectures improves performance. For UniVTG, mAP@Avg rises from $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$6 to $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$7 and [email protected] from $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$8 to $$\tilde{\bold V} = \hat{\bold V} + \bold E^{p}_V + \bold E^{m}_V,\quad \tilde{\bold T} = \bold T + \bold E^{p}_T + \bold E^{m}_T,\quad \tilde{\bold G} = \bold G + \bold E^{p}_G + \bold E^{m}_G,$$9. For VMS, mAP@Avg rises from $L_t$00 to $L_t$01 and [email protected] from $L_t$02 to $L_t$03. This suggests that query-conditioned pre-filtering is not specific to DRNet’s internal decoder design.

The qualitative visualizations support a two-stage interpretation. The blue curve corresponds to TCD text-clip alignment, the orange curve to decoder alignment, the red dashed line to the threshold $L_t$04, and gray clips to filtered regions. The reported behavior is that TCD assigns lower scores to noisy clips and retains semantically relevant clips, while the decoder further sharpens localization within the retained region.

Several limitations and open questions are explicitly acknowledged or follow directly from the reported formulation. Hard thresholding by $L_t$05 can, in principle, over-filter weakly relevant clips, and sensitivity to $L_t$06 is not analyzed in detail. Because some non-target clips can provide useful temporal context, an inaccurate mask may damage downstream localization. Some implementation details remain underspecified, including the exact masking operator and the final test-time fusion among prediction heads. The system also depends on frozen CLIP and SlowFast features, so performance may depend materially on feature quality. Even with those caveats, the reported evidence supports the broader claim that filtering irrelevant clips before retrieval improves both optimization and representation quality in VMR, particularly under stricter IoU criteria (Liu et al., 15 Aug 2025).