Motion-Semantics DETR for Joint Video Analysis

• The paper introduces MS-DETR, a joint framework for video moment retrieval and highlight detection that disentangles motion and semantics to boost discrimination.
• It employs a Motion-Semantics Disentangled Encoder and a Mutual Task-Collaborated Decoder to fuse motion and semantic features through dual transformer towers.
• The approach overcomes dataset sparsity with motion-semantics corpus generation and contrastive denoising, achieving significant performance gains on multiple benchmarks.

Motion-Semantics DETR (MS-DETR) is a DETR-based joint framework for Video Moment Retrieval (MR) and Highlight Detection (HD) that seeks to capture rich motion-semantics features through unified learning. It is motivated by three limitations identified in prior DETR-based joint models: motion and semantics are typically collapsed too early, MR and HD are only weakly coupled, and MR/HD datasets exhibit an inherent sparsity dilemma in both motion and semantics dimensions. To address these issues, the framework combines a Motion-Semantics Disentangled Encoder (MSDE), a Mutual Task-Collaborated Decoder (MTCD), and motion-semantics corpus generation with contrastive denoising learning (Ma et al., 16 Jul 2025).

1. Task formulation and conceptual scope

MS-DETR studies two language-conditioned video understanding tasks. In MR, the input is an untrimmed video $V$ composed of $L$ clips and a text query $Q$ composed of $M$ words, and the output is a temporal moment represented by center and span, $\{c,s\}$. In HD, the same video-query pair is used to predict a salience or relevance score for every clip, denoted $S(\cdot)$ (Ma et al., 16 Jul 2025).

The paper formalizes three input feature streams: $$F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},$$

$$F_v^m = \{v_1^m, v_2^m, \ldots, v_L^m\} \in \mathbb{R}^{L \times d},$$

$$F_t = \{t_1, t_2, \ldots, t_M\} \in \mathbb{R}^{M \times d}.$$

Here, $F_v^s$ denotes clip-level semantic video features, $L$0 denotes clip-level motion video features, and $L$1 denotes word-level text features. The method uses frozen CLIP features for semantic video and text inputs, and frozen SlowFast features for motion inputs (Ma et al., 16 Jul 2025).

A central claim of the framework is that MR and HD are complementary rather than merely co-trained. HD provides clip-wise salience priors that can guide MR toward relevant temporal regions, while MR provides precise temporal boundaries and foreground/background discrimination that can sharpen HD. The paper therefore uses “joint learning” in two distinct senses: joint modeling of motion and semantics dimensions, and joint optimization of MR and HD with explicit task interaction (Ma et al., 16 Jul 2025).

2. Motion-Semantics Disentangled Encoder

The Motion-Semantics Disentangled Encoder is the representational core of MS-DETR. Its purpose is to avoid the early collapse of motion and semantics into a single global video representation, which the paper argues obscures motion-specific and semantic-specific discrimination and their different relationships with the text query (Ma et al., 16 Jul 2025).

MSDE uses two separate query-guided cross-modal towers:

• TMCT: Temporal Motion Cross-modal Transformer
• SSCT: Spatial Semantics Cross-modal Transformer

Both towers contain two layers of cross-attention transformers. The text features are used as key and value,

$L$2

while the branch queries are

$L$3

The resulting query-guided video representation is defined as

$L$4

The surrounding description states that TMCT captures dynamics and transitions between consecutive clips for sequence and duration understanding, whereas SSCT focuses on static visual details for contextual relevance within candidate clips. This yields a query-conditioned video sequence

$L$5

After this disentangled fusion, the model appends a learnable salience token $L$6,

$L$7

and feeds the sequence into a Transformer encoder to obtain

$L$8

The encoder outputs serve a dual role: $L$9 are clip-level features for HD, while the full memory $Q$0 is used by the decoder for MR. The salience token feature $Q$1 is described as an adaptive predictor related to matched versus unmatched video-text pairs (Ma et al., 16 Jul 2025).

3. Mutual Task-Collaborated Decoder

The Mutual Task-Collaborated Decoder implements the reciprocal coupling between HD and MR. Its first direction of influence is from HD to MR: predicted clip salience is used to construct decoder queries for moment retrieval. Its second direction is from MR back to HD: retrieval supervision is used to refine HD salience and temporal references (Ma et al., 16 Jul 2025).

For each clip representation $Q$2, the salience score is computed as

$Q$3

The model then selects the top-$Q$4 salient clips to form semantic content queries: $Q$5 where

$Q$6

Each selected content query $Q$7 is passed through an auxiliary span head to generate a temporal reference: $Q$8

This temporal reference is then converted into a position query through sinusoidal encoding: $Q$9 with

$M$0

The decoder therefore receives both semantic content queries $M$1 and temporal position queries $M$2. This input-adaptive query construction is one of the paper’s main departures from earlier DETR-style MR/HD models, where decoder queries are more weakly informed by HD outputs. Conversely, the model uses MR losses to supervise the temporal references $M$3 and the clip salience scores $M$4, so that HD benefits directly from moment-level localization supervision. The paper presents this as a remedy to the information bottleneck created when HD is confined to the encoder and MR to the decoder (Ma et al., 16 Jul 2025).

4. Sparsity dilemma and motion-semantics corpus generation

A distinctive element of MS-DETR is its diagnosis of an inherent sparsity dilemma in MR/HD datasets. The paper argues that video content is much richer in both semantic and motion dimensions than what is expressed by a short text query or sparse annotations. It identifies semantic sparsity in datasets such as QVHighlights and motion sparsity in datasets such as TACoS, and treats this mismatch as a data problem in addition to a modeling problem (Ma et al., 16 Jul 2025).

To enrich semantic supervision, the method uses two generation strategies. First, it performs finer caption generation with pretrained LLaVA for each video clip in the ground truth, determines corresponding temporal intervals by thresholding a cosine-similarity matrix between CLIP visual and textual embeddings, filters captions whose associated intervals are shorter than 3 clips, and selects the top-2 caption-video pairs based on clip interval length. Second, it performs query rewriting by replacing nouns in text queries with synonyms or antonyms. Synonym-based rewrites become positive pairs, and antonym-based rewrites become hard-negative pairs (Ma et al., 16 Jul 2025).

For motion-side enrichment, the paper applies a similar rewriting strategy, but replaces verbs rather than nouns. These generated motion variants also produce positive and hard-negative pairs. The intended effect is to expose the model to richer action descriptions while preserving the same general training structure (Ma et al., 16 Jul 2025).

This corpus expansion is paired with contrastive denoising learning because the generated auxiliary data is acknowledged to be noisy. A plausible implication is that MS-DETR treats motion-semantic richness as both a representational factorization problem and a supervision-density problem, which distinguishes it from DETR-style joint models that modify architecture alone (Ma et al., 16 Jul 2025).

5. Losses and denoising objectives

MS-DETR is trained with a composite objective spanning MR localization, HD collaboration, decoder denoising, and encoder-side contrastive ranking. The MR loss is

$M$5

The HD collaboration loss is

$M$6

The denoising branch perturbs ground-truth moments

$M$7

encodes them, and decodes them as

$M$8

The corresponding contrastive denoising loss is

$M$9

For encoder-side ranking, the paper adds three further terms. The negative-pair salience suppression loss is

$\{c,s\}$0

The margin ranking loss is

$\{c,s\}$1

with default margin

$\{c,s\}$2

The encoder contrastive loss is

$\{c,s\}$3

The full objective is

$\{c,s\}$4

The paper also specifies the denoising perturbation regime: $\{c,s\}$5 for positive noise scale and $\{c,s\}$6 for negative noise scale, with perturbations

$\{c,s\}$7

This design is intended to make decoder training robust to imperfect auxiliary supervision and noisy generated pairs (Ma et al., 16 Jul 2025).

6. Benchmarks, quantitative results, and ablations

MS-DETR is evaluated on four benchmarks: QVHighlights for joint MR/HD, Charades-STA and TACoS for MR, and TVSum for HD. On QVHighlights test, it reports MR results of mAP@0.5 66.41, [email protected] 44.91, Avg mAP 44.89, [email protected] 64.72, and [email protected] 48.77, together with HD mAP 40.45 and HIT@1 65.95. On the validation split, it reports MR [email protected] 67.19, [email protected] 46.05, Avg mAP 46.00, [email protected] 66.9, [email protected] 51.68, and HD mAP 40.57 with HIT@1 66.58. The paper notes that, on test Avg MR mAP, this improves over CG-DETR from 42.86 to 44.89, a gain of 2.03 points; on validation Avg MR mAP, it improves over CG-DETR from 42.33 to 46.00 (Ma et al., 16 Jul 2025).

On TACoS validation, MS-DETR reports $\{c,s\}$8, $\{c,s\}$9, $S(\cdot)$0, and $S(\cdot)$1. On TACoS test, it reports $S(\cdot)$2, $S(\cdot)$3, $S(\cdot)$4, and $S(\cdot)$5. The paper highlights exact gains over CG-DETR test of 3.39 at $S(\cdot)$6 and 2.75 in $S(\cdot)$7. On Charades-STA, MS-DETR reaches $S(\cdot)$8, $S(\cdot)$9, $$F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},$$0, and $$F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},$$1, with reported improvements over CG-DETR of 1.18 at $$F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},$$2, 0.14 at $$F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},$$3, and 0.46 in $$F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},$$4. On TVSum, it achieves an average HD score of 88.43, exceeding TR-DETR at 88.1, CG-DETR at 86.8, and QD-DETR at 85.0 (Ma et al., 16 Jul 2025).

The ablation study on QVHighlights validation isolates the contribution of each component. The baseline records MR Avg mAP 40.71, HD mAP 39.27, and HIT@1 61.61. Adding only MSDE yields MR Avg mAP 42.76, HD mAP 40.21, and HIT@1 65.35. Adding only MTCD yields MR Avg mAP 42.84, HD mAP 39.64, and HIT@1 63.61. Auxiliary Data only gives MR Avg mAP 43.27, HD mAP 40.42, and HIT@1 65.55. The full model reaches MR Avg mAP 46.00, MR [email protected] 51.68, HD mAP 40.57, and HIT@1 66.58. A parameter-count fairness ablation further compares baseline and MS-DETR under matched encoder settings, reporting Avg mAP improvements from 40.71 to 42.08 for 2CAT + 2SAT, and from 41.13 to 42.76 for 4CAT + 2SAT, which the paper uses to argue that the gain is not reducible to parameter scaling alone (Ma et al., 16 Jul 2025).

7. Disambiguation, related directions, and limitations

The acronym “MS-DETR” is overloaded in the literature. In the present context it denotes Motion-Semantics DETR for joint MR/HD (Ma et al., 16 Jul 2025). It does not denote Moment Sampling DETR, which is a proposal-based method for Natural Language Video Localization and explicitly states that “MS-DETR does not mean ‘Motion-Semantics DETR’” (Wang et al., 2023). Nor does it denote Multispectral pedestrian detection transformer, where MS-DETR refers to a multispectral pedestrian detector with loosely coupled fusion and modality-balanced optimization (Xing et al., 2023).

Within the broader DETR landscape, MS-DETR is specifically motion-semantic rather than merely motion-aware. This differentiates it from end-to-end tracking frameworks such as the Motion-Aware Transformer, whose contribution is explicit motion-conditioned query propagation and query-collision reduction, but not an explicit semantics branch (Yang et al., 26 Sep 2025). A plausible implication is that MS-DETR also belongs to a wider family of architectures that separate heterogeneous motion and semantic information rather than forcing both into a single latent representation; this design logic closely resembles divide-and-merge motion/semantic learning in end-to-end autonomous driving, where separate motion and semantic queries are used to mitigate negative transfer (Shen et al., 11 Feb 2025).

The paper states one explicit limitation: it does not extensively incorporate other modalities, especially audio, which may constrain performance when non-visual context is important (Ma et al., 16 Jul 2025). An additional implication stated in the paper’s own discussion is that the method depends on pretrained CLIP and SlowFast features and on generated auxiliary data that may contain noise, which is why denoising and contrastive learning are built into the training design (Ma et al., 16 Jul 2025).