---
title: Motion-Semantics DETR for Joint Video Analysis
url: https://www.emergentmind.com/topics/motion-semantics-detr-ms-detr
type: topic
---

# Motion-Semantics DETR for Joint Video Analysis

Motion-Semantics DETR (MS-DETR) is a DETR-based joint framework for Video Moment Retrieval (MR) and Highlight Detection (HD) that seeks to capture rich motion-semantics features through unified learning. It is motivated by three limitations identified in prior DETR-based joint models: motion and semantics are typically collapsed too early, MR and HD are only weakly coupled, and MR/HD datasets exhibit an inherent sparsity dilemma in both motion and semantics dimensions. To address these issues, the framework combines a Motion-Semantics Disentangled Encoder (MSDE), a Mutual Task-Collaborated Decoder (MTCD), and motion-semantics corpus generation with contrastive denoising learning [2507.12062].

## 1. Task formulation and conceptual scope

MS-DETR studies two language-conditioned video understanding tasks. In MR, the input is an untrimmed video \(V\) composed of \(L\) clips and a text query \(Q\) composed of \(M\) words, and the output is a temporal moment represented by center and span, \(\{c,s\}\). In HD, the same video-query pair is used to predict a salience or relevance score for every clip, denoted \(S(\cdot)\) [2507.12062].

The paper formalizes three input feature streams:
\[
F_v^s = \{v_1^s, v_2^s, \ldots, v_L^s\} \in \mathbb{R}^{L \times d},
\]
\[
F_v^m = \{v_1^m, v_2^m, \ldots, v_L^m\} \in \mathbb{R}^{L \times d},
\]
\[
F_t = \{t_1, t_2, \ldots, t_M\} \in \mathbb{R}^{M \times d}.
\]

Here, \(F_v^s\) denotes clip-level semantic video features, \(F_v^m\) denotes clip-level motion video features, and \(F_t\) denotes word-level text features. The method uses frozen CLIP features for semantic video and text inputs, and frozen SlowFast features for motion inputs [2507.12062].

A central claim of the framework is that MR and HD are complementary rather than merely co-trained. HD provides clip-wise salience priors that can guide MR toward relevant temporal regions, while MR provides precise temporal boundaries and foreground/background discrimination that can sharpen HD. The paper therefore uses “joint learning” in two distinct senses: joint modeling of motion and semantics dimensions, and joint optimization of MR and HD with explicit task interaction [2507.12062].

## 2. Motion-Semantics Disentangled Encoder

The Motion-Semantics Disentangled Encoder is the representational core of MS-DETR. Its purpose is to avoid the early collapse of motion and semantics into a single global video representation, which the paper argues obscures motion-specific and semantic-specific discrimination and their different relationships with the text query [2507.12062].

MSDE uses two separate query-guided cross-modal towers:

- **TMCT**: Temporal Motion Cross-modal Transformer  
- **SSCT**: Spatial Semantics Cross-modal Transformer  

Both towers contain two layers of cross-attention transformers. The text features are used as key and value,
\[
K_t = V_t = F_t,
\]
while the branch queries are
\[
Q_v^m = F_v^m, \qquad Q_v^s = F_v^s.
\]

The resulting query-guided video representation is defined as
\[
\hat{F_v} = \phi\left(\left(\mbox{TMCT}(Q^m_v, K_t, V_t)\oplus \mbox{SSCT}(Q^t_v, K_t, V_t)\right)\right). \tag{1}
\]

The surrounding description states that TMCT captures dynamics and transitions between consecutive clips for sequence and duration understanding, whereas SSCT focuses on static visual details for contextual relevance within candidate clips. This yields a query-conditioned video sequence
\[
\hat{F_v} = \{\hat{v}_1, \hat{v}_2, \ldots, \hat{v}_L\} \in \mathbb{R}^{L \times d}.
\]

After this disentangled fusion, the model appends a learnable salience token \(v_s\),
\[
\tilde{F_v} = \{v_s, \hat{v}_1, \hat{v}_2, \ldots, \hat{v}_L\},
\]
and feeds the sequence into a Transformer encoder to obtain
\[
X = \{x_s, x_1, x_2, \ldots, x_L\}.
\]

The encoder outputs serve a dual role: \(x_1,\ldots,x_L\) are clip-level features for HD, while the full memory \(X\) is used by the decoder for MR. The salience token feature \(x_s\) is described as an adaptive predictor related to matched versus unmatched video-text pairs [2507.12062].

## 3. Mutual Task-Collaborated Decoder

The Mutual Task-Collaborated Decoder implements the reciprocal coupling between HD and MR. Its first direction of influence is from HD to MR: predicted clip salience is used to construct decoder queries for moment retrieval. Its second direction is from MR back to HD: retrieval supervision is used to refine HD salience and temporal references [2507.12062].

For each clip representation \(x_i\), the salience score is computed as
\[
S(x_i) = \frac{\omega_s^T x_s \cdot \omega_v^T x_i}{\sqrt{d}}. \tag{3}
\]

The model then selects the top-\(K\) salient clips to form semantic content queries:
\[
Q_c = \{x_k\in X' \mid k \in \operatorname{top-K}(S(x_k))\}, \tag{2}
\]
where
\[
X' = X \setminus \{x_s\} = \{x_1, x_2, \ldots, x_L\}.
\]

Each selected content query \(q_k \in Q_c\) is passed through an auxiliary span head to generate a temporal reference:
\[
R_k = \mbox{MLP}_{Span}(q_k), \quad q_k \in Q_c. \tag{4}
\]

This temporal reference is then converted into a position query through sinusoidal encoding:
\[
\Phi(R_k) = \left( \sin\left(\frac{2\pi R_k}{10000^{2i/\frac{d}{2}}}\right) \oplus \cos\left(\frac{2\pi R_k}{10000^{(2i+1)/\frac{d}{2}}}\right) \right), \tag{5}
\]
with
\[
i\in\{0, ..., \frac{d-1}{2}\}.
\]

The decoder therefore receives both semantic content queries \(Q_c\) and temporal position queries \(Q_p\). This input-adaptive query construction is one of the paper’s main departures from earlier DETR-style MR/HD models, where decoder queries are more weakly informed by HD outputs. Conversely, the model uses MR losses to supervise the temporal references \(R_k\) and the clip salience scores \(S(x_i)\), so that HD benefits directly from moment-level localization supervision. The paper presents this as a remedy to the information bottleneck created when HD is confined to the encoder and MR to the decoder [2507.12062].

## 4. Sparsity dilemma and motion-semantics corpus generation

A distinctive element of MS-DETR is its diagnosis of an inherent sparsity dilemma in MR/HD datasets. The paper argues that video content is much richer in both semantic and motion dimensions than what is expressed by a short text query or sparse annotations. It identifies semantic sparsity in datasets such as QVHighlights and motion sparsity in datasets such as TACoS, and treats this mismatch as a data problem in addition to a modeling problem [2507.12062].

To enrich semantic supervision, the method uses two generation strategies. First, it performs finer caption generation with pretrained LLaVA for each video clip in the ground truth, determines corresponding temporal intervals by thresholding a cosine-similarity matrix between CLIP visual and textual embeddings, filters captions whose associated intervals are shorter than 3 clips, and selects the top-2 caption-video pairs based on clip interval length. Second, it performs query rewriting by replacing nouns in text queries with synonyms or antonyms. Synonym-based rewrites become positive pairs, and antonym-based rewrites become hard-negative pairs [2507.12062].

For motion-side enrichment, the paper applies a similar rewriting strategy, but replaces verbs rather than nouns. These generated motion variants also produce positive and hard-negative pairs. The intended effect is to expose the model to richer action descriptions while preserving the same general training structure [2507.12062].

This corpus expansion is paired with contrastive denoising learning because the generated auxiliary data is acknowledged to be noisy. A plausible implication is that MS-DETR treats motion-semantic richness as both a representational factorization problem and a supervision-density problem, which distinguishes it from DETR-style joint models that modify architecture alone [2507.12062].

## 5. Losses and denoising objectives

MS-DETR is trained with a composite objective spanning MR localization, HD collaboration, decoder denoising, and encoder-side contrastive ranking. The MR loss is
\[
\mathcal{L}_{MR} = \lambda^{MR}_{L1}||m - \hat{m}|| + \lambda^{MR}_{gIoU}\mathcal{L}_{gIoU}\left(m, \hat{m}\right) + \lambda^{MR}_{ce}\mathcal{L}_{ce}(f_i, y_i). \tag{6}
\]

The HD collaboration loss is
\[
\begin{aligned}
\mathcal{L}_{collab}^{HD} = \lambda^{HD}_{L1}&||m - R_k|| + \lambda^{HD}_{gIoU}\mathcal{L}_{gIoU}\left(m, R_k\right) \\
&+\lambda^{HD}_{ce}\mathcal{L}_{ce}(Sigmoid(S(x_i)), y_i).
\end{aligned} \tag{7}
\]

The denoising branch perturbs ground-truth moments
\[
m_{noise} = \{[c_i + \triangle c_i, s_i + \triangle s_i]\}, \quad i \in \{1,\ldots,G\}, \tag{10}
\]
encodes them, and decodes them as
\[
\tilde{m} = \mbox{MTCD}\left(\Phi \left( m_{noise} \right), X\right). \tag{9}
\]
The corresponding contrastive denoising loss is
\[
\mathcal{L}_{cont\_dn} = \lambda^{DN}_{L1}||m - \tilde{m}|| + \lambda^{DN}_{gIoU}\mathcal{L}_{gIoU}\left(m, \tilde{m}\right) + \lambda^{DN}_{ce}\mathcal{L}_{ce} (f_i, y_i). \tag{8}
\]

For encoder-side ranking, the paper adds three further terms. The negative-pair salience suppression loss is
\[
\mathcal{L}_{enc\_neg} = -\sum_{x_i \in X_{neg}} \log \left( 1 - Sigmoid\left(S\left(x_i \right)\right) \right). \tag{11}
\]

The margin ranking loss is
\[
\begin{aligned}
\mathcal{L}_{margin} & = \operatorname{max}\left(0, \delta + S(x_{l}) - S(x_{h})\right) \\
&+ \operatorname{max}\left(0, \delta + S(x_{out}) - S(x_{in})\right),
\end{aligned} \tag{12}
\]
with default margin
\[
\delta = 0.2.
\]

The encoder contrastive loss is
\[
\mathcal{L}_{enc\_cont} = - \frac{1}{N} \sum_{n=1}^N{\log{\left( \frac{\sum_{x_i \in X_{pos}^n} \exp(S\left( x_i \right) / \xi)}{\sum_{x \in {X_{pos}^n \cup X_{neg}^n} \exp(S\left( x_i \right) / \xi)} \right)}}. \tag{13}
\]

The full objective is
\[
\begin{aligned}
\mathcal{L}_{total} = &\mathcal{L}^{HD}_{collab} + \mathcal{L}_{MR} + \lambda_1\mathcal{L}_{cont\_dn} \\
&+ \lambda_2(\mathcal{L}_{enc\_neg} + \mathcal{L}_{margin} + \mathcal{L}_{enc\_cont}).
\end{aligned} \tag{14}
\]

The paper also specifies the denoising perturbation regime: \(\lambda_1 \in [0,1]\) for positive noise scale and \(\lambda_2 \in [1,2]\) for negative noise scale, with perturbations
\[
|\triangle c| = \frac{\delta_2 \lambda \cdot s}{2}, \qquad |\triangle s| = \frac{\delta_2 \lambda \cdot s}{2}.
\]
This design is intended to make decoder training robust to imperfect auxiliary supervision and noisy generated pairs [2507.12062].

## 6. Benchmarks, quantitative results, and ablations

MS-DETR is evaluated on four benchmarks: QVHighlights for joint MR/HD, Charades-STA and TACoS for MR, and TVSum for HD. On QVHighlights test, it reports MR results of mAP@0.5 **66.41**, mAP@0.75 **44.91**, Avg mAP **44.89**, R1@0.5 **64.72**, and R1@0.7 **48.77**, together with HD mAP **40.45** and HIT@1 **65.95**. On the validation split, it reports MR mAP@0.5 **67.19**, mAP@0.75 **46.05**, Avg mAP **46.00**, R1@0.5 **66.9**, R1@0.7 **51.68**, and HD mAP **40.57** with HIT@1 **66.58**. The paper notes that, on test Avg MR mAP, this improves over CG-DETR from **42.86** to **44.89**, a gain of **2.03** points; on validation Avg MR mAP, it improves over CG-DETR from **42.33** to **46.00** [2507.12062].

On TACoS validation, MS-DETR reports \(R@0.3 = 53.16\), \(R@0.5 = 39.65\), \(R@0.7 = 23.42\), and \(mIoU = 37.01\). On TACoS test, it reports \(R@0.3 = 56.51\), \(R@0.5 = 43.00\), \(R@0.7 = 25.37\), and \(mIoU = 39.23\). The paper highlights exact gains over CG-DETR test of **3.39** at \(R@0.5\) and **2.75** in \(mIoU\). On Charades-STA, MS-DETR reaches \(R@0.3 = 71.34\), \(R@0.5 = 59.62\), \(R@0.7 = 36.48\), and \(mIoU = 50.59\), with reported improvements over CG-DETR of **1.18** at \(R@0.5\), **0.14** at \(R@0.7\), and **0.46** in \(mIoU\). On TVSum, it achieves an average HD score of **88.43**, exceeding TR-DETR at **88.1**, CG-DETR at **86.8**, and QD-DETR at **85.0** [2507.12062].

The ablation study on QVHighlights validation isolates the contribution of each component. The baseline records MR Avg mAP **40.71**, HD mAP **39.27**, and HIT@1 **61.61**. Adding only MSDE yields MR Avg mAP **42.76**, HD mAP **40.21**, and HIT@1 **65.35**. Adding only MTCD yields MR Avg mAP **42.84**, HD mAP **39.64**, and HIT@1 **63.61**. Auxiliary Data only gives MR Avg mAP **43.27**, HD mAP **40.42**, and HIT@1 **65.55**. The full model reaches MR Avg mAP **46.00**, MR R1@0.7 **51.68**, HD mAP **40.57**, and HIT@1 **66.58**. A parameter-count fairness ablation further compares baseline and MS-DETR under matched encoder settings, reporting Avg mAP improvements from **40.71** to **42.08** for **2CAT + 2SAT**, and from **41.13** to **42.76** for **4CAT + 2SAT**, which the paper uses to argue that the gain is not reducible to parameter scaling alone [2507.12062].

## 7. Disambiguation, related directions, and limitations

The acronym “MS-DETR” is overloaded in the literature. In the present context it denotes **Motion-Semantics DETR** for joint MR/HD [2507.12062]. It does not denote **Moment Sampling DETR**, which is a proposal-based method for Natural Language Video Localization and explicitly states that “MS-DETR does not mean ‘Motion-Semantics DETR’” [2305.18969]. Nor does it denote **Multispectral pedestrian detection transformer**, where MS-DETR refers to a multispectral pedestrian detector with loosely coupled fusion and modality-balanced optimization [2302.00290].

Within the broader DETR landscape, MS-DETR is specifically motion-semantic rather than merely motion-aware. This differentiates it from end-to-end tracking frameworks such as the Motion-Aware Transformer, whose contribution is explicit motion-conditioned query propagation and query-collision reduction, but not an explicit semantics branch [2509.21715]. A plausible implication is that MS-DETR also belongs to a wider family of architectures that separate heterogeneous motion and semantic information rather than forcing both into a single latent representation; this design logic closely resembles divide-and-merge motion/semantic learning in end-to-end autonomous driving, where separate motion and semantic queries are used to mitigate negative transfer [2502.07631].

The paper states one explicit limitation: it does **not extensively incorporate other modalities, especially audio**, which may constrain performance when non-visual context is important [2507.12062]. An additional implication stated in the paper’s own discussion is that the method depends on pretrained CLIP and SlowFast features and on generated auxiliary data that may contain noise, which is why denoising and contrastive learning are built into the training design [2507.12062].

Source: https://www.emergentmind.com/topics/motion-semantics-detr-ms-detr