---
title: 'MeViS-M: Temporal Annotation for RVOS'
url: https://www.emergentmind.com/topics/mevis-m
type: topic
---

# MeViS-M: Temporal Annotation for RVOS

MeViS-M is a temporally grounded extension of the MeViS benchmark for referring video object segmentation (RVOS). It augments the original motion-expression-centric MeViS dataset with manually annotated object-wise temporal moments that specify when an object is semantically referred to by a language expression, rather than merely when it is visible. In this formulation, the supervision problem shifts from generic expression-conditioned segmentation over arbitrary clips to semantically aligned text-to-clip learning, enabling moment-aware propagation and object-level selective supervision. The concrete MeViS-M formulation is introduced together with the SAMDWICH framework, whereas the original MeViS benchmark established the broader motion-expression RVOS setting in which targets often cannot be identified from a single frame alone [2508.11955], [2308.08544].

## 1. Historical placement and nomenclature

MeViS was introduced as a large-scale benchmark for video segmentation with motion expressions, with **2,006 videos**, **8,171 objects**, **28,570 motion expressions**, and **443k masks**. Its defining premise is that referring expressions emphasize motion attributes, so the referred target often cannot be identified from one frame alone; MeViS also supports multi-object expressions, making it a harder variant of RVOS than datasets dominated by static attributes [2308.08544].

Within subsequent challenge reports and solution papers, a recurrent nomenclature issue is that **“MeViS-M” is often not explicitly defined**. The Sa2VA-based 3rd-place report for the 4th PVUW MeViS challenge states that it does not explicitly mention “MeViS-M” anywhere and reports only MeViS results [2504.00476]. The ReferDINO-Plus challenge report likewise discusses MeViS, single-object versus multi-object cases, and multi-object expressions, but does not define a separate benchmark split or official metric labeled MeViS-M [2503.23509]. The PVUW 2026 challenge report also notes that the exact phrase “MeViS-M” does not appear, and instead discusses MeViS-Text and MeViS-Audio as modality-specific challenge tracks [2604.26031].

In the literature summarized here, the term acquires a concrete technical meaning in SAMDWICH: **MeViS-M is an annotation extension of the MeViS RVOS benchmark** that introduces explicit temporal supervision about when objects are referred to by language [2508.11955]. This usage distinguishes MeViS-M from informal interpretations such as “multi-object MeViS” or “multimodal MeViS.”

## 2. Semantic annotation model

The central annotation primitive in MeViS-M is the **temporal moment**. A moment is not defined as the span in which an object is visible. Instead, it is the set of frames during which the object is **semantically relevant** to the referring expression. This distinction is fundamental for MeViS-style motion language, because an object may be visible throughout a video while the text applies only during a specific action or interaction interval [2508.11955].

For each object \(i\), MeViS-M defines an object-specific moment index set
\[
\mathcal{M}_i \subseteq \{1,\dots,T_V\},
\]
where \(T_V\) is the number of video frames. The union of these sets gives the text-relevant frames,
\[
\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,
\]
and the complement gives the text-irrelevant frames,
\[
\mathcal{M}^- = \{1,\dots,T_V\}\setminus \mathcal{M}^+.
\]
The paper gives a toy example: if \(\mathcal{M}_1=\{1,2,3\}\) and \(\mathcal{M}_2=\{3,5\}\), then
\[
\mathcal{M}^+=\{1,2,3,5\}, \qquad \mathcal{M}^-=\{4\}.
\]
This makes explicit that moment annotations are expression-object semantics, not visibility metadata [2508.11955].

A common misconception is to equate MeViS-M moments with key frames or action boundaries. The annotation is broader than either. It supports frame-level distinction between semantically aligned and semantically irrelevant supervision, and it can differ across referred objects within the same expression.

## 3. Dataset curation and composition

MeViS-M is built on top of the original MeViS dataset and adds **manually annotated temporal moments**. In the SAMDWICH description, original MeViS has three splits: **train with 1,662 videos**, **valid\(_u\) with 50 videos**, and **valid with 140 videos**. MeViS-M adds detailed object-wise moment annotations for **train** and **valid\(_u\)**. For the **valid** split, because mask ground truth is unavailable, the annotation is only at the video level for expression-relevant frames rather than object-specific moment spans [2508.11955].

The annotation process was carried out manually by **about 20 annotators**. During this process, the authors also curated the underlying benchmark labels. They removed training samples where referred objects lacked valid masks, eliminating **66 videos** and **1,278 expressions** from the original training set. They further corrected missing referred objects, fixed label-ID switches, removed masks for irrelevant objects, and excluded cases in which partial segmentation errors could not be repaired because the original masks were insufficient [2508.11955].

After curation, the dataset used for training contains **27,292 motion-focused expressions** across **1,596 training videos**, **50 valid\(_u\) videos**, and **140 valid videos**. The MeViS-M paper does **not** report the total number of object instances, the total number of annotated moments, category counts, average moment duration, or average expression length. This omission matters because it limits purely statistical characterization of the annotation extension [2508.11955].

The following summary captures the dataset relation.

| Resource | Key content | Quantitative details |
|---|---|---|
| MeViS | Motion-expression RVOS benchmark | 2,006 videos, 8,171 objects, 28,570 expressions, 443k masks |
| MeViS-M | MeViS plus explicit temporal moment annotation and label curation | 27,292 expressions across 1,596 training videos, 50 valid\(_u\), 140 valid |

## 4. Training regime enabled by MeViS-M

The main methodological importance of MeViS-M is that it enables **semantically aligned text-to-clip pairs**. Standard RVOS training typically samples frames indiscriminately and supervises all visible objects in those frames, even when the expression refers only to a subset of objects and only during certain parts of the video. MeViS-M replaces that assumption with explicit knowledge of \(\mathcal{M}^+\) and \(\mathcal{M}^-\), so clip construction and supervision can be conditioned on temporal semantics rather than visibility alone [2508.11955].

In SAMDWICH, clip sampling depends directly on these annotations. The implementation samples **8 frames per video for Hiera-Base** and **6 for Hiera-Large**; **half the frames are always sampled from \(\mathcal{M}^+\)**, while the other half are drawn from either \(\mathcal{M}^+\) or \(\mathcal{M}^-\). This means training clips deliberately mix semantically aligned and semantically irrelevant frames, but the model now knows which is which [2508.11955].

This regime supports **Moment-guided Dual-path Propagation (MDP)**. For frames in \(\mathcal{M}^+\), the model applies language-conditioned features and text prompts; for frames in \(\mathcal{M}^-\), it uses frozen SAM2 visual features to avoid what the paper calls semantic contamination. The memory-attended representations are described as
\[
\mathbf{F}_{\text{mem}^{+}} = \mathrm{MemoryAttn}(\mathbf{F}_{\text{Adp}}, \mathcal{B}), \qquad
\mathbf{F}_{\text{mem}^{-}} = \mathrm{MemoryAttn}(\mathbf{F}_{\text{SAM}}, \mathcal{B}),
\]
and decoding follows
\[
\mathbf{P}= \begin{cases}
\mathcal{D}(\mathbf{F}_{\text{mem}^{+}}, \rho_{\text{txt}}), & t\in \mathcal{M}^{+},\\[4pt]
\mathcal{D}(\mathbf{F}_{\text{mem}^{-}}), & t\in \mathcal{M}^{-}.
\end{cases}
\]
The text prompt is formed from adapter-enhanced textual features as
\[
\rho_\text{txt} = \operatorname{MLP}\!\left(\psi(\mathbf{E}_\text{C}, \mathbf{E}_\text{M})\right).
\]
Only \(\mathcal{M}^+\) features are stored in the memory bank, and the implementation attends to features from the **6 nearest frames** in memory [2508.11955].

MeViS-M also enables **Object-level Selective Supervision (OSS)**. Given a sampled frame index set \(\mathcal{T}\), object \(i\) is discarded if
\[
\mathcal{T}\cap \mathcal{M}_i = \emptyset.
\]
Only objects satisfying \(\mathcal{T}\cap \mathcal{M}_i \neq \emptyset\) are retained in the filtered supervision set. The training loss is then applied to this filtered target:
\[
\mathcal{L} =
\lambda_{\mathrm{Dice}}\,\mathcal{L}_{\mathrm{Dice}}(\mathbf{Y}_{\mathrm{OSS}}, \mathbf{P}_{\mathcal{T}})
+
\lambda_{\mathrm{Foc}}\,\mathcal{L}_{\mathrm{Foc}}(\mathbf{Y}_{\mathrm{OSS}}, \mathbf{P}_{\mathcal{T}}).
\]
The paper does not provide numerical values for \(\lambda_{\mathrm{Dice}}\) or \(\lambda_{\mathrm{Foc}}\) [2508.11955].

## 5. SAMDWICH and empirical effects of moment-aware supervision

The MeViS-M annotations were introduced together with **SAMDWICH**, a moment-aware RVOS framework. Its reported ablations directly quantify the value of moment-aware supervision on the **valid\(_u\)** split. A baseline without moment-aware design scores **56.8 \(\mathcal{J}{data}\mathcal{F}\)**. Training with MeViS-M moment-aware sampling raises this to **58.0**, a gain of **+1.2**. Adding MDP yields **59.4**; adding OSS yields **58.3**; and combining all three reaches **60.8**, which is **+4.0** over the baseline [2508.11955].

The sampling ablation is especially revealing. **Random sampling** gives **56.8**, **BLIP-2-based sampling** gives **55.5**, and **MeViS-M-based sampling** gives **58.0**. The paper therefore reports MeViS-M as **+1.2** better than random sampling and **+2.5** better than BLIP-2-based keyframe sampling, supporting the argument that accurate temporal semantics are more useful than indiscriminate or noisy temporal selection [2508.11955].

The MDP ablation separates two roles of moment annotations. Without moment-aware feature enhancement and with memory built from all frames, the score is **58.0**. Restricting memory to \(\mathcal{M}^+\) raises this to **58.9**. Using moment-aware feature enhancement alone with memory from all frames gives **60.3**. Using both reaches **60.8**. This suggests that MeViS-M improves RVOS both by telling the model **where** language-conditioned features should be applied and by telling it **which** frames should populate memory [2508.11955].

The paper also reports that top-1 accuracy of several VLMs for keyframe selection is only **around 50%** on the valid split after excluding videos where the ground-truth moment covers the entire video. This is consistent with the claim that moment localization is itself difficult and that MeViS-M addresses a genuine supervision gap rather than a trivial annotation refinement [2508.11955].

## 6. Inference bottlenecks, scope, and broader significance

MeViS-M changes the assumptions of RVOS training in a substantive way. The expression is no longer treated as uniformly applicable to every visible frame and object. Instead, supervision becomes temporally selective and object selective. This enables aligned text-to-clip training, moment-aware propagation, and selective object filtering, and it suggests that a large portion of MeViS difficulty originates from semantic misalignment during training rather than only from backbone capacity [2508.11955].

At the same time, the framework built on top of MeViS-M reveals a new bottleneck: **moment retrieval at inference**. The SAMDWICH appendix reports a separate moment retrieval model, **Chrono**. On valid\(_u\), Chrono achieves **R1@.5 = 74.7**, **R1@.7 = 61.8**, **mAP = 62.1**, **mAP@.5 = 72.8**, **mAP@.75 = 60.0**, and downstream **\(\mathcal{J}{data}\mathcal{F} = 57.6\)**. On valid, it achieves **R1@.5 = 61.3**, **R1@.7 = 53.0**, **mAP = 52.2**, **mAP@.5 = 60.6**, **mAP@.75 = 51.2**, and downstream **\(\mathcal{J}{data}\mathcal{F} = 50.4\)**. Under ground-truth moments, the model reaches **60.8** on valid\(_u\) and **53.5** on valid; prioritizing top BLIP-2 frames within the ground-truth moment raises this further to **62.9** on valid\(_u\), **54.5** on valid with top-4, and **54.8** on valid with top-8 [2508.11955].

These results imply that MeViS-M is not merely a cleaner label set. It exposes a separate temporal grounding problem: identifying when the expression applies. A plausible implication is that future RVOS systems may need joint moment retrieval and segmentation rather than the conventional single-stage formulation. The paper also notes a limitation with expressions that describe **multiple temporally separated actions of the same object**, because the model uses a single expression-level feature across the whole video and may overemphasize one action span [2508.11955].

In summary, MeViS-M is best understood as a moment-annotated, semantically curated extension of MeViS that makes object-wise temporal relevance explicit. Its primary significance lies in formalizing a supervision axis that earlier MeViS challenge papers left implicit: not only **which** object the language refers to, but **when** the language refers to it.

Source: https://www.emergentmind.com/topics/mevis-m