MeViS-M: Temporal Annotation for RVOS

• MeViS-M is a temporally grounded extension of the MeViS benchmark that explicitly annotates the moments when objects are semantically referenced by language.
• It introduces manual object-wise temporal moment annotations, enabling moment-aware propagation and object-level selective supervision for improved clip construction.
• Empirical results using the SAMDWICH framework show that moment-aware sampling and supervision can boost RVOS performance by up to +4.0 over baseline methods.

MeViS-M is a temporally grounded extension of the MeViS benchmark for referring video object segmentation (RVOS). It augments the original motion-expression-centric MeViS dataset with manually annotated object-wise temporal moments that specify when an object is semantically referred to by a language expression, rather than merely when it is visible. In this formulation, the supervision problem shifts from generic expression-conditioned segmentation over arbitrary clips to semantically aligned text-to-clip learning, enabling moment-aware propagation and object-level selective supervision. The concrete MeViS-M formulation is introduced together with the SAMDWICH framework, whereas the original MeViS benchmark established the broader motion-expression RVOS setting in which targets often cannot be identified from a single frame alone (Lee et al., 16 Aug 2025, Ding et al., 2023).

1. Historical placement and nomenclature

MeViS was introduced as a large-scale benchmark for video segmentation with motion expressions, with 2,006 videos, 8,171 objects, 28,570 motion expressions, and 443k masks. Its defining premise is that referring expressions emphasize motion attributes, so the referred target often cannot be identified from one frame alone; MeViS also supports multi-object expressions, making it a harder variant of RVOS than datasets dominated by static attributes (Ding et al., 2023).

Within subsequent challenge reports and solution papers, a recurrent nomenclature issue is that “MeViS-M” is often not explicitly defined. The Sa2VA-based 3rd-place report for the 4th PVUW MeViS challenge states that it does not explicitly mention “MeViS-M” anywhere and reports only MeViS results (Yuan et al., 1 Apr 2025). The ReferDINO-Plus challenge report likewise discusses MeViS, single-object versus multi-object cases, and multi-object expressions, but does not define a separate benchmark split or official metric labeled MeViS-M (Liang et al., 30 Mar 2025). The PVUW 2026 challenge report also notes that the exact phrase “MeViS-M” does not appear, and instead discusses MeViS-Text and MeViS-Audio as modality-specific challenge tracks (Liu et al., 28 Apr 2026).

In the literature summarized here, the term acquires a concrete technical meaning in SAMDWICH: MeViS-M is an annotation extension of the MeViS RVOS benchmark that introduces explicit temporal supervision about when objects are referred to by language (Lee et al., 16 Aug 2025). This usage distinguishes MeViS-M from informal interpretations such as “multi-object MeViS” or “multimodal MeViS.”

2. Semantic annotation model

The central annotation primitive in MeViS-M is the temporal moment. A moment is not defined as the span in which an object is visible. Instead, it is the set of frames during which the object is semantically relevant to the referring expression. This distinction is fundamental for MeViS-style motion language, because an object may be visible throughout a video while the text applies only during a specific action or interaction interval (Lee et al., 16 Aug 2025).

For each object $i$, MeViS-M defines an object-specific moment index set

$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$

where $T_V$ is the number of video frames. The union of these sets gives the text-relevant frames,

$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$

and the complement gives the text-irrelevant frames,

$$\mathcal{M}^- = \{1,\dots,T_V\}\setminus \mathcal{M}^+.$$

The paper gives a toy example: if $\mathcal{M}_1=\{1,2,3\}$ and $\mathcal{M}_2=\{3,5\}$, then

$$\mathcal{M}^+=\{1,2,3,5\}, \qquad \mathcal{M}^-=\{4\}.$$

This makes explicit that moment annotations are expression-object semantics, not visibility metadata (Lee et al., 16 Aug 2025).

A common misconception is to equate MeViS-M moments with key frames or action boundaries. The annotation is broader than either. It supports frame-level distinction between semantically aligned and semantically irrelevant supervision, and it can differ across referred objects within the same expression.

3. Dataset curation and composition

MeViS-M is built on top of the original MeViS dataset and adds manually annotated temporal moments. In the SAMDWICH description, original MeViS has three splits: train with 1,662 videos, valid$_u$ with 50 videos, and valid with 140 videos. MeViS-M adds detailed object-wise moment annotations for train and valid$_u$. For the valid split, because mask ground truth is unavailable, the annotation is only at the video level for expression-relevant frames rather than object-specific moment spans (Lee et al., 16 Aug 2025).

The annotation process was carried out manually by about 20 annotators. During this process, the authors also curated the underlying benchmark labels. They removed training samples where referred objects lacked valid masks, eliminating 66 videos and 1,278 expressions from the original training set. They further corrected missing referred objects, fixed label-ID switches, removed masks for irrelevant objects, and excluded cases in which partial segmentation errors could not be repaired because the original masks were insufficient (Lee et al., 16 Aug 2025).

After curation, the dataset used for training contains 27,292 motion-focused expressions across 1,596 training videos, 50 valid$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$0 videos, and 140 valid videos. The MeViS-M paper does not report the total number of object instances, the total number of annotated moments, category counts, average moment duration, or average expression length. This omission matters because it limits purely statistical characterization of the annotation extension (Lee et al., 16 Aug 2025).

The following summary captures the dataset relation.

Resource	Key content	Quantitative details
MeViS	Motion-expression RVOS benchmark	2,006 videos, 8,171 objects, 28,570 expressions, 443k masks
MeViS-M	MeViS plus explicit temporal moment annotation and label curation	27,292 expressions across 1,596 training videos, 50 valid$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$1, 140 valid

4. Training regime enabled by MeViS-M

The main methodological importance of MeViS-M is that it enables semantically aligned text-to-clip pairs. Standard RVOS training typically samples frames indiscriminately and supervises all visible objects in those frames, even when the expression refers only to a subset of objects and only during certain parts of the video. MeViS-M replaces that assumption with explicit knowledge of $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$2 and $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$3, so clip construction and supervision can be conditioned on temporal semantics rather than visibility alone (Lee et al., 16 Aug 2025).

In SAMDWICH, clip sampling depends directly on these annotations. The implementation samples 8 frames per video for Hiera-Base and 6 for Hiera-Large; half the frames are always sampled from $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$4, while the other half are drawn from either $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$5 or $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$6. This means training clips deliberately mix semantically aligned and semantically irrelevant frames, but the model now knows which is which (Lee et al., 16 Aug 2025).

This regime supports Moment-guided Dual-path Propagation (MDP). For frames in $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$7, the model applies language-conditioned features and text prompts; for frames in $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$8, it uses frozen SAM2 visual features to avoid what the paper calls semantic contamination. The memory-attended representations are described as

$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$9

and decoding follows

$T_V$0

The text prompt is formed from adapter-enhanced textual features as

$T_V$1

Only $T_V$2 features are stored in the memory bank, and the implementation attends to features from the 6 nearest frames in memory (Lee et al., 16 Aug 2025).

MeViS-M also enables Object-level Selective Supervision (OSS). Given a sampled frame index set $T_V$3, object $T_V$4 is discarded if

$T_V$5

Only objects satisfying $T_V$6 are retained in the filtered supervision set. The training loss is then applied to this filtered target: $T_V$7 The paper does not provide numerical values for $T_V$8 or $T_V$9 (Lee et al., 16 Aug 2025).

5. SAMDWICH and empirical effects of moment-aware supervision

The MeViS-M annotations were introduced together with SAMDWICH, a moment-aware RVOS framework. Its reported ablations directly quantify the value of moment-aware supervision on the valid$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$0 split. A baseline without moment-aware design scores 56.8 $\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$1. Training with MeViS-M moment-aware sampling raises this to 58.0, a gain of +1.2. Adding MDP yields 59.4; adding OSS yields 58.3; and combining all three reaches 60.8, which is +4.0 over the baseline (Lee et al., 16 Aug 2025).

The sampling ablation is especially revealing. Random sampling gives 56.8, BLIP-2-based sampling gives 55.5, and MeViS-M-based sampling gives 58.0. The paper therefore reports MeViS-M as +1.2 better than random sampling and +2.5 better than BLIP-2-based keyframe sampling, supporting the argument that accurate temporal semantics are more useful than indiscriminate or noisy temporal selection (Lee et al., 16 Aug 2025).

The MDP ablation separates two roles of moment annotations. Without moment-aware feature enhancement and with memory built from all frames, the score is 58.0. Restricting memory to $\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$2 raises this to 58.9. Using moment-aware feature enhancement alone with memory from all frames gives 60.3. Using both reaches 60.8. This suggests that MeViS-M improves RVOS both by telling the model where language-conditioned features should be applied and by telling it which frames should populate memory (Lee et al., 16 Aug 2025).

The paper also reports that top-1 accuracy of several VLMs for keyframe selection is only around 50% on the valid split after excluding videos where the ground-truth moment covers the entire video. This is consistent with the claim that moment localization is itself difficult and that MeViS-M addresses a genuine supervision gap rather than a trivial annotation refinement (Lee et al., 16 Aug 2025).

6. Inference bottlenecks, scope, and broader significance

MeViS-M changes the assumptions of RVOS training in a substantive way. The expression is no longer treated as uniformly applicable to every visible frame and object. Instead, supervision becomes temporally selective and object selective. This enables aligned text-to-clip training, moment-aware propagation, and selective object filtering, and it suggests that a large portion of MeViS difficulty originates from semantic misalignment during training rather than only from backbone capacity (Lee et al., 16 Aug 2025).

At the same time, the framework built on top of MeViS-M reveals a new bottleneck: moment retrieval at inference. The SAMDWICH appendix reports a separate moment retrieval model, Chrono. On valid$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$3, Chrono achieves [email protected] = 74.7, [email protected] = 61.8, mAP = 62.1, [email protected] = 72.8, [email protected] = 60.0, and downstream $\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$4. On valid, it achieves [email protected] = 61.3, [email protected] = 53.0, mAP = 52.2, [email protected] = 60.6, [email protected] = 51.2, and downstream $\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$5. Under ground-truth moments, the model reaches 60.8 on valid$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$6 and 53.5 on valid; prioritizing top BLIP-2 frames within the ground-truth moment raises this further to 62.9 on valid$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$7, 54.5 on valid with top-4, and 54.8 on valid with top-8 (Lee et al., 16 Aug 2025).

These results imply that MeViS-M is not merely a cleaner label set. It exposes a separate temporal grounding problem: identifying when the expression applies. A plausible implication is that future RVOS systems may need joint moment retrieval and segmentation rather than the conventional single-stage formulation. The paper also notes a limitation with expressions that describe multiple temporally separated actions of the same object, because the model uses a single expression-level feature across the whole video and may overemphasize one action span (Lee et al., 16 Aug 2025).

In summary, MeViS-M is best understood as a moment-annotated, semantically curated extension of MeViS that makes object-wise temporal relevance explicit. Its primary significance lies in formalizing a supervision axis that earlier MeViS challenge papers left implicit: not only which object the language refers to, but when the language refers to it.