SAMDWICH: Moment-Aware RVOS Framework

• SAMDWICH is a moment-aware RVOS framework that segments and tracks objects in video using fine-grained temporal annotations to enhance text-video alignment.
• It introduces Moment-guided Dual-path Propagation (MDP) to distinctly process text-relevant and text-irrelevant frames, improving both object grounding and tracking.
• Object-level Selective Supervision (OSS) reduces semantic noise by applying loss only to temporally aligned objects, thus boosting referential accuracy.

SAMDWICH is a moment-aware framework for Referring Video Object Segmentation (RVOS) that targets a specific failure mode in video-language learning: semantic misalignment between natural-language expressions and the video frames and objects used for supervision during training. RVOS requires a model to segment and track objects in video according to a referring expression, so correctness depends not only on visual localization and temporal tracking but also on whether the supervised frames and objects are actually relevant to the text. SAMDWICH addresses this by coupling a newly annotated dataset, MeViS-M, with moment-aware training, Moment-guided Dual-path Propagation (MDP), and Object-level Selective Supervision (OSS), yielding state-of-the-art performance on the challenging MeViS benchmark and particularly strong behavior in complex scenarios involving diverse expressions (Lee et al., 16 Aug 2025).

1. Problem setting and failure mode

RVOS aims to segment and track objects in videos based on a natural language referring expression. The central difficulty is precise video-text alignment: the model must detect, localize, and track only those object(s), and only during those time periods, that are described by the expression (Lee et al., 16 Aug 2025).

The motivating diagnosis behind SAMDWICH is that existing methods often suffer from semantic misalignment because they sample frames indiscriminately and supervise all visible objects during training, regardless of actual relevance to the expression. The reported consequences are twofold: noisy supervision from frames that are not referenced by the text, and weak learning of the temporal boundaries at which the described object or action should be considered text-relevant (Lee et al., 16 Aug 2025).

This problem formulation implies that RVOS is not merely a dense prediction task with language conditioning. It is also a temporally indexed alignment problem in which the supervision policy determines whether the model learns referential semantics or simply co-occurrence patterns among objects, frames, and expressions.

2. MeViS-M and moment annotation

MeViS-M augments the prior MeViS dataset with manual, fine-grained temporal moment annotations. For each video and each referred object, annotators specify a set of frames, or moment intervals, in which the object is relevant to the referring expression (Lee et al., 16 Aug 2025).

Formally, for each object $i$, SAMDWICH defines a moment index set

$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$

where $T_V$ is the total frame count. The union over objects gives the set of all text-relevant moments,

$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i,$

and the complement $\mathcal{M}^-$ denotes text-irrelevant frames (Lee et al., 16 Aug 2025).

The significance of this annotation scheme is explicit semantically grounded supervision at object-wise temporal granularity. The paper states that experiments show vision-LLMs perform poorly, at approximately 50% top-1 accuracy, on key-frame selection for moment retrieval, which emphasizes the need for accurate fine-grained ground-truth annotations rather than relying on automatic frame selection (Lee et al., 16 Aug 2025). The broader implication is that moment labels are not a peripheral dataset refinement; they are the supervision primitive that makes moment-aware alignment operational.

3. Architecture and representational interface

SAMDWICH is presented as a moment-aware pipeline for RVOS with three principal innovations: moment-aware training, MDP, and OSS. The framework is built on strong foundational segmentation architectures, specifically SAM2 with either Hiera-Base or Hiera-Large as backbone, and uses RoBERTa as the text encoder for the natural-language query (Lee et al., 16 Aug 2025).

Vision-language fusion is implemented through a lightweight adapter module using bidirectional cross-attention between visual and textual features. In this construction, visual features $\mathbf{F}^k$ and textual embeddings $\mathbf{E}^k$ are mutually updated through cross-attention, producing adapter-enhanced representations used for language-conditioned segmentation. A text prompt $\rho_{\mathrm{txt}}$ is formed by concatenating context ([CLS]) and motion-centric (verbs) embeddings from the expression and then processing the result by an MLP (Lee et al., 16 Aug 2025).

Architecturally, this design separates the base visual representation from the text-conditioned representation rather than forcing a single feature stream to serve both grounding and propagation. That separation becomes operationally important in MDP, where text-relevant and text-irrelevant frames are processed differently.

4. Moment-guided Dual-path Propagation

MDP is the framework’s propagation mechanism for decoupling how the model handles text-relevant and text-irrelevant frames during mask propagation. Its stated goal is to improve both object grounding and tracking by training on both relevant and irrelevant frames through a moment-centric memory mechanism (Lee et al., 16 Aug 2025).

For frames in $\mathcal{M}^+$, the model uses adapter-enhanced, text-conditioned visual features $\mathbf{F}_{\mathrm{Adp}}$, passes them to the decoder together with the text prompt, and updates the memory bank with the generated masks. For frames in $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$0, the model uses raw visual features $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$1 from the backbone, the decoder does not receive the text prompt, and the memory bank is not updated (Lee et al., 16 Aug 2025).

Frame category	Features and decoder input	Memory bank behavior
$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$2	Uses $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$3 and $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$4	Updated with generated masks
$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$5	Uses $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$6 and no text prompt	Not updated

Both paths use shared memory attention to retrieve spatio-temporal guidance from stored masks and features. The paper gives the propagation rule as

$\mathcal{M}_i \subseteq \{1,\dots,T_V\},$7

where $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$8 is the decoder and $\mathcal{M}_i \subseteq \{1,\dots,T_V\},$9 is the memory bank (Lee et al., 16 Aug 2025).

A common misunderstanding would be to equate moment-awareness with ignoring irrelevant frames. SAMDWICH does not do that. Instead, it trains on both relevant and irrelevant frames, but only relevant frames contribute text-grounded memory updates. This suggests that the method treats irrelevant frames as useful for temporal continuity while preventing them from contaminating the language-conditioned memory state.

5. Object-level Selective Supervision

OSS is introduced to address a second source of semantic noise: prior methods supervise segmentation on all objects visible in sampled frames, even when those objects are not mentioned by the referring expression or are not relevant in those frames. According to the paper, this impedes learning proper video-text alignment (Lee et al., 16 Aug 2025).

The OSS strategy applies loss only to mask predictions for objects and frames aligned with the expression’s moment annotation. Given a sampled frame set $T_V$0 and objects $T_V$1 with moments $T_V$2, the selected supervision target is written as

$T_V$3

Here, $T_V$4 denotes the ground-truth mask for object $T_V$5 at time $T_V$6 (Lee et al., 16 Aug 2025).

The functional role of OSS is therefore narrower than generic sample filtering. It is an object-level filtering rule tied to temporal alignment. By supervising only the objects temporally aligned with the expression in each training clip, OSS reduces semantic noise and reinforces language-conditioned learning (Lee et al., 16 Aug 2025).

6. Empirical significance and interpretation

The paper reports that SAMDWICH achieves state-of-the-art performance on the challenging MeViS benchmark, with particular strength in complex scenarios involving diverse expressions (Lee et al., 16 Aug 2025). Although the provided summary does not enumerate benchmark metrics, it attributes the gains to semantically grounded supervision and moment-aware propagation rather than to indiscriminate scaling alone.

Two empirical points are especially consequential. First, the moment annotations in MeViS-M expose a substantial gap between automatic key-frame selection by vision-LLMs and manual temporal grounding, with the reported approximately 50% top-1 accuracy for moment retrieval. Second, the framework’s improvement is linked not only to text-relevant frames but also to the disciplined use of text-irrelevant frames through a memory mechanism that does not admit irrelevant updates (Lee et al., 16 Aug 2025).

Taken together, these findings suggest a broader methodological lesson for RVOS: improvements in referential understanding may depend as much on the supervision topology—what objects and moments are eligible for language-conditioned learning—as on the backbone architecture. In that sense, SAMDWICH reframes alignment as a training-data and memory-management problem in addition to a multimodal representation problem.

7. Scope, related ambiguity, and nomenclature

SAMDWICH is specifically an RVOS framework for moment-aware video-text alignment and should not be conflated with similarly named systems in unrelated domains. In particular, "Sandwich: Separating Prefill-Decode Compilation for Efficient CPU LLM Serving" addresses CPU-based LLM serving through separate execution plans for prefill and decode phases, which is a distinct systems problem and not a video-language segmentation method (Zhao et al., 19 May 2025).

Within RVOS itself, another potential misconception is that temporal moment supervision simply restricts training to a subset of frames. SAMDWICH’s design is more precise: relevant and irrelevant frames are both used, but they are assigned different representational paths, different decoder conditions, and different memory-update rules, while OSS further restricts supervision to temporally aligned objects (Lee et al., 16 Aug 2025). This combination is what gives the framework its moment-aware character.

The importance of SAMDWICH therefore lies in making temporal relevance an explicit training variable. MeViS-M supplies the supervision signal, MDP operationalizes moment-conditioned propagation, and OSS removes object-level supervision noise. The resulting framework defines a semantically grounded approach to RVOS in which alignment is enforced at the level of frames, objects, and memory updates rather than treated as a by-product of generic multimodal fusion (Lee et al., 16 Aug 2025).