Object-Level Selective Supervision (OSS)

• OSS is defined in SAMDWICH as an object-level filtering technique that retains supervision only for objects with overlapping temporal moments, thus reducing semantic noise.
• It leverages per-object moment annotations from MeViS-M to ensure that only text-relevant objects receive supervision, thereby improving the alignment between video frames and referring expressions.
• Integrated into the training pipeline before loss computation, OSS has demonstrated modest yet consistent improvements in segmentation quality in ablation studies.

Searching arXiv for the cited SAMDWICH paper and a few closely related references to ground the article. Object-level Selective Supervision (OSS) is a training-time mechanism introduced in the moment-aware referring video object segmentation framework SAMDWICH to make supervision respect the temporal semantics of a referring expression. In this formulation, a model is not supervised on all visible annotated objects in a sampled training clip. Instead, it retains supervision only for objects whose object-specific referred temporal moments overlap the sampled clip, using per-object moment annotations from MeViS-M. The stated purpose is to reduce semantic noise and reinforce language-conditioned learning in Referring Video Object Segmentation (RVOS) (Lee et al., 16 Aug 2025).

1. Definition and scope

OSS is defined in SAMDWICH as an object-level filtering strategy for RVOS training. Its scope is narrow: when a sampled clip contains several annotated objects, the method does not automatically supervise all of them. It supervises only those objects whose referred temporal moments overlap the sampled frame index set for that clip. In the paper’s wording, “each object $i$ is associated with a distinct text-relevant moment $\mathcal{M}_i$, which specifies the temporal span during which the object is referred to by the expression $E$” (Lee et al., 16 Aug 2025).

This distinguishes OSS from conventional RVOS supervision, where models are typically trained on randomly sampled clips or frames and “all visible annotated objects in those sampled frames” are supervised. SAMDWICH identifies that practice as a source of semantic misalignment, because a referring expression may apply only during specific temporal spans and may involve only a subset of the visible objects. OSS addresses that issue at the object level rather than only at the clip level.

The mechanism is specific to training. The paper states that OSS is part of SAMDWICH’s training-time machinery and gives no indication that it is used during inference. A plausible implication is that OSS should be understood not as an inference-time proposal filter, but as a supervision-target filter operating before loss computation.

2. Motivation in moment-aware RVOS

The motivation for OSS comes from the observation that object relevance is time-varying in MeViS-style data. Expressions often describe actions, states, or relations that hold only during specific temporal spans. Multiple objects in the same video may each have distinct referred moments, and the same object may be relevant only during certain moments. If supervision is imposed indiscriminately on all visible objects, the training target no longer faithfully matches the semantics of the text (Lee et al., 16 Aug 2025).

SAMDWICH’s broader moment-aware perspective therefore treats temporal grounding as fundamental to video-text alignment. The paper first partitions the video into text-relevant and irrelevant moments,

$$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$

but OSS goes further by using the per-object sets $\mathcal{M}_i$. This means that even within a clip that is broadly text-relevant, not every visible object is assumed relevant to the expression at that time.

The enabling resource is MeViS-M, which “manually annotate[s] text-relevant temporal spans on a per-object basis.” For train and valid_u, these annotations provide the object-wise moments needed by OSS. The paper also notes that MeViS-M curation corrected annotation issues, including missing referred objects, irrelevant masks, and ID-switch cases. This matters because object-level filtering depends on reliable object identities and masks (Lee et al., 16 Aug 2025).

3. Formalization and selection rule

The paper defines the full video as

$V = \{I_t\}_{t=1}^{T_V},$

with referring expression $E$, object index $i$, and per-object referred moment set

$\mathcal{M}_i \subseteq \{1, \dots, T_V\}.$

For a sampled training clip, the frame indices are denoted by $\mathcal{T}$, with length $\mathcal{M}_i$0. Ground-truth masks are $\mathcal{M}_i$1, and model predictions over the sampled interval are $\mathcal{M}_i$2 (Lee et al., 16 Aug 2025).

The core OSS rule is stated explicitly:

“Given a sampled frame index set $\mathcal{M}_i$3 of length $\mathcal{M}_i$4, we discard the GT masks of any object $\mathcal{M}_i$5 whose annotated moment $\mathcal{M}_i$6 does not overlap with $\mathcal{M}_i$7, i.e., when $\mathcal{M}_i$8. We retain only those masks $\mathcal{M}_i$9 for which the condition $E$0 holds.” (Lee et al., 16 Aug 2025)

This yields a binary keep-or-discard decision per object. If there is no overlap, the object is excluded from supervision for that clip even if it is visible in some sampled frame. If there is overlap, its masks are retained in the supervision target. The paper defines the filtered target as

$E$1

The typesetting is noted as corrupted in the provided description, but the intended meaning is that $E$2 contains only masks for objects whose moment sets overlap the sampled clip.

An interpretive reconstruction given in the source material is the clip-level object-alignment indicator

$E$3

This equation is not explicit in the paper, but it matches the stated OSS mechanism exactly. This suggests that OSS is best understood as exact object-clip temporal gating rather than soft reweighting.

4. Place in the SAMDWICH training pipeline

Within SAMDWICH, OSS is applied after clip sampling and after annotations have been associated with the sampled clip, but before loss computation. The required inputs are the expression $E$4, the sampled frame index set $E$5, object identities $E$6, per-frame object masks $E$7, and per-object moment annotations $E$8. The paper states that OSS “does not alter the visual backbone, text encoder, or memory attention itself. It filters supervision targets” (Lee et al., 16 Aug 2025).

This establishes a hierarchy inside SAMDWICH. Moment-aware sampling and Moment-guided Dual-path Propagation (MDP) operate mainly at the temporal or frame-pathway level, whereas OSS operates at the object-supervision level. MDP uses text-conditioned features $E$9 on $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$0, raw SAM2 features $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$1 on $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$2, and stores memory entries from $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$3 only. OSS complements that design by deciding which object masks contribute supervisory signal inside a sampled clip (Lee et al., 16 Aug 2025).

Several implementation details are given. Clips are sampled with $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$4 frames for Hiera-Base and $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$5 for Hiera-Large. Half of the frames are always sampled from $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$6, while the other half are drawn from either $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$7 or $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$8. Multiple aligned objects are allowed: all objects satisfying $$\mathcal{M}^+ = \bigcup_i \mathcal{M}_i, \qquad \mathcal{M}^- = \{1, \dots, T_V\} \setminus \mathcal{M}^+,$$9 are kept. The rule is binary and exact; the paper does not specify a separate threshold or soft overlap score.

Two caveats in the provided material are explicitly marked as inference rather than direct paper statements. First, because half the frames are sampled from $\mathcal{M}_i$0, one would expect at least one aligned object in normal training. Second, ignored objects are best interpreted as excluded from loss rather than treated as negatives, because the paper says their masks are “discarded” and only $\mathcal{M}_i$1 is used to supervise predictions (Lee et al., 16 Aug 2025).

5. Loss formulation and learning effect

OSS does not introduce a new standalone auxiliary objective. Its mathematical effect is to change the supervision target before standard segmentation losses are computed. The loss is given as

$\mathcal{M}_i$2

The provided material explains the intended meaning despite corrupted typesetting: $\mathcal{M}_i$3 is Dice loss, $\mathcal{M}_i$4 is Focal loss, $\mathcal{M}_i$5 and $\mathcal{M}_i$6 are scalar weights, and the target passed to the loss is the OSS-filtered mask set rather than the full set of visible-object masks (Lee et al., 16 Aug 2025).

The significance of this formulation is that OSS does not continuously reweight objects by relevance. Instead, it removes semantically mismatched masks from the target set. The paper frames this as reducing “supervision noise” and reinforcing language-conditioned learning. Better grounding and video-text alignment are the primary intended effects, with improved segmentation quality arising indirectly.

A plausible implication is that OSS is clip-level selective at the object level, not necessarily frame-by-frame selective inside the clip. The provided material notes that an object may still be retained for all sampled frames if its moment overlaps the clip somewhere, even if it is not semantically active in every frame of that clip. This is identified as an inferred caveat rather than an explicit claim in the paper.

6. Empirical behavior and limitations

The paper validates OSS in ablation studies on MeViS. In Table 2, the baseline without moment-aware design scores $\mathcal{M}_i$7 $\mathcal{M}_i$8. Adding MeViS-M moment-aware sampling alone gives $\mathcal{M}_i$9. Adding OSS alone on top of MeViS-M gives $V = \{I_t\}_{t=1}^{T_V},$0, which is a $V = \{I_t\}_{t=1}^{T_V},$1 gain over moment-aware sampling alone and $V = \{I_t\}_{t=1}^{T_V},$2 over the baseline. Adding MDP alone gives $V = \{I_t\}_{t=1}^{T_V},$3. Combining MeViS-M, MDP, and OSS yields $V = \{I_t\}_{t=1}^{T_V},$4, the best result (Lee et al., 16 Aug 2025).

These numbers support the paper’s interpretation that OSS contributes complementary gains rather than replacing moment-aware sampling or MDP. Its standalone gain is modest but consistent. The appendix also credits OSS, together with MDP, for producing more focused feature maps and better exclusion of distractor objects.

The paper does not provide an OSS-specific failure analysis separated from the broader framework, but several caveats are clear in the provided material. Explicitly, SAMDWICH’s moment-aware training depends on temporally relevant segments, and inference quality can decline without accurate moment retrieval. Since OSS requires object-wise moments during training, it is tied to datasets such as MeViS-M that provide $V = \{I_t\}_{t=1}^{T_V},$5 labels. The material also notes possible sensitivity to noisy or incomplete object-specific moment annotations, and states that in multi-action expressions involving the same object across disjoint moments, the broader system can struggle because it uses a single expression-level feature over the whole video. OSS alone does not resolve that temporal compositional ambiguity (Lee et al., 16 Aug 2025).

7. Terminological context and relation to other supervision schemes

The term “object-level supervision” appears in several neighboring literatures, but those uses are not equivalent to OSS as defined in SAMDWICH. In saliency detection, CLASS introduces “cross-level supervision (CLS)” with pixel-level, region-level, and object-level terms, where the object-level component is a global differentiable $V = \{I_t\}_{t=1}^{T_V},$6-based loss on the full saliency map. The provided analysis explicitly states that this is not selective supervision in the stronger sense, but rather global object-level supervision applied jointly with lower-level losses (Tang et al., 2020).

In instance segmentation, SPOCO studies a positive-unlabeled regime in which only a subset of object instances is annotated and unlabeled regions are not treated as background. This is a direct object-level selective supervision setting, but it differs from OSS in SAMDWICH because the selection criterion is annotation sparsity over instances rather than temporal alignment of referred moments (Wolny et al., 2021).

Related selective mechanisms also appear in detection. “Learning Object Scale With Click Supervision for Object Detection” uses one center click per object instance and proposal selection to generate pseudo ground truth (Zhang et al., 2020). “Salvage of Supervision in Weakly Supervised Object Detection” uses proposal selection, pseudo-box filtering, and reliability-based splitting of pseudo-labeled data (Sui et al., 2021). Open-set semi-supervised detection papers filter pseudo-labels at the object level based on in-distribution versus out-of-distribution criteria rather than temporal relevance (Liu et al., 2022, Wang et al., 2023, Liu et al., 2023).

This terminological spread suggests that “object-level selective supervision” is best treated as a family resemblance term rather than a single uniform method name. In SAMDWICH, OSS refers specifically to temporal object-clip overlap filtering in RVOS. Its defining criterion is neither confidence, nor objectness, nor annotation sparsity, but whether $V = \{I_t\}_{t=1}^{T_V},$7 (Lee et al., 16 Aug 2025).