Moment-guided Dual-path Propagation (MDP)

• The paper introduces MDP as a strategy that separates language-conditioned grounding on text-relevant frames from propagation on text-irrelevant frames, boosting segmentation consistency.
• It employs a moment-centric memory bank where memory writes occur only on semantically grounded frames, thereby reducing semantic misalignment during visual processing.
• Empirical evaluations on RVOS benchmarks demonstrate that combining MDP with selective supervision significantly improves performance metrics, while its design inspires analogous dual-path constructions in stochastic processes.

Searching arXiv for the cited papers to ground the article in the latest available records. Moment-guided Dual-path Propagation (MDP) is a moment-aware propagation strategy introduced within SAMDWICH for Referring Video Object Segmentation (RVOS), where it is used to separate language-conditioned grounding on text-relevant frames from language-agnostic propagation on text-irrelevant frames. The method is built around object-specific temporal moments, a moment-centric memory bank, and a dual-path execution schedule that writes memory only from semantically grounded frames. In SAMDWICH, MDP operates together with Object-level Selective Supervision (OSS) and MeViS-M moment annotations to address semantic misalignment caused by indiscriminate frame sampling and supervision (Lee et al., 16 Aug 2025). In a separate mathematical exposition, the same phrase is used to describe a dual-path construction in which moment duality guides paired forward and backward processes under propagation of exchangeability, with duality function $H(x,n)=x^n$ (Casanova et al., 26 Jun 2026).

1. RVOS setting and the role of temporal moments

Referring Video Object Segmentation aims to localize and temporally track the object(s) in a video that are described by a natural-language expression. The input is a video $V=\{I_t\}_{t=1}^{T_V}$ and an expression $E$, and the output is a segmentation mask sequence for the referred object(s). The central problem identified in SAMDWICH is semantic misalignment: prior RVOS methods commonly train with randomly sampled frames and supervise all visible objects regardless of their relevance to the expression. This forces the model to segment referred objects even in frames where the text is not semantically grounded, which dilutes language-visual alignment and harms temporal consistency and tracking.

MDP is defined relative to object-specific temporal moments. For each object $i$, MeViS-M annotates the temporal frames in which the object is actually referred to by the expression, giving an object-wise moment index set

$M_i \subset \{1,\dots,T_V\}.$

The global set of text-relevant frames is

$M^+ = \bigcup_i M_i,$

and the set of text-irrelevant frames is

$M^- = \{1,\dots,T_V\}\setminus M^+.$

This formalization changes the training target from “all visible instances in sampled frames” to “instances whose temporal presence is semantically aligned with the expression.” In the RVOS formulation of MDP, moments therefore act as a supervision prior and as a control signal for propagation. A plausible implication is that temporal grounding is treated as a first-class structural variable rather than as an emergent by-product of segmentation training.

2. Dual-path propagation and moment-centric memory

The defining mechanism of MDP is a dual-path design coupled to a moment-centric memory bank. The memory bank $B$ stores keys and values derived only from frames in $M^+$. If $t\in M^+$, the model grounds the object with language and writes to memory; if $V=\{I_t\}_{t=1}^{T_V}$0, the model reads from memory to propagate masks but does not write back.

The memory operations are specified as follows. Let $V=\{I_t\}_{t=1}^{T_V}$1 be frame features and $V=\{I_t\}_{t=1}^{T_V}$2 the soft mask prediction. For memory write on relevant frames,

$V=\{I_t\}_{t=1}^{T_V}$3

and

$V=\{I_t\}_{t=1}^{T_V}$4

For memory read,

$V=\{I_t\}_{t=1}^{T_V}$5

where $V=\{I_t\}_{t=1}^{T_V}$6 and $V=\{I_t\}_{t=1}^{T_V}$7 stack the memory entries stored in $V=\{I_t\}_{t=1}^{T_V}$8. In compact form,

$V=\{I_t\}_{t=1}^{T_V}$9

$E$0

$E$1

The two paths differ in both feature source and conditioning regime.

Path	Condition	Core operations
Relevant path	$E$2	use $E$3; read memory; decode with $E$4; write $E$5
Irrelevant path	$E$6	use $E$7; read memory; decode without text; no write

For $E$8, the relevant path uses text-conditioned features: $E$9 For $i$0, the irrelevant path uses raw visual features from frozen SAM2: $i$1 The decoder outputs a soft mask $i$2, which is binarized as

$i$3

with fixed threshold $i$4 in the SAM2 thresholding formulation in the paper.

The architectural intention is explicit: grounding and tracking are decoupled. Language is used where the expression is semantically grounded, while propagation over irrelevant frames is driven by a curated memory constructed only from grounded frames. This prevents contamination of the memory by visually present but text-irrelevant content.

3. Cross-modal alignment, sampling, and selective supervision

MDP is embedded in a larger SAMDWICH pipeline that combines frozen encoders with lightweight trainable modules. The visual backbone is SAM2 with Hiera-Base or Hiera-Large encoders, producing multi-scale visual features $i$5. The text encoder is RoBERTa, producing text embeddings $i$6. A lightweight bidirectional cross-attention adapter fuses the two modalities. At adapter layer $i$7,

$i$8

$i$9

After $M_i \subset \{1,\dots,T_V\}.$0 adapter layers, the framework uses $M_i \subset \{1,\dots,T_V\}.$1 and $M_i \subset \{1,\dots,T_V\}.$2. The text prompt is then formed by extracting a contextual embedding $M_i \subset \{1,\dots,T_V\}.$3 and a verb-centric motion embedding $M_i \subset \{1,\dots,T_V\}.$4, concatenating them, and passing them through an MLP: $M_i \subset \{1,\dots,T_V\}.$5

Training uses short clips. Hiera-Base uses $M_i \subset \{1,\dots,T_V\}.$6 frames and Hiera-Large uses $M_i \subset \{1,\dots,T_V\}.$7 frames. Half of the frames are always sampled from $M_i \subset \{1,\dots,T_V\}.$8, and the rest are mixed from $M_i \subset \{1,\dots,T_V\}.$9 or $M^+ = \bigcup_i M_i,$0. The memory update schedule follows the dual-path logic: write to memory only at $M^+ = \bigcup_i M_i,$1, but read from memory at all $M^+ = \bigcup_i M_i,$2.

Object-level Selective Supervision filters supervision targets using moment annotations. For a sampled clip with frame index set $M^+ = \bigcup_i M_i,$3, only objects whose moments intersect with the clip are supervised: $M^+ = \bigcup_i M_i,$4 The segmentation loss combines Dice and Focal terms: $M^+ = \bigcup_i M_i,$5 and the total training objective is

$M^+ = \bigcup_i M_i,$6

OSS removes masks for objects not temporally aligned with the expression in the sampled clip. In combination with MDP, relevant frames both populate memory and receive semantically grounded supervision, while irrelevant frames are decoded without text and do not contribute to memory. This pairing directly targets the semantic noise identified in the RVOS setting.

4. Forward procedure, inference regime, and implementation profile

The concise forward procedure is chronological. Memory is initialized as empty: $M^+ = \bigcup_i M_i,$7 The model first extracts $M^+ = \bigcup_i M_i,$8, encodes text with RoBERTa, and builds $M^+ = \bigcup_i M_i,$9. Then, for each time step $M^- = \{1,\dots,T_V\}\setminus M^+.$0, it executes the relevant path if $M^- = \{1,\dots,T_V\}\setminus M^+.$1 and the irrelevant path otherwise. On relevant frames,

$M^- = \{1,\dots,T_V\}\setminus M^+.$2

$M^- = \{1,\dots,T_V\}\setminus M^+.$3

$M^- = \{1,\dots,T_V\}\setminus M^+.$4

On irrelevant frames,

$M^- = \{1,\dots,T_V\}\setminus M^+.$5

$M^- = \{1,\dots,T_V\}\setminus M^+.$6

No memory write is performed on $M^- = \{1,\dots,T_V\}\setminus M^+.$7.

At test time, if ground-truth moments are unavailable, SAMDWICH uses a retrieval signal to form $M^- = \{1,\dots,T_V\}\setminus M^+.$8. The stated options are vision-LLMs for keyframe selection, including BLIP-2, CLIP, LLaMA-VID, and Chat-UniVi, or a dedicated moment retrieval model, Chrono. The best performance combines Chrono segments with BLIP-2 top-$M^- = \{1,\dots,T_V\}\setminus M^+.$9 frames prioritized inside the retrieved window. The inference procedure is: identify $B$0, run the relevant path on $B$1 with memory updates, then run the irrelevant path on $B$2 for propagation using memory reads only.

The implementation profile is deliberately lightweight relative to the frozen encoders. Only approximately $B$3M parameters, approximately $B$4 of the total, are trained; these cover the adapter, decoder, and memory modules. The encoders remain frozen. Memory attention typically attends to the $B$5 nearest memory entries. Optimization uses Adam with learning rate $B$6 and batch size $B$7. Pretraining lasts $B$8 epochs on RefCOCO/+/g, followed by $B$9 epoch on MeViS-M, using $M^+$0 NVIDIA A100 40GB.

This configuration suggests that MDP is designed less as a backbone replacement than as a propagation policy layered on top of a strong frozen segmentation foundation.

5. Datasets, evaluation, and reported empirical behavior

MeViS is the benchmark used for the main RVOS evaluation. It is described as a challenging RVOS benchmark with complex multi-object motion expressions. The reported metrics are region similarity $M^+$1, contour accuracy $M^+$2, and their average $M^+$3. MeViS-M augments MeViS with object-wise temporal moment annotations. Its reported sizes are: Train, $M^+$4 videos; Valid$M^+$5, $M^+$6; Valid, $M^+$7; and $M^+$8 expressions total. The Valid split lacks mask ground truth and has moment-only annotations. The annotation protocol corrects label issues including missing objects, ID switches, and irrelevant masks (Lee et al., 16 Aug 2025).

On the main MeViS table, under regular methods without VLM at inference, SAMWISE with Hiera-B reports $M^+$9 $t\in M^+$0, while Ours† with VLM keyframe selection reports $t\in M^+$1. Under oracle settings with ground-truth moments available at inference, SAMWISE with Hiera-B reports $t\in M^+$2, GLUS with Hiera-L reports $t\in M^+$3, Ours with Hiera-B reports $t\in M^+$4, and Ours with Hiera-L reports $t\in M^+$5. The hybrid Oracle + Ours† using VLM-selected frames within the ground-truth moment reports $t\in M^+$6 for Hiera-B and $t\in M^+$7 for Hiera-L.

The ablations on Valid$t\in M^+$8, Hiera-B, using $t\in M^+$9, isolate the contribution of moment-aware components. The baseline without moment-aware training reports $V=\{I_t\}_{t=1}^{T_V}$00. Adding MeViS-M sampling gives $V=\{I_t\}_{t=1}^{T_V}$01. Adding MDP only gives $V=\{I_t\}_{t=1}^{T_V}$02. Adding OSS only gives $V=\{I_t\}_{t=1}^{T_V}$03. The full model with MDP + OSS gives $V=\{I_t\}_{t=1}^{T_V}$04. A separate component study reports: no MFE and memory from all frames, $V=\{I_t\}_{t=1}^{T_V}$05; no MFE and memory from $V=\{I_t\}_{t=1}^{T_V}$06, $V=\{I_t\}_{t=1}^{T_V}$07; MFE and memory from all frames, $V=\{I_t\}_{t=1}^{T_V}$08; MFE and memory from $V=\{I_t\}_{t=1}^{T_V}$09, $V=\{I_t\}_{t=1}^{T_V}$10. These results show that both moment-aware feature extraction and restricting memory to $V=\{I_t\}_{t=1}^{T_V}$11 are critical for propagation quality.

Sampling strategy also affects performance. Random sampling gives $V=\{I_t\}_{t=1}^{T_V}$12, BLIP-2 keyframe sampling gives $V=\{I_t\}_{t=1}^{T_V}$13, and MeViS-M moments give $V=\{I_t\}_{t=1}^{T_V}$14. Zero-shot generalization is reported on two external datasets: on Ref-YouTube-VOS, Ours with Hiera-B gives $V=\{I_t\}_{t=1}^{T_V}$15, compared with SAMWISE at $V=\{I_t\}_{t=1}^{T_V}$16 and DsHmp at $V=\{I_t\}_{t=1}^{T_V}$17; on Ref-DAVIS, Ours with Hiera-B gives $V=\{I_t\}_{t=1}^{T_V}$18, compared with SAMWISE at $V=\{I_t\}_{t=1}^{T_V}$19 and DsHmp at $V=\{I_t\}_{t=1}^{T_V}$20.

The reported limitations are specific. Without ground-truth moments, performance depends on the quality of $V=\{I_t\}_{t=1}^{T_V}$21 prediction; VLMs are sub-optimal, and Chrono improves results but remains below ground truth. Multi-action temporal compositionality is challenging because a single expression embedding across the entire video can bias the model toward the visually dominant action span. Ambiguous expressions, rapid motion and occlusions, and heavy interactions among multiple similar objects can still confuse propagation despite moment-centric memory.

6. Mathematical reinterpretation: moment duality and propagation of exchangeability

In a separate line of work, the phrase “Moment-guided Dual-path Propagation” is used to describe a construction in which moment duality organizes a forward/backward pair of stochastic processes under propagation of exchangeability (Casanova et al., 26 Jun 2026). The setting is an exchangeable $V=\{I_t\}_{t=1}^{T_V}$22-valued sequence $V=\{I_t\}_{t=1}^{T_V}$23, with exchangeability defined by invariance under finite permutations. By de Finetti’s theorem, there exists $V=\{I_t\}_{t=1}^{T_V}$24 such that, conditionally on $V=\{I_t\}_{t=1}^{T_V}$25, the coordinates are i.i.d. Bernoulli$V=\{I_t\}_{t=1}^{T_V}$26, and

$V=\{I_t\}_{t=1}^{T_V}$27

The random mapping $V=\{I_t\}_{t=1}^{T_V}$28 acts on sequences by

$V=\{I_t\}_{t=1}^{T_V}$29

with $V=\{I_t\}_{t=1}^{T_V}$30 taking values in non-empty subsets of $V=\{I_t\}_{t=1}^{T_V}$31. The paper defines propagation of exchangeability as the property that whenever $V=\{I_t\}_{t=1}^{T_V}$32 is exchangeable and independent of $V=\{I_t\}_{t=1}^{T_V}$33, the image sequence $V=\{I_t\}_{t=1}^{T_V}$34 is also exchangeable. The key necessary and sufficient condition is label-forgetfulness: $V=\{I_t\}_{t=1}^{T_V}$35

Under label-forgetfulness, iterating independent copies of $V=\{I_t\}_{t=1}^{T_V}$36 produces two paths. The forward path is the de Finetti process $V=\{I_t\}_{t=1}^{T_V}$37, whose moments satisfy

$V=\{I_t\}_{t=1}^{T_V}$38

The backward path is a block-counting chain $V=\{I_t\}_{t=1}^{T_V}$39 on $V=\{I_t\}_{t=1}^{T_V}$40 defined by

$V=\{I_t\}_{t=1}^{T_V}$41

The central identity is the moment duality

$V=\{I_t\}_{t=1}^{T_V}$42

and in continuous time,

$V=\{I_t\}_{t=1}^{T_V}$43

with duality function $V=\{I_t\}_{t=1}^{T_V}$44.

The same exposition develops continuous-time limits involving Wright–Fisher diffusion, frequency-dependent selection drift, and $V=\{I_t\}_{t=1}^{T_V}$45-jump components in the forward generator, paired with branching, pairwise coalescence, and multiple-merger terms in the backward generator. It also constructs lookdown models for $V=\{I_t\}_{t=1}^{T_V}$46-Fleming–Viot processes with frequency-dependent selection and extends the framework to mixed selection in random environments.

This mathematical usage is conceptually parallel to the RVOS usage but not operationally the same. In the RVOS system, “moment-guided” refers to text-relevant temporal spans and their effect on supervision and memory writes. In the stochastic-process setting, “moment-guided” refers to the identity between forward moments and backward generating functions. The common structural motif is a split into two coordinated paths governed by moment information, but the underlying objects—segmentation propagation in videos versus dual stochastic evolution under exchangeability—are distinct.