Selective Mask Propagation Overview

• Selective mask propagation is a family of techniques that uses explicit selection constraints to transfer mask signals across frames, support-query pairs, and domains via mechanisms like TopK filtering, local matching, and uncertainty control.
• Methods employ affinity-based propagation, motion-aware matching, and selective activation (e.g., uncertainty-triggered dispatch) to optimize computational resources while preserving temporal continuity and segmentation accuracy.
• These techniques demonstrate practical improvements in video object segmentation, medical imaging, beamforming, and recommendation retrieval by balancing selectivity with performance across diverse applications.

Selective mask propagation denotes a family of mechanisms in which a mask, a mask-conditioned representation, or a mask-derived control signal is transferred across frames, clips, support–query pairs, temporal windows, or task stages under explicit selection constraints rather than by dense indiscriminate propagation. In the video literature, selectivity is typically realized through TopK affinity filtering, curated long-/short-term memory, optical-flow-guided local matching, attention, re-identification, score thresholds, or uncertainty-triggered dispatch to a heavier propagation module. In other domains, the same logic appears as registration-based support-mask transfer in medical imaging, keyword-conditioned mask estimation for beamforming, and binary parameter masks that partition a shared retrieval network into objective-specific subspaces (Miao et al., 2021, Cheng et al., 2021, Holmberg, 11 Jun 2026, Xu et al., 2024, Kida et al., 2018, Deng et al., 17 Apr 2025).

1. Core formalism and selection mechanisms

A recurring formulation is an affinity-based propagation operator in which a query location reads from only a restricted subset of memory locations. In MiVOS, after flattening memory and query keys, the raw affinity matrix is defined by $F_{i,j}=\langle k^M_i,k^Q_j\rangle$, and for each query position $j$ only the indices $\mathrm{Top}^k_j(F)$ are retained. The resulting attention weights are

$$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$

followed by the memory read $m_j=\sum_i W_{i,j}\,v^M_i$ (Cheng et al., 2021). In this formulation, propagation is selective because correspondence is explicitly sparsified before normalization.

A second pattern is motion-aware local matching. MAMP computes optical-flow offsets, warps reference features and masks, restricts each query location to a local retrieval window $R(i)$, and then performs TopK selection over the resulting local correlation scores. With training radius $r=6$, test radius $r=12$, and $K=36$, the method keeps only the highest-scoring correspondences rather than all candidates in the local window (Miao et al., 2021). This suggests that selectivity can be imposed after motion compensation rather than before it.

A third pattern uses uncertainty to decide whether propagation should occur at all. In SAM-Deep-EIoU, the dispatch signal is the Hungarian assignment margin

$m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$

and a window is opened when $j$0 (Holmberg, 11 Jun 2026). Here, selectivity is not only about which pixels or memory entries are read, but also about when an expensive propagation subsystem is activated.

A fourth pattern transfers the masking concept from image space to parameter space. In CSMF, each objective $j$1 receives a binary mask $j$2 over the full parameter set $j$3, and the objective-specific parameters are obtained as $j$4 (Deng et al., 17 Apr 2025). In this case, propagation occurs through sequential fine-tuning over disjoint parameter subsets rather than through temporal transport of segmentation labels.

2. Video object segmentation and interactive propagation

Early high-capacity formulations coupled temporal propagation with explicit identity recovery. DyeNet combines a Re-ID module with an attention-aware recurrent mask propagation module. At each step, the previous hidden state and mask are warped by FlowNet2, the current RoI is extracted by RoIAlign, a conv-based recurrent unit updates the hidden state, and a spatial attention map derived from the warped hidden state suppresses distractors before a three-layer output network decodes the next mask (Li et al., 2018). Quantitatively, mask propagation alone obtains $j$5 on DAVIS-17 val; adding the recurrent unit without attention raises this to $j$6; full Re-MP with attention reaches $j$7; and adding Re-ID plus template expansion yields $j$8 on val and $j$9 on test-dev (Li et al., 2018).

MAMP reformulates selective propagation in a self-supervised regime. It trains a modified ResNet-18 encoder by reconstructing dropped CIELab channels from neighboring frames, then performs inference with a fixed-size two-tier memory bank that always keeps the first frame $\mathrm{Top}^k_j(F)$0 and a mid-video frame $\mathrm{Top}^k_j(F)$1 as long-term anchors while retaining $\mathrm{Top}^k_j(F)$2 as short-term memory (Miao et al., 2021). Mask transfer is motion-aware: optical flow from RAFT is used to warp reference features and masks, and only the TopK local correspondences are retained. On DAVIS-2017, “ALL” yields $\mathrm{Top}^k_j(F)$3 mean $\mathrm{Top}^k_j(F)$4, “Top-36” yields $\mathrm{Top}^k_j(F)$5, “Top-9” yields $\mathrm{Top}^k_j(F)$6, and “Top-1” yields $\mathrm{Top}^k_j(F)$7 (Miao et al., 2021). The same paper reports $\mathrm{Top}^k_j(F)$8 mean $\mathrm{Top}^k_j(F)$9 on DAVIS and $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$0 on YouTube-VOS, together with improvements of $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$1 mean $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$2 on DAVIS-2017 and $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$3 mean $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$4 on unseen YouTube-VOS categories over the nearest competitor (Miao et al., 2021).

MiVOS decouples interaction-to-mask from mask propagation and applies selectivity inside the Space–Time Memory read. Its propagation module stores key and value features for memory frames and retains only the top-$$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$5 memory locations for each query pixel before softmax normalization (Cheng et al., 2021). This design is coupled to a difference-aware fusion module that aligns positive and negative mask corrections from the latest interaction frame to intermediate frames and predicts the fused mask from the concatenation of the image, old mask, new propagated mask, aligned differences, and linear blending weights. On DAVIS-17 val in the semi-supervised setting, the baseline STM score is $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$6, MiVOS without top-$$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$7 gives $$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$8, and MiVOS with top-$$W_{i,j}= \begin{cases} \frac{\exp(F_{i,j})}{\sum_{p\in \mathrm{Top}^k_j(F)}\exp(F_{p,j})}, & i\in \mathrm{Top}^k_j(F) \ 0, & \text{otherwise} \end{cases}$$9 ($m_j=\sum_i W_{i,j}\,v^M_i$0) gives $m_j=\sum_i W_{i,j}\,v^M_i$1; memory read time decreases from $m_j=\sum_i W_{i,j}\,v^M_i$2 ms/frame without top-$m_j=\sum_i W_{i,j}\,v^M_i$3 to $m_j=\sum_i W_{i,j}\,v^M_i$4 ms/frame with top-$m_j=\sum_i W_{i,j}\,v^M_i$5 (Cheng et al., 2021). In the interactive track, the full system with difference-aware fusion reaches $m_j=\sum_i W_{i,j}\,v^M_i$6-$m_j=\sum_i W_{i,j}\,v^M_i$7 (Cheng et al., 2021).

Across these systems, selectivity is not equivalent to a single architectural primitive. It may be imposed by recurrent attention, motion-aware TopK filtering, or memory-bank restriction, but the common function is to suppress unreliable propagation paths while preserving useful temporal continuity.

3. Diffusion and generative formulations

Diffusion-based work recasts propagation as an operation on learned attention kernels. DRIFT interprets self-attention maps from a pretrained text-to-image diffusion U-Net as semantic label-propagation kernels and extends them across frames by matching the queries of frame $m_j=\sum_i W_{i,j}\,v^M_i$8 with the keys of frame $m_j=\sum_i W_{i,j}\,v^M_i$9, giving

$R(i)$0

Aggregated temporal kernels are spatially masked within a radius $R(i)$1, sparsified by keeping only the top-$R(i)$2 affinities per row, and then used for one-step or multi-frame mask propagation (Kim et al., 25 Nov 2025). The method further applies DDIM inversion, mask-specific textual inversion, adaptive head weighting, and SAM-guided refinement by sampling point prompts from the propagated soft mask and selecting the SAM mask with maximum soft-IoU (Kim et al., 25 Nov 2025). Reported results include $R(i)$3 on DAVIS-2017 without SAM and $R(i)$4 with SAM, as well as $R(i)$5 on YouTube-VOS 2018 without SAM and $R(i)$6 with SAM (Kim et al., 25 Nov 2025).

GenProp uses a different generative formulation. A frozen image-to-video model $R(i)$7 is paired with a Selective Content Encoder $R(i)$8 that encodes the unedited portions of the original video while ideally zeroing out the edited regions, and a Mask Prediction Decoder attached to the tail of $R(i)$9 predicts a mask sequence (Liu et al., 2024). Training uses a region-aware objective

$r=6$0

with $r=6$1, $r=6$2, and $r=6$3 (Liu et al., 2024). Synthetic training data are generated by balanced copy-paste, mask-fill, and color-fill operations, and the model can optionally use the predicted masks at inference to gate encoder features or blend original and generated pixels (Liu et al., 2024). The paper reports video-editing performance up to $r=6$4 dB $r=6$5, $r=6$6-$r=6$7, and $r=6$8-$r=6$9; object-removal $r=12$0-$r=12$1; and an ablation in which removing MPD reduces $r=12$2-$r=12$3 from $r=12$4 to $r=12$5 and $r=12$6-$r=12$7 from $r=12$8 to $r=12$9 (Liu et al., 2024). The same paper states that GenProp masks can track objects together with their reflections and shadows (Liu et al., 2024).

These diffusion and generative systems broaden the meaning of propagation. Instead of treating masks only as outputs to be transported, they use masks to constrain latent denoising, content injection, or attention interpretation. A plausible implication is that selective mask propagation can operate as a latent conditioning mechanism as much as a pixelwise label-transfer procedure.

4. Tracking, instance-level segmentation, and semantic segmentation

MaskProp extends Mask R-CNN to video by adding a mask propagation branch that transports center-frame instance masks to the surrounding clip. For each detected instance $K=36$0 in frame $K=36$1, the method constructs an instance-masked feature $K=36$2, predicts deformable-convolution offsets from frame-pair feature differences, warps the instance feature to frame $K=36$3, fuses the warped feature with the native feature $K=36$4, and predicts propagated masks through an instance-softmax modulated by an instance-agnostic attention map $K=36$5 (Bertasius et al., 2019). Inference is selective in that only center-frame instances with score $K=36$6 after NMS are propagated (Bertasius et al., 2019). On YouTube-VIS val, the reported results are $K=36$7 mAP and $K=36$8 AP@75 with ImageNet+COCO pretraining, and $K=36$9 mAP with additional OpenImages pretraining (Bertasius et al., 2019). Replacing the deformable-convolution warp with FlowNet2 reduces mAP to $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$0, and replacing it with MaskTrack’s tracking head gives $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$1 (Bertasius et al., 2019).

SAM-Deep-EIoU places selectivity at the system level. A lightweight base tracker runs on all frames, while a VOS model is called only on temporal windows triggered by low assignment margins, reappearance gaps, or witness overlaps (Holmberg, 11 Jun 2026). Within each window, masks are matched back to tracker boxes by intersection-over-mask-area,

$m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$2

and the window exits only after $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$3 consecutive frames satisfy containment, margin, box isolation, and mask isolation criteria (Holmberg, 11 Jun 2026). Only the outcome SWAP changes tracker identities; CLEAN, STALE, DEGRADED, EDGE, and END preserve the base output (Holmberg, 11 Jun 2026). On DanceTrack, SORT improves from $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$4 HOTA to $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$5, ByteTrack from $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$6 to $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$7, and Deep-EIoU from $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$8 to $m_{j^*}=\min_{i\neq i^*} C_{i,j^*}-C_{i^*,j^*},$9 (Holmberg, 11 Jun 2026). On SportsMOT, SAM 3-Deep-EIoU with global track association reaches $j$00 HOTA, $j$01 AssA, and $j$02 IDF1 (Holmberg, 11 Jun 2026).

MPVSS transfers the same principle to video semantic segmentation. A heavy query-based segmentor is run only on sparse key frames, producing $j$03 binary masks and class scores. For non-key frames, a motion encoder and query-conditioned flow decoder predict segment-aware flow fields $j$04, and each key-frame mask is warped to the target frame by bilinear warping (Weng et al., 2023). Final labels are formed by reusing the key-frame class scores. On VSPW with Swin-L, Mask2Former gives $j$05 mIoU at $j$06 GFLOPs, while MPVSS gives $j$07 mIoU at $j$08 GFLOPs; on Cityscapes with Swin-L, the corresponding numbers are $j$09 mIoU at $j$10 GFLOPs and $j$11 mIoU at $j$12 GFLOPs (Weng et al., 2023). The paper further reports that non-key frames cost only approximately $j$13 GFLOPs, and that varying the key-frame interval from $j$14 to $j$15 yields the expected accuracy-efficiency trade-off (Weng et al., 2023).

Taken together, these systems show three distinct selective regimes: instance-score selection, uncertainty-triggered window selection, and sparse key-frame selection. All three restrict propagation to cases in which its marginal utility is expected to exceed its cost.

5. Generalizations beyond standard video segmentation

In SAM-MPA, selective mask propagation is used for few-shot medical image segmentation. A frozen SAM image encoder embeds all images, representative support images are chosen by one-shot cosine-similarity initialization followed by $j$16-Center-Greedy seeding and Lloyd’s $j$17-means refinement, and each support image is registered to query images by unsupervised B-spline elastic registration (Xu et al., 2024). The support mask is then warped to each query as a coarse mask, from which automatic point, box, and mask prompts are generated and fed to frozen SAM; a second SAM pass serves as post-refinement (Xu et al., 2024). On Breast US and Chest X-ray, the method reports Dice scores of $j$18 and $j$19, respectively, and in the 10-shot Breast US ablation the progression is $j$20 for mask propagation only, $j$21 for mask propagation plus prompt auto-generation, $j$22 after example selection, and $j$23 after post-refinement (Xu et al., 2024).

In "Speaker Selective Beamformer with Keyword Mask Estimation," selectivity operates in the time-frequency domain. A DNN estimates a keyword mask $j$24 and a non-keyword mask $j$25 on the detected wake-word segment, and those masks are used to estimate covariance matrices for MVDR beamforming (Kida et al., 2018). The resulting time-independent beamformer is then propagated to the subsequent command utterance:

$j$26

The paper reports SDR improvements on the keyword region of $j$27 dB for the keyword mask and $j$28 dB for the non-keyword mask, and character error rate improvements from $j$29 to $j$30 on the simulated set, with consistent reductions of $j$31-$j$32 on real recordings (Kida et al., 2018). Although this is not a spatial segmentation task, it preserves the same structure: a selectively estimated mask on a short anchor segment governs subsequent propagation.

CSMF moves the idea into recommendation retrieval. After pre-training on exposure, the model prunes parameters to obtain an exposure mask $j$33, recovers accuracy on a subset of exposure data, freezes that sub-network, and fine-tunes only the freed parameters for click; the procedure is then repeated for conversion (Deng et al., 17 Apr 2025). Final retrieval uses three objective scores,

$j$34

with weights adjustable at serving time (Deng et al., 17 Apr 2025). Reported results include Industrial offline gains of $j$35 Recall@50 and $j$36 nDCG@50 on click, $j$37 Recall@50 and $j$38 nDCG@50 on conversion, online A/B gains of $j$39 RPM, $j$40 CTR, and $j$41 CVR, and serving cost of no increase in embedding size (approximately $j$42 GB) with only $j$43 ANN latency versus $j$44 for MVKE (Deng et al., 17 Apr 2025).

These cross-domain examples show that the “mask” in selective mask propagation need not be a foreground bitmap. It may instead be a deformation-guided coarse prior, a spectral selector used to estimate a spatial filter, or a binary parameter-allocation operator over a shared backbone.

6. Recurring trade-offs, misconceptions, and open problems

One common misconception is that stronger selectivity always means more aggressive sparsification. The ablations do not support that simplification. In MAMP, Top-36 outperforms both using all correspondences and using only Top-1, indicating that the optimal regime is selective but not maximally sparse (Miao et al., 2021). In MiVOS, top-$j$45 filtering improves both DAVIS accuracy and memory-read latency, but the gain is tied to a specific memory-read design rather than to sparsity in the abstract (Cheng et al., 2021).

A second misconception is that propagation systems should always override upstream outputs when they disagree. SAM-Deep-EIoU adopts the opposite principle: only SWAP changes base-tracker labels, while inconclusive outcomes preserve the original output (Holmberg, 11 Jun 2026). This suggests a conservative interpretation of selectivity in which propagation is an exception handler rather than the primary inference path.

The literature also identifies several persistent failure modes. DyeNet notes that small objects and heavy occlusions remain most challenging, even though recurrent attention helps small-object recall and Re-ID with template expansion recovers heavily occluded objects when they reappear (Li et al., 2018). SAM-MPA identifies computational cost from repeated registrations, domain gap arising from SAM pre-training on natural images, and the limitation that the method is currently $j$46D rather than volumetric (Xu et al., 2024). The beamforming work explicitly points to the remaining gap to the IBM oracle and proposes better mask estimators, more sophisticated beamforming, online filter adaptation beyond the keyword window, and joint mask-beamformer training as future work (Kida et al., 2018).

Across the surveyed papers, the main research tension is between selectivity as a robustness device and selectivity as a computational budgeter. In some systems it chiefly suppresses distractors and noisy correspondences; in others it determines when a heavy model is invoked or which parameters remain trainable. The broader pattern suggests that selective mask propagation is best understood not as a single algorithmic family, but as a design principle for controlling information flow under temporal, spatial, or task-structural uncertainty.