CLUE: Lightweight VOS with Clue Refining Network

• The paper presents CRVOS, a real-time Semi-VOS method that replaces heavy target-specification modules with a lightweight 5-channel Clue built from the previous frame's mask and coordinate information.
• It employs a ResNet-50 encoder and deconvolution-based refine modules to progressively enhance a coarse mask, achieving competitive accuracy while maintaining high speed.
• Ablation studies reveal that integrating explicit coordinate encoding with the prior mask boosts performance on DAVIS 2016 from 78.6% to 81.6% without sacrificing real-time inference.

CRVOS, short for Clue Refining Network for Video Object Segmentation, is a real-time method for semi-supervised video object segmentation (Semi-VOS) that replaces the heavy target-specification modules common in encoder–decoder pipelines with a lightweight prior called the Clue. In the Semi-VOS setting, the ground-truth object mask is given in the first frame, and the task is to predict the target mask for every subsequent frame. CRVOS is built around the observation that, in many video sequences, the previous frame’s mask already provides a sufficiently strong coarse spatial prior for the current frame, especially when motion between adjacent frames is limited. On that basis, the method dispenses with an intermediate network and instead refines a simple 5-channel clue tensor using current-frame features, yielding a system that the paper reports as the fastest among the compared methods on DAVIS 2016 while maintaining competitive accuracy (Cho et al., 2020).

1. Semi-VOS context and design rationale

CRVOS is situated in the family of Semi-VOS methods that do not use online fine-tuning. The paper distinguishes two broad families of Semi-VOS systems: methods with online fine-tuning, which adapt to the specific test video at inference time and usually achieve strong accuracy but are slow, and methods without online fine-tuning, which are faster but often less accurate (Cho et al., 2020). CRVOS belongs to the second category and is designed explicitly around runtime efficiency.

The paper positions recent no-online-tuning systems as increasingly dependent on encoder–decoder backbones augmented with expensive intermediate networks that generate object-specific guidance. Examples named in the paper include RGMP with a Siamese encoder and Global Convolution Block, STM with Space-Time Memory Networks, and RANet with a Ranking Attention Module (Cho et al., 2020). The authors’ argument is not that such modules are ineffective, but that they are disproportionately costly for many relatively simple sequences.

The central design assumption is temporal continuity. In adjacent frames, object position usually changes only a little, so the previous frame’s mask already gives a useful coarse estimate of where the target is likely to be. This makes it possible to replace memory banks, Siamese matching, ranking, or long-range retrieval with a directly constructed temporal prior. This suggests that CRVOS should be understood less as a general replacement for all target-specification machinery than as a minimalist architecture optimized for scenarios in which coarse temporal propagation is already highly informative.

2. The Clue as a lightweight target specifier

The defining concept in CRVOS is the Clue, which the paper describes as a simple but central specifier. The Clue has 5 channels at $1/16$ of the input resolution: 2 channels for the previous frame’s coarse mask and 3 channels for coordinate information (Cho et al., 2020).

The previous mask component is itself two-channel, with one channel for foreground probability and one for background probability. This matches the decoder output format. The paper uses the notation $I^i$ for the current frame input image, $M^i$ for the current frame mask, and $M^{i-1}$ for the previous frame mask. Since the encoder output is at $1/16$ resolution, the previous-frame mask is used in coarse form at the same spatial scale (Cho et al., 2020).

The coordinate component is inspired by CoordConv and consists of three channels with values in $[-1,1]$: one sorted by height, one by width, and one by distance from the center. The paper does not give an explicit formula for the center-distance channel, but it states its semantic role clearly. These channels make absolute position explicit and strengthen the mask prior by encoding where the coarse object hypothesis lies within the frame.

This representation is central to CRVOS’s efficiency. The Clue is not produced by a learned intermediate target-matching network; it is constructed directly from the previous prediction and fixed coordinate layers. A common misconception would be to treat the Clue as a reduced memory module. It is more precise to describe it as a cheap temporal propagation prior: previous-frame localization plus positional encoding, passed directly to the decoder (Cho et al., 2020).

3. Architecture and refine module

CRVOS has three named parts: Encoder, Decoder, and Clue. The encoder uses ResNet-50, pretrained on ImageNet, and extracts current-frame appearance features from $I^i$. The final encoder feature map is at $1/16$ of the input image size, and skip connections link encoder features to the decoder (Cho et al., 2020).

The decoder predicts the current mask by refining the Clue with three refine modules. This is an important architectural distinction: the decoder does not infer target identity from scratch. Instead, it starts from the coarse target-specific prior supplied by the Clue and progressively improves it using current-frame features. The decoder output is again a 2-channel foreground/background probability mask (Cho et al., 2020).

The paper’s second main contribution is the design of the refine module. Standard Semi-VOS refine modules are described as typically using bilinear interpolation for upsampling. CRVOS replaces this with a deconvolution layer, motivated by the claim that bilinear upsampling is fixed and non-trainable, whereas deconvolution can learn to generate more detailed spatial structure. The proposed refine module contains a convolution layer, a deconvolution layer, and skip connections connecting refine modules (Cho et al., 2020).

Because deconvolution is usually more expensive, CRVOS keeps the decoder lightweight by reducing the output channel size to 2. The resulting decoder therefore remains compatible with real-time inference while improving over a more general refine module that uses convolution for dimension reduction, bilinear upsampling for size increase, and no skip connections across refine stages. In the paper’s framing, the deconvolution-based refine module and the Clue are complementary contributions: neither alone explains the full performance of the system.

4. Training and inference procedure

Training is two-step. First, CRVOS is pretrained on the YouTube-VOS train set using 8 randomly chosen frames, input size $240 \times 432$, for 100 epochs with learning rate $1 \times 10^{-4}$. Second, it is fine-tuned on the DAVIS 2016 train set using 16 randomly chosen frames, original input size $I^i$0, for 500 epochs with learning rate $I^i$1 (Cho et al., 2020).

Optimization uses Adam, and the loss is negative log-likelihood (NLL). During fine-tuning, data augmentation includes horizontal flip, rotation $I^i$2, shearing $I^i$3, and scaling $I^i$4 (Cho et al., 2020).

At inference time, the procedure is recursive across frames. For frame $I^i$5, the model takes the current image $I^i$6, builds the Clue from the previous frame’s coarse mask $I^i$7 and coordinate channels, encodes the image with the ResNet-50 backbone, and decodes the current prediction $I^i$8 by refining the Clue through the three refine modules. The predicted mask is then reused to form the next frame’s Clue (Cho et al., 2020). This makes CRVOS a frame-to-frame propagation model rather than a long-range memory model.

For multiple targets, the paper states that one target is chosen at random during training if multiple targets exist, whereas during test time the masks of the entire targets are generated and then overlapped. This confirms that the architecture is fundamentally recursive and target-specific, but accommodates multi-object evaluation by independent mask generation at test time.

5. Empirical performance and ablation evidence

The principal empirical claim is the speed–accuracy operating point. On the DAVIS 2016 validation set, CRVOS achieves 63.5 fps, $I^i$9, $M^i$0, and combined $M^i$1, measured on a single GeForce RTX 2080 Ti GPU (Cho et al., 2020). The paper reports this as the fastest among the compared methods while remaining competitive in accuracy. On the DAVIS 2017 validation set, performance is 53.0 fps, $M^i$2, $M^i$3, and combined 54.3\%, showing that the method remains fast but degrades substantially in the harder multi-object regime (Cho et al., 2020).

The ablation study on DAVIS 2016 is especially informative because it isolates the role of the Clue and the refine module. Here, RM denotes the proposed refine module, PM denotes the previous frame’s coarse mask only, and Clue denotes the previous mask plus coordinate information.

Variant	Components	$M^i$4
I	RM only	72.1%
II	RM + PM	78.6%
III	RM + Clue	81.6%
IV	Clue + general refine module	79.1%

These numbers show three distinct effects (Cho et al., 2020). First, without any target specifier, performance is much lower, indicating that current-frame appearance alone is insufficient for reliable target selection. Second, adding only the previous coarse mask produces a substantial gain from 72.1% to 78.6%, confirming the value of temporal propagation. Third, adding the coordinate channels yields a further increase to 81.6%, demonstrating that explicit positional encoding materially strengthens the prior. Finally, replacing the proposed refine module with a more general one drops performance to 79.1%, which indicates that the deconvolution-based multi-stage refine design also contributes independently.

6. Operational regime, limitations, and place among Semi-VOS methods

CRVOS works best in what the paper calls relatively simple scenarios: single-target cases, limited motion between adjacent frames, and smooth positional change (Cho et al., 2020). This matches the underlying assumption of the Clue. When the target moves fast and scale rapidly changes, the paper explicitly notes that the network can produce poor-quality segmentations because the Clue then provides inaccurate information.

This limitation explains the contrast between DAVIS 2016 and DAVIS 2017 results. DAVIS 2016 is more compatible with the model’s local temporal prior, whereas DAVIS 2017 introduces harder multi-object conditions in which coarse previous-frame propagation is less reliable. A plausible implication is that CRVOS occupies a specific point in the Semi-VOS design space: it sacrifices robustness in difficult dynamics to eliminate the computational cost of heavy intermediate specification modules.

Relative to prior no-online-tuning systems, CRVOS is best understood as a minimalist alternative. Instead of memory retrieval, ranking attention, or Siamese matching, it uses a directly constructed target cue and a lightweight refinement decoder. The method is therefore not presented as a universal replacement for stronger target-specification schemes in all regimes. Its significance lies in showing that, for many practical sequences, target specification can be reduced to a previous-frame coarse mask plus explicit coordinates, and that this simplification is sufficient to enable real-time Semi-VOS with competitive accuracy (Cho et al., 2020).