HQ-SMem: Enhancing Video Segmentation & Tracking

• HQ-SMem is a memory-based enhancement that refines segmentation masks, mitigates tracking drift, and optimizes memory usage in video object segmentation systems.
• It integrates SAM-HQ prompting with DINO appearance-based candidate selection to validate mask quality and prevent prompt ambiguity.
• The method employs an online self-supervised feedback loop and dynamic smart memory replacement to improve accuracy and stabilize tracking in long videos.

Searching arXiv for the primary HQ-SMem paper and closely related VOS baselines to ground the article in current papers. HQ-SMem, short for High-Quality video segmentation and tracking using Smart Memory, is a plug-and-play enhancement for memory-based video object segmentation and tracking systems such as DeAOT and DMAOT. It is designed to address three persistent weaknesses in modern VOS systems: coarse masks that leak background or blur object boundaries, tracking drift caused by propagating imperfect predictions over time, and memory bloat / inefficiency in long videos where storing every frame is wasteful and can dilute salient appearance information. Its central design combines SAM-HQ with appearance-based candidate selection, a dynamic smart memory mechanism, and an online self-supervised distillation feedback loop, with the stated aim of making strong VOS trackers both more accurate and more efficient (Kazemi et al., 25 Jul 2025).

1. Position within memory-based video object segmentation

HQ-SMem is built around a memory-based encoder-decoder VOS model. The paper writes the base formulation as

$$y_t = \mathcal{D}\left(\mathcal{E}(x_t); \mathbf{M}; q\right), \qquad \mathbf{M} = \{\mathcal{E}(x_f)\}_{f=0}^{t-1},$$

where $x_t$ is the current frame, $\mathcal{E}$ is the encoder, $\mathcal{D}$ is the decoder, $\mathbf{M}$ is the memory bank of previous embeddings, and $q$ is the prompt or mask input, often the previous prediction $y_{t-1}$. In this formulation, standard VOS has two recurring difficulties: memory typically grows with time, $\mathcal{O}(T)$, and the masks passed forward are often too coarse (Kazemi et al., 25 Jul 2025).

The method is not introduced as a new standalone tracker. Its function is to enhance a base tracker by modifying the mask-refinement path, the feedback signal used in autoregressive propagation, and the policy that determines which frames remain in memory. The paper positions this intervention against specific failure modes: coarse segmentations that mix in background pixels, fixed memory update schedules, brittleness to drift and occlusions, and prompt ambiguity issues associated with SAM.

A common misconception is to treat HQ-SMem as merely a SAM post-processing stage. The paper argues for a broader interpretation: HQ-SMem is a sequence-level modification to a memory-based tracker, in which mask quality, memory compactness, and feedback reliability are coupled. This suggests that its contribution lies less in isolated per-frame refinement than in changing the temporal operating regime of the underlying VOS model.

2. High-quality mask refinement with appearance-verified candidate selection

The first major component is the use of SAM-HQ with appearance-based candidate selection, which the paper also frames as an online teacher-forcing mechanism. For each frame, the VOS model’s current object masks $\{y_{t,1}, \dots, y_{t,N}\}$ are used as visual prompts to SAM-HQ, which produces multiple mask candidates per object:

$$\mathbf{s}_{t, i} = \{s_{t, i}^{1}, s_{t, i}^{2}, s_{t, i}^{3}\}.$$

The method does not blindly accept the top SAM-HQ output. Instead, it compares each candidate to the current VOS prediction using DINO appearance features. The selection rule is given as

$x_t$0

and

$x_t$1

Here $x_t$2 denotes DINO appearance features for object $x_t$3 in frame $x_t$4, and $x_t$5 is a similarity threshold (Kazemi et al., 25 Jul 2025).

This mechanism is intended to resolve prompt ambiguity. The paper explicitly notes that SAM prompts can be ambiguous, especially when multiple objects look similar or the prompt is not precise. HQ-SMem therefore allows SAM-HQ to improve a mask only if the proposed mask remains appearance-consistent with the tracked object. In the paper’s framing, this prevents “improvements” that switch the object identity or introduce incorrect boundaries.

The significance of this design is empirical as well as conceptual. The ablation study reports that naively adding SAM without validation hurts Q to 0.622, despite raising accuracy somewhat, whereas the validated SAM-HQTF configuration raises performance to Q = 0.654 and Acc = 0.788. The result is used to support a central claim of the paper: SAM alone can be unstable, and the DINO-based validation is what makes high-quality feedback reliable.

3. Online high-quality feedback and self-supervised distillation

After candidate validation, the accepted high-quality mask is fed back into the tracker. The autoregressive update is rewritten as

$x_t$6

The paper describes this as online teacher forcing: rather than feeding the base tracker its raw previous prediction, HQ-SMem feeds back a refined mask $x_t$7 (Kazemi et al., 25 Jul 2025).

The paper also describes this feedback path as dynamic appearance-model update / self-supervised distillation feedback. The term does not refer to a new offline distillation objective. The paper explicitly states that it does not introduce an explicit new distillation loss term beyond the base segmentation loss. Instead, the distillation effect is attributed to inference-time conditioning: refined masks from SAM-HQ are checked against DINO appearance consistency and then reinjected into the base tracker as improved priors. In that sense, the tracker is repeatedly conditioned on better pseudo-targets without external labels during inference.

The intended effect is sequence stabilization. Later frames are conditioned on improved structure and boundaries rather than on drifted, coarse predictions. The paper emphasizes that this is especially relevant for scenes with topologically transforming objects, occlusion, and fast motion. A plausible implication is that HQ-SMem treats boundary quality not as an endpoint but as a state variable that shapes later temporal predictions.

4. Dynamic smart memory and selective replacement

The second major contribution is the dynamic smart memory mechanism. The paper argues that many state-of-the-art trackers retain too many redundant frames, so memory grows unnecessarily and salient appearance information becomes diluted, which increases drift. HQ-SMem therefore selectively stores key frames and removes redundant ones based on a relevance and freshness score computed in embedding space (Kazemi et al., 25 Jul 2025).

Given a memory snapshot

$x_t$8

the current frame embedding $x_t$9 is compared with each memory item $\mathcal{E}$0 using cosine similarity:

$\mathcal{E}$1

Freshness is defined as inverse age:

$\mathcal{E}$2

The removal score is then

$\mathcal{E}$3

and the frame with highest score is deleted:

$\mathcal{E}$4

The updated memory is

$\mathcal{E}$5

This is a selective replacement rule: if the current frame is sufficiently relevant to the candidate being removed, the old one is swapped out; otherwise the new frame is appended. The paper characterizes the resulting memory as representative, non-redundant, discriminative. It further reports close-to-constant memory scaling on long videos, contrasting this behavior with DeAOT’s roughly linear growth.

The significance of this component is not only memory reduction. The paper ties memory compactness directly to robustness: by retaining useful discriminative keyframes while removing redundant ones, the tracker is said to handle occlusion more robustly and reduce drift over extended temporal durations. A common misconception is that the memory module serves only an efficiency function; the paper instead treats it as an accuracy mechanism as well.

5. Optimization protocol, datasets, and reported performance

For optimization, HQ-SMem uses the same training loss family as AOT, DeAOT, and RMem: bootstrapped cross-entropy loss and Jaccard loss with equal weighting and averaging (Kazemi et al., 25 Jul 2025). Training is performed with AdamW, for 100k steps on VOTS/LVOS/Long Video experiments, with batch size 4, on 4 V100 GPUs, using learning rate $\mathcal{E}$6 decaying polynomially to $\mathcal{E}$7:

$\mathcal{E}$8

The encoder learning rate is set to 10% of the decoder’s, and weight decay is 0.07. For VOTSt, the paper uses 21k steps and sequence length 15; for the other datasets, it uses sequence length 5 in training and a memory bank update frequency of $\mathcal{E}$9 during inference on long videos. The paper also discusses training on DAVIS2017 and YouTubeVOS2019 for the DeAOT backbone.

The main evaluation benchmarks are VOTS2024, VOTSt 2024, Long Video Dataset, and LVOS. On VOTS-style benchmarks, the reported metrics are Quality (Q),

$\mathcal{D}$0

together with Accuracy (Acc), Robustness (Rob), Not-Reported Error (NRE), Drift-Rate Error (DRE), and Absence-Detection Quality (ADQ). For long-video benchmarks, the paper reports region similarity $\mathcal{D}$1 and contour accuracy $\mathcal{D}$2.

Benchmark	Configuration	Reported result
VOTSt 2024	HQ-SMem	$\mathcal{D}$3, $\mathcal{D}$4, $\mathcal{D}$5, $\mathcal{D}$6, $\mathcal{D}$7, $\mathcal{D}$8
Long Video Dataset	DeAOT + HQ-SMem	$\mathcal{D}$9, $\mathbf{M}$0, $\mathbf{M}$1
LVOS	DeAOT + HQ-SMem	$\mathbf{M}$2, $\mathbf{M}$3, $\mathbf{M}$4
VOTS2024 leaderboard	HQ-SMem	$\mathbf{M}$5, $\mathbf{M}$6

The paper states that HQ-SMem consistently ranks among the top two on VOTS and VOTSt 2024 datasets, and describes it as ranking second overall on the official VOTS2024 leaderboard. On VOTSt 2024, the reported numbers are presented as outperforming baselines such as S3_Track, AOTPlus, VOTST2024_RMemAOT, and RMem in quality and drift-related measures. The paper further notes that this corresponds to about a 3% gain in $\mathbf{M}$7 and $\mathbf{M}$8 over the base DeAOT tracker in the VOTSt setting.

The ablation results are central to the paper’s interpretation. On VOTS, baseline DeAOT gives Q = 0.636 and Acc = 0.751. Adding Smart Memory only increases this to Q = 0.639 and Acc = 0.758. Using SAM-HQTF raises performance to Q = 0.654 and Acc = 0.788. The full system, combining SAM-HQTF and SMem, reaches Q = 0.660 and Acc = 0.793. Across multiple DeAOT variants and DMAOT variants, the paper reports average improvements of about 4.9% in Q and 5.6% in Acc. The qualitative analysis is aligned with these figures: HQ-SMem is reported to produce tighter boundaries, better object separation in crowded scenes, and more stable tracking under rapid motion or similar-looking instances.

6. Failure modes, limitations, and broader significance

The paper is explicit that the contribution is combinatorial rather than singular: high-quality mask refinement alone is not enough, and efficient memory alone is not enough. Its main takeaway is that the gains arise from combining appearance-verified SAM-HQ prompting, selective memory replacement, and online high-quality feedback in the tracker’s autoregressive loop (Kazemi et al., 25 Jul 2025).

At the same time, the method retains several limitations. HQ-SMem still struggles with very complex topological change, severe occlusion, and strong scale variation. It can confuse similar objects in crowded scenes, and its number of trackable objects is still limited by the base DeAOT memory design, roughly around ten objects. These caveats are important for interpreting the reported benchmark gains: they indicate an advance in robustness and scalability, but not a complete resolution of hard multi-object VOS failure modes.

The future directions proposed in the paper are hierarchical memory structures, multiple smart-memory update mechanisms, temporal position embeddings, real-time optimization, and multimodal extension. The paper also implies that future VOS and VOT methods may increasingly rely on dynamic memory policies and quality-checked foundation-model refinement, rather than simply larger backbones or denser memory banks. This suggests a shift in emphasis from static capacity scaling to adaptive sequence control, with HQ-SMem as a representative example of that design trajectory.