HiM2SAM: Enhancing Video Object Tracking

• HiM2SAM is a trainless enhancement of SAM2 that uses hierarchical motion estimation and dual-memory banks to address challenges in long-term video tracking.
• It fuses coarse linear prediction with on-demand nonlinear refinement to robustly re-rank proposal masks amid fast movement, occlusions, and clutter.
• The method achieves significant AUC gains on benchmarks with minimal latency, ensuring efficient and reliable tracking performance.

HiM2SAM is a training-free enhancement of the Segment Anything Model 2 (SAM2) for video object tracking. It is designed for long sequences in which SAM2 can drift under fast or nonlinear motion, occlusions, background clutter, and target disappearance or reappearance. The method augments SAM2 with two coupled mechanisms: hierarchical motion estimation, which re-ranks proposal masks using coarse linear prediction and selective nonlinear refinement, and memory optimization, which replaces a single FIFO memory with coordinated short-term and long-term banks. The reported outcome is a low-overhead improvement in long-term tracking; with the large model on LaSOT, the paper reports a 9.6% relative AUC improvement over the original SAM2 (Chen et al., 10 Jul 2025).

1. Conceptual scope and design rationale

HiM2SAM addresses three failure modes identified for SAM2 in long-term tracking. First, SAM2 lacks explicit motion modeling, so proposal selection can become unstable when motion is fast or highly non-rigid. Second, its FIFO memory can evict useful historical context, which weakens re-identification after long occlusions or strong appearance change. Third, proposal ambiguity in cluttered scenes can cause identity switches when several plausible masks are returned for the same frame (Chen et al., 10 Jul 2025).

The method introduces two orthogonal modifications. The first is Hierarchical Motion Estimation (HME), a two-stage module that combines lightweight linear prediction with on-demand nonlinear refinement. The second is a memory-bank redesign that separates recent reliable context from sparse long-term anchors. Because both changes operate at inference time and do not retrain SAM2, HiM2SAM is positioned as a trainless, low-overhead upgrade rather than a new end-to-end architecture (Chen et al., 10 Jul 2025).

A central implication of this design is that HiM2SAM preserves SAM2’s foundation-model priors while intervening only at proposal selection and memory writing. This suggests a modular philosophy: temporal robustness is improved not by changing the segmentation backbone, but by constraining how its outputs are interpreted and stored.

2. Integration with the SAM2 inference pipeline

HiM2SAM is built on the standard SAM2 tracking stack. In the baseline, a ViT-based Hiera image encoder produces multi-scale per-frame features, the mask decoder outputs multiple proposal masks per frame together with IoU or objectness scores, and the selected mask is encoded and pushed into a FIFO memory bank. Tracking is then driven by memory attention between the current frame and stored memory entries (Chen et al., 10 Jul 2025).

HiM2SAM preserves this structure but alters two decision points. Proposal selection becomes motion-aware: instead of accepting the decoder’s highest-scoring mask, the method re-ranks the proposal set with HME. Memory updating becomes structured: instead of writing every accepted mask into a single FIFO queue, HiM2SAM writes reliable recent frames into short-term memory and selectively stores distinctive high-confidence frames in a separate long-term bank (Chen et al., 10 Jul 2025).

The remainder of SAM2’s mechanism, including the existing memory attention, is left unchanged. This is significant because the method’s empirical gains arise from controlling the information fed into the memory system, not from redesigning the memory-attention operator itself.

3. Hierarchical Motion Estimation

HME is a two-stage motion module. Its coarse stage uses a constant-velocity state-space model for the target’s bounding box, with state

$x_t = [c_x, c_y, v_x, v_y, s, r]^\top,$

where $c_x,c_y$ are the box center, $v_x,v_y$ are velocities, and $s,r$ encode scale and aspect or edge-length ratio. Prediction and update follow a standard linear Gaussian form:

$$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$

$$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$

$K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$

$$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$

Given the predicted box $\hat{B}_t$ and SAM2 proposals $\{M_i^t,B_i^t\}$ with decoder scores $c_x,c_y$0, the coarse motion score is

$c_x,c_y$1

and the coarse confidence is

$c_x,c_y$2

If the best coarse score exceeds a threshold $c_x,c_y$3, HiM2SAM selects that proposal directly (Chen et al., 10 Jul 2025).

If the best coarse confidence falls below $c_x,c_y$4, the method triggers selective nonlinear refinement. For each proposal, it samples $c_x,c_y$5 points on the mask, tracks them backward for $c_x,c_y$6 historical frames with a point tracker such as CoTracker3, and reconstructs a soft historical mask by radial basis interpolation:

$c_x,c_y$7

This reconstructed mask is compared with the previously accepted historical mask using Dice similarity,

$c_x,c_y$8

and the nonlinear score is obtained by averaging over valid look-back frames:

$c_x,c_y$9

Final re-ranking uses

$v_x,v_y$0

The selected proposal is then used as the measurement $v_x,v_y$1 for the Kalman update (Chen et al., 10 Jul 2025).

This hierarchy is central to HiM2SAM’s efficiency. The coarse stage is effectively negligible relative to SAM2, while the expensive pixel-level stage is invoked only on difficult frames. The paper reports that a point-tracker-only refinement adds $v_x,v_y$2 ms per frame, whereas the full selective HME-based system adds only $v_x,v_y$3 ms per frame on an NVIDIA A6000 GPU and Intel i7-9700K CPU (Chen et al., 10 Jul 2025).

4. Split-memory organization and distinctive-frame retention

HiM2SAM replaces SAM2’s single FIFO memory with short-term memory (SM) and long-term memory (LM). SM is a recent-context bank updated in FIFO order, but only when the selected proposal is sufficiently reliable. The gating variable is the HME confidence $v_x,v_y$4; a frame is written to SM only if

$v_x,v_y$5

This suppresses pollution from low-confidence predictions, especially around occlusion boundaries or recovery phases (Chen et al., 10 Jul 2025).

LM is designed for re-identification and distractor resistance. Rather than storing frames by recency, HiM2SAM stores a small set of distinctive high-confidence frames. Distinctiveness is measured by disagreement among the proposal set. After selecting $v_x,v_y$6, the method compares it with the remaining proposals using the directed Hausdorff distance on contours:

$v_x,v_y$7

If

$v_x,v_y$8

and the selected proposal is also high-confidence, the frame is admitted to LM (Chen et al., 10 Jul 2025).

The interpretation given in the paper is that strong proposal disagreement indicates ambiguity caused by distractors or alternative target hypotheses. Retaining such frames in LM equips future memory attention with sparse anchors that are informative precisely when the target reappears after absence or when distractors resemble the target. SM thus preserves local continuity, while LM preserves identity across long temporal gaps (Chen et al., 10 Jul 2025).

5. Benchmark performance, attribute analysis, and ablations

HiM2SAM is evaluated on LaSOT, LaSOT_ext, VOT-LT2020, and VOT-LT2022. For LaSOT and LaSOT_ext, the main metrics are success AUC, precision $v_x,v_y$9, and normalized precision $s,r$0. For VOT-LT benchmarks, the paper reports precision, recall, and F-score (Chen et al., 10 Jul 2025).

Scale	LaSOT AUC: SAM2.1 → HiM2SAM	LaSOT_ext AUC: SAM2.1 → HiM2SAM
T	66.7 → 72.4	52.3 → 58.6
S	66.5 → 73.1	56.1 → 60.3
B	66.0 → 73.4	55.5 → 59.3
L	68.5 → 75.1	58.6 → 62.8

These results correspond to relative AUC gains of $s,r$1, $s,r$2, $s,r$3, and $s,r$4 on LaSOT for the T, S, B, and L models, respectively. On LaSOT_ext, the corresponding relative gains are $s,r$5, $s,r$6, $s,r$7, and $s,r$8 (Chen et al., 10 Jul 2025). The detailed VOT-LT results reported for the large model are $s,r$9 on VOT-LT2020 and $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$0 on VOT-LT2022, described as the best among the zero-shot baselines reported (Chen et al., 10 Jul 2025).

The paper also reports attribute-level AUC gains over SAMURAI for the base model, including view change $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$1, camera motion $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$2, fast motion $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$3, deformation $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$4, partial occlusion $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$5, background clutter $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$6, full occlusion $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$7, low resolution $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$8, and illumination variation $$x_{t|t-1} = A x_{t-1|t-1}, \qquad P_{t|t-1} = A P_{t-1|t-1} A^\top + Q,$$9 (Chen et al., 10 Jul 2025). This pattern is consistent with the method’s stated objective: its strongest gains occur on precisely those attributes that expose SAM2’s lack of explicit motion reasoning and recency-biased memory.

Ablation on LaSOT with the large model clarifies the role of each component. The reported baseline is $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$0 AUC. Adding the Kalman filter alone yields $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$1; SM alone yields $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$2 with negligible overhead; LM alone yields $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$3 with $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$4 ms; KF+SM reaches $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$5 with $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$6 ms; KF+PT+SM reaches $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$7 with $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$8 ms; and the full system reaches $$z_t = Hx_t + \epsilon,\qquad \epsilon \sim \mathcal{N}(0,R),$$9 AUC, $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$0 normalized precision, and $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$1 precision with $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$2 ms per frame (Chen et al., 10 Jul 2025). The point-tracker-only variant reaches $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$3 AUC but incurs the much larger $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$4 ms latency, which is the empirical justification for the selective hierarchy.

6. Terminology, related usages, limitations, and deployment

In the literature covered here, “HiM2SAM” specifically denotes the long-term tracking method that enhances SAM2 with hierarchical motion estimation and memory optimization (Chen et al., 10 Jul 2025). It should not be conflated with the Hierarchical Multi-view Interaction Module (HMIM) in MGD-SAM2; that paper explicitly states that it does not use the name “HiM2SAM,” and instead uses HMIM as one module inside a different high-resolution class-agnostic segmentation framework (Shen et al., 31 Mar 2025). Nor is it synonymous with MedSAM-2, which also modifies SAM2’s memory handling but does so for universal 2D and 3D medical segmentation by treating medical images as video via a self-sorting memory bank (Zhu et al., 2024).

The reported limitations are operational rather than architectural. The fine stage can degrade when the point tracker fails under extreme motion blur, drastic illumination change, or very low texture. Performance is sensitive to the refinement trigger $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$5 and to memory budgets: an overly high trigger can invoke refinement too frequently and increase latency, while an overly small LM weakens re-identification after long occlusions. The coarse state model is linear, so abrupt scale or aspect changes can temporarily reduce the quality of $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$6. Finally, the method still depends on the underlying proposal quality of SAM2; if the proposal set is consistently poor under strong domain shift, re-ranking and memory selection have less leverage (Chen et al., 10 Jul 2025).

The deployment guidance given in the paper is correspondingly pragmatic. It recommends starting from the released default configurations for $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$7, $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$8, $K_t = P_{t|t-1}H^\top(HP_{t|t-1}H^\top + R)^{-1},$9, $$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$0, $$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$1, $$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$2, the number of sampled points $$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$3, the look-back horizon $$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$4, and the RBF width $$x_{t|t} = x_{t|t-1} + K_t(z_t - Hx_{t|t-1}),\qquad P_{t|t} = (I-K_tH)P_{t|t}.$$5, all available in the official repository. Stricter gating is suggested for real-time or resource-constrained settings, while more permissive refinement and somewhat larger LM capacity are suggested for videos with frequent reappearance, heavy clutter, or strong nonlinear motion. The implementation is released at https://github.com/LouisFinner/HiM2SAM (Chen et al., 10 Jul 2025).