HMHI-Net: Hierarchical Memory for UVOS

• The paper introduces HMHI-Net, a framework that combines dual memory banks to preserve fine pixel details and global semantics for unsupervised video segmentation.
• It employs a heterogeneous interaction mechanism using PLAM and SGIM to effectively align local features with high-level contextual cues.
• The method outperforms benchmarks by achieving notable improvements in region similarity and boundary accuracy across diverse datasets.

HMHI-Net, short for Hierarchical Memory with Heterogeneous Interaction Network, is a framework for unsupervised video object segmentation (UVOS) that predicts pixel-level masks for the most salient object or objects in a video without annotations or first-frame masks. It is designed around the claim that UVOS is not well served by memory systems that store only high-level semantic features, because the absence of pixel-level priors makes fine-grained spatial information especially consequential. HMHI-Net addresses this by combining a hierarchical memory that stores both shallow and high-level features with a heterogeneous interaction mechanism that fuses them according to their differing statistical roles, namely pixel-oriented locality and semantic-oriented globality. The model uses two memory banks, attention-based memory readout, the Pixel-guided Local Alignment Module (PLAM), and the Semantic-guided Global Integration Module (SGIM), and reports state-of-the-art performance across UVOS and video saliency detection benchmarks (Xiangyu et al., 30 Jul 2025).

1. Problem setting and motivation

UVOS aims to predict pixel-wise masks of the most salient object(s) in a video without any annotations or first-frame masks. This distinguishes it from semi-supervised video object segmentation, where pixel-level priors anchor object identity and boundaries. In HMHI-Net’s formulation, the central difficulty is that UVOS lacks those priors, so the model must recover both object semantics and spatially precise contours from appearance-motion cues and temporal memory alone (Xiangyu et al., 30 Jul 2025).

The motivating diagnosis is that existing memory-based UVOS methods usually store and retrieve high-level features only. These features are semantically rich but spatially coarse: they emphasize a few discriminative tokens and lose fine-grained detail through downsampling and abstraction. The reported layer-wise attention visualizations indicate that shallow encoder levels such as level 1–2 distribute attention broadly over foreground pixels, whereas high encoder levels such as level 3–4 focus on sparse semantic keypoints. This makes high-level memory insufficient for precise boundaries and small structures, particularly under occlusion, motion blur, or background clutter (Xiangyu et al., 30 Jul 2025).

HMHI-Net is therefore organized around two complementary premises. First, shallow features matter because they preserve fine detail that is otherwise lost in high-level semantics. Second, shallow and high-level features should not be fused with a homogeneous mechanism, because their intrinsic discrepancies are substantial. The architecture consequently stores shallow/pixel memory and high/semantic memory separately and couples them through bidirectional interaction modules specialized for local alignment and global integration.

2. Architectural organization

The model takes as input an RGB frame and optical flow,

$$I_t \in \mathbb{R}^{H \times W \times 3}, \qquad O_t \in \mathbb{R}^{H \times W \times 3},$$

and processes them with a 4-level hierarchical encoder, using SegFormer mit_b1 by default (Xiangyu et al., 30 Jul 2025). For each level $i \in \{1,2,3,4\}$, the encoder produces multi-scale appearance and motion features,

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$

with

$H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$

For mit_b1, the channels are typically [64, 128, 320, 512], and the appearance and flow streams are merged by addition:

$F_t^i = I_t^i + O_t^i.$

The architectural decomposition is as follows.

Component	Definition	Role
Encoder	4-level hierarchical encoder	Produces multi-scale appearance-flow features
Pixel memory	Layer $i=2$ bank	Stores shallow, fine-grained features
Semantic memory	Layer $i=4$ bank	Stores high-level semantic features
PLAM	Pixel-guided Local Alignment Module	Injects shallow detail into high-level features
SGIM	Semantic-guided Global Integration Module	Injects global semantics into shallow features
Decoder	Hierarchical multi-scale decoder	Produces $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$

The two memory banks are positioned at layer 2 and layer 4. The shallow bank,

$$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$

stores pixel-oriented information, while the semantic bank,

$$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$

stores semantic-oriented information from the latest $i \in \{1,2,3,4\}$0 reference frames in a sliding window (Xiangyu et al., 30 Jul 2025).

After memory read and heterogeneous interaction, the decoder fuses

$i \in \{1,2,3,4\}$1

in a bottom-up manner to produce the segmentation mask

$i \in \{1,2,3,4\}$2

The refined features and predicted mask are then written back into memory under a FIFO sliding window with stride $i \in \{1,2,3,4\}$3.

3. Hierarchical memory design and memory operations

The memory system consists of two banks, $i \in \{1,2,3,4\}$4 and $i \in \{1,2,3,4\}$5, each storing the most recent $i \in \{1,2,3,4\}$6 reference elements under a sliding window of capacity $i \in \{1,2,3,4\}$7, with $i \in \{1,2,3,4\}$8 (Xiangyu et al., 30 Jul 2025). The model’s central claim is not merely that memory is useful, but that the level at which memory is stored is structurally important for UVOS. Pixel-level details are primarily maintained in the shallow bank, and semantic cues are maintained in the high-level bank.

The memory read rule is shared across shallow and semantic banks. For a feature $i \in \{1,2,3,4\}$9, the model first performs self-attention to strengthen internal structure:

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$0

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$1

It then performs cross-memory attention with the reference features $$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$2:

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$3

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$4

followed by

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$5

This yields the refined shallow and high-level features $$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$6 and $$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$7 (Xiangyu et al., 30 Jul 2025).

Memory writing is also explicit. Each stored element is produced by a memory encoder that fuses the refined feature with the predicted mask:

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$8

The update rule is

$$I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},$$9

with updates performed every $H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$0 frames, where $H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$1 by default. The first frame does not perform memory read or interaction and is used only to initialize the banks. The design uses full softmax across all reference tokens and no sparsification such as top-$H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$2 selection (Xiangyu et al., 30 Jul 2025).

This organization makes the memory hierarchy functional rather than merely representational. The shallow memory supplies fine local evidence for later alignment, while the semantic memory supplies object-level context for later global integration.

4. Heterogeneous interaction mechanism

The heterogeneous interaction mechanism is the second core component of HMHI-Net. It operates on the refined shallow and high-level features,

$H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$3

and explicitly models their different roles. The design is bidirectional: PLAM performs shallow-to-high refinement, and SGIM performs high-to-shallow refinement (Xiangyu et al., 30 Jul 2025).

PLAM: Pixel-guided Local Alignment

PLAM is intended to inject local, fine-grained detail from shallow features into high-level features while preserving spatial coherence. It first aligns the shallow features to the high-level resolution and channel dimensionality:

$H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$4

Here, ConvReLU downsamples from $H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$5 to $H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$6, and Linear maps $H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$7.

The aligned shallow feature is then concatenated with the high-level feature:

$H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$8

PLAM applies channel attention in squeeze-excitation style:

$H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.$9

and then spatial attention with pooled descriptors:

$F_t^i = I_t^i + O_t^i.$0

$F_t^i = I_t^i + O_t^i.$1

The output is

$F_t^i = I_t^i + O_t^i.$2

The stated purpose of this sequence is locality-preserving alignment. Shallow detail is introduced into semantic tokens, but gated so that unrelated local structure is not indiscriminately fused into high-level representations.

SGIM: Semantic-guided Global Integration

SGIM propagates global semantic context from high-level features into shallow features so that shallow tokens gain stronger object-level semantics. High-level channels are first aligned:

$F_t^i = I_t^i + O_t^i.$3

The shallow feature then undergoes self-attention:

$F_t^i = I_t^i + O_t^i.$4

$F_t^i = I_t^i + O_t^i.$5

Global semantic integration is then performed by cross-attention from shallow queries to high-level keys and values:

$F_t^i = I_t^i + O_t^i.$6

$F_t^i = I_t^i + O_t^i.$7

$F_t^i = I_t^i + O_t^i.$8

The paper characterizes the balance as heterogeneity-aware. PLAM preserves locality and alignment when injecting shallow detail into coarse semantics, whereas SGIM uses global cross-attention so each shallow token can access broad semantic context despite different spatial resolutions and token cardinalities. The bidirectional interaction is therefore not symmetric; it is specialized to the different statistics of shallow and high-level features (Xiangyu et al., 30 Jul 2025).

5. Training objective, inference pipeline, and implementation

HMHI-Net is optimized with a mixture of segmentation losses applied frame-wise within each training clip:

• Binary cross entropy $F_t^i = I_t^i + O_t^i.$9
• Focal loss $i=2$0
• Dice loss $i=2$1

The per-frame loss is

$i=2$2

and the final objective over $i=2$3 frames is

$i=2$4

The role of these terms is explicit in the formulation: focal loss addresses class imbalance, and Dice loss improves overlap/contours (Xiangyu et al., 30 Jul 2025).

The unsupervised cues are limited and clearly specified. Optical flow provides motion information complementary to appearance and is injected at all encoder levels through

$i=2$5

The predicted mask $i=2$6 is written into memory and used to guide subsequent frames, but no explicit entropy regularization, no contrastive objective, and no explicit pseudo-label generation beyond the predicted masks stored in memory are described (Xiangyu et al., 30 Jul 2025).

The inference pipeline is sequential:

1. Resize frames to $i=2$7 and compute optical flow.
2. Encode $i=2$8 and $i=2$9 into multi-scale features.
3. For $i=4$0 and $i=4$1, apply self-attention and cross-attention with memory to obtain $i=4$2.
4. Apply PLAM and SGIM to obtain $i=4$3 and $i=4$4.
5. Decode $i=4$5 to predict $i=4$6.
6. Write $i=4$7 and $i=4$8 into memory and update $i=4$9 and $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$0.
7. Repeat for the next frame.

The reported implementation uses SegFormer mit_b1 as backbone by default, with mit_b2, mit_b3, and Swin-Tiny also evaluated. Training uses AdamW, with pretraining lr = $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$1 for 150 epochs on YouTube-VOS, and fine-tuning lr = $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$2 with CosineAnnealingLR until convergence. The experiments use 4 × NVIDIA RTX 4090, and the memory parameters are $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$3 and $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$4 for both training and inference. The paper reports $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$5 FPS and $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$6M parameters for HMHI-Net with mit_b1, compared with $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$7M for the baseline and $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$8M for Multi-Mem; batch size and FLOPs are not reported (Xiangyu et al., 30 Jul 2025).

6. Benchmarks, quantitative results, and ablations

HMHI-Net is evaluated on both UVOS and video saliency detection (VSOD) benchmarks. For UVOS, the metrics are region similarity $$M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$$9 (IoU), boundary $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$0, and their average $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$1. For VSOD, the metrics are MAE, max F-measure $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$2, enhanced alignment $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$3, and structure measure $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$4 (Xiangyu et al., 30 Jul 2025).

UVOS results

Dataset	Metric	Result
DAVIS-16	$$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$5 / $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$6 / $$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$7	89.8% / 88.6% / 91.0%
FBMS	$$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$8	86.9%
YouTube-Objects	$$M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},$$9	76.2%

The model is reported to achieve state-of-the-art UVOS performance, with improvements versus recent state of the art of approximately +1.6% $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$0 on DAVIS-16, +3.5% $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$1 on FBMS, and +1.5% $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$2 on YouTube-Objects (Xiangyu et al., 30 Jul 2025).

VSOD results

Dataset	Metrics	Result
DAVSOD	MAE / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$3 / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$4 / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$5	0.054 / 0.801 / 0.896 / 0.847
DAVIS-16	MAE / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$6 / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$7 / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$8	0.009 / 0.947 / 0.990 / 0.951
ViSal	MAE / $$M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},$$9 / $i \in \{1,2,3,4\}$00 / $i \in \{1,2,3,4\}$01	0.012 / 0.962 / 0.991 / 0.960
FBMS	MAE / $i \in \{1,2,3,4\}$02 / $i \in \{1,2,3,4\}$03 / $i \in \{1,2,3,4\}$04	0.030 / 0.946 / 0.977 / 0.930

Across these datasets, HMHI-Net achieves best or runner-up scores (Xiangyu et al., 30 Jul 2025).

The ablation results are important because they directly support the paper’s design claims. Memory layer selection shows the largest gain from shallow layer-2 memory (+0.8% $i \in \{1,2,3,4\}$05), followed by a modest gain from layer-3 (+0.6%), while layer-4 only yields only a marginal gain (+0.2%). This is presented as empirical support for the necessity of shallow memory. In module ablations, hierarchical memory alone outperforms single-level memory; PLAM (S2H) adds +1.0% on DAVIS-16, SGIM (H2S) adds +1.3%, and the full HMHI-Net adds +1.4%. The paper further reports that swapping modules, thereby violating the intended heterogeneity, degrades performance (Xiangyu et al., 30 Jul 2025).

Robustness is also reported along several axes. HMHI-Net yields consistent gains across mit_b1, mit_b2, mit_b3, and Swin-Tiny, and using both image and flow performs best, although even single-modality input benefits from HMHI-Net relative to the baseline. This suggests that the claimed contribution is not restricted to a single backbone choice or to the two-stream setting alone.

7. Relation to prior work, limitations, and prospective extensions

HMHI-Net is situated at the intersection of memory-augmented video segmentation and motion-appearance fusion. In semi-supervised VOS, methods such as STM, STCN, AOT, XMem, Cutie, and SAM2 are identified as representative memory-based systems that emphasize spatio-temporal memory and, in later variants, memory selection or compression. In UVOS, methods such as TGFormer, PMN, DPA, and GSA are described as introducing memory but mostly at high semantic levels, which the HMHI-Net paper argues explains their marginal gains. A separate line of work, including MATNet, RTNet, HFAN, TMO, and SimulFlow, relies on motion-appearance fusion without memory and is described as limited by short-term flow cues (Xiangyu et al., 30 Jul 2025).

The paper’s principal claim is therefore a design correction rather than a general endorsement of “more memory.” The point is that UVOS requires memory at the right representational levels, coupled through interaction mechanisms that respect heterogeneity. This suggests that the contribution is as much about feature stratification as about temporal storage.

The reported limitations are concrete. The use of dual banks and bidirectional interactions adds computation and storage overhead, and the authors identify token pruning and memory compression as possible efficiency directions. Failure cases remain in extremely complex scenes with rapid scale changes or severe occlusions, where cross-resolution alignment may still be difficult. Proposed extensions include generalization to other backbones and tasks, including semi-/fully supervised VOS and video saliency, as well as the integration of richer unsupervised signals such as self-supervised contrastive objectives or entropy regularization (Xiangyu et al., 30 Jul 2025).

A common source of confusion is nominal rather than conceptual: HMHI-Net refers specifically to the UVOS model introduced in “Shallow Features Matter: Hierarchical Memory with Heterogeneous Interaction for Unsupervised Video Object Segmentation” (Xiangyu et al., 30 Jul 2025). It is distinct from similarly named systems in unrelated domains. Within video segmentation, its defining proposition is that shallow features are not auxiliary cues but indispensable memory elements when dense object masks must be inferred without annotation-derived priors.