---
title: 'HMHI-Net: Hierarchical Memory for UVOS'
url: https://www.emergentmind.com/topics/hmhi-net
type: topic
---

# HMHI-Net: Hierarchical Memory for UVOS

HMHI-Net, short for **Hierarchical Memory with Heterogeneous Interaction Network**, is a framework for **unsupervised video object segmentation (UVOS)** that predicts pixel-level masks for the most salient object or objects in a video without annotations or first-frame masks. It is designed around the claim that UVOS is not well served by memory systems that store only high-level semantic features, because the absence of pixel-level priors makes fine-grained spatial information especially consequential. HMHI-Net addresses this by combining a **hierarchical memory** that stores both shallow and high-level features with a **heterogeneous interaction mechanism** that fuses them according to their differing statistical roles, namely pixel-oriented locality and semantic-oriented globality. The model uses two memory banks, attention-based memory readout, the **Pixel-guided Local Alignment Module (PLAM)**, and the **Semantic-guided Global Integration Module (SGIM)**, and reports state-of-the-art performance across UVOS and video saliency detection benchmarks [2507.22465].

## 1. Problem setting and motivation

UVOS aims to predict pixel-wise masks of the most salient object(s) in a video **without any annotations or first-frame masks**. This distinguishes it from semi-supervised video object segmentation, where pixel-level priors anchor object identity and boundaries. In HMHI-Net’s formulation, the central difficulty is that UVOS lacks those priors, so the model must recover both object semantics and spatially precise contours from appearance-motion cues and temporal memory alone [2507.22465].

The motivating diagnosis is that existing memory-based UVOS methods usually **store and retrieve high-level features only**. These features are semantically rich but spatially coarse: they emphasize a few discriminative tokens and lose fine-grained detail through downsampling and abstraction. The reported layer-wise attention visualizations indicate that shallow encoder levels such as **level 1–2** distribute attention broadly over foreground pixels, whereas high encoder levels such as **level 3–4** focus on sparse semantic keypoints. This makes high-level memory insufficient for precise boundaries and small structures, particularly under **occlusion, motion blur, or background clutter** [2507.22465].

HMHI-Net is therefore organized around two complementary premises. First, **shallow features matter** because they preserve fine detail that is otherwise lost in high-level semantics. Second, shallow and high-level features should not be fused with a homogeneous mechanism, because their intrinsic discrepancies are substantial. The architecture consequently stores **shallow/pixel memory** and **high/semantic memory** separately and couples them through bidirectional interaction modules specialized for local alignment and global integration.

## 2. Architectural organization

The model takes as input an RGB frame and optical flow,
$$
I_t \in \mathbb{R}^{H \times W \times 3}, \qquad O_t \in \mathbb{R}^{H \times W \times 3},
$$
and processes them with a **4-level hierarchical encoder**, using **SegFormer mit\_b1 by default** [2507.22465]. For each level $i \in \{1,2,3,4\}$, the encoder produces multi-scale appearance and motion features,
$$
I_t^i \in \mathbb{R}^{H_iW_i \times C_i}, \qquad O_t^i \in \mathbb{R}^{H_iW_i \times C_i},
$$
with
$$
H_i = H/2^{i+1}, \qquad W_i = W/2^{i+1}.
$$
For **mit\_b1**, the channels are typically **[64, 128, 320, 512]**, and the appearance and flow streams are merged by addition:
$$
F_t^i = I_t^i + O_t^i.
$$

The architectural decomposition is as follows.

| Component | Definition | Role |
|---|---|---|
| Encoder | 4-level hierarchical encoder | Produces multi-scale appearance-flow features |
| Pixel memory | Layer $i=2$ bank | Stores shallow, fine-grained features |
| Semantic memory | Layer $i=4$ bank | Stores high-level semantic features |
| PLAM | Pixel-guided Local Alignment Module | Injects shallow detail into high-level features |
| SGIM | Semantic-guided Global Integration Module | Injects global semantics into shallow features |
| Decoder | Hierarchical multi-scale decoder | Produces $M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}$ |

The two memory banks are positioned at **layer 2** and **layer 4**. The shallow bank,
$$
M_p \equiv R^2 \in \mathbb{R}^{T \cdot H_2W_2 \times C_2},
$$
stores pixel-oriented information, while the semantic bank,
$$
M_s \equiv R^4 \in \mathbb{R}^{T \cdot H_4W_4 \times C_4},
$$
stores semantic-oriented information from the latest **$T$ reference frames** in a sliding window [2507.22465].

After memory read and heterogeneous interaction, the decoder fuses
$$
[F_t^1, F_t^{2''}, F_t^3, F_t^{4''}]
$$
in a **bottom-up manner** to produce the segmentation mask
$$
M_{\text{pred}} \in \mathbb{R}^{H \times W \times 1}.
$$
The refined features and predicted mask are then written back into memory under a **FIFO sliding window with stride $k$**.

## 3. Hierarchical memory design and memory operations

The memory system consists of two banks, **$R_t^2$** and **$R_t^4$**, each storing the most recent **$T$ reference elements** under a sliding window of capacity **$N$**, with $T \leq N$ [2507.22465]. The model’s central claim is not merely that memory is useful, but that the **level at which memory is stored** is structurally important for UVOS. Pixel-level details are primarily maintained in the shallow bank, and semantic cues are maintained in the high-level bank.

The memory read rule is shared across shallow and semantic banks. For a feature $F_t^i$, the model first performs **self-attention** to strengthen internal structure:
$$
Q_{sa}^{i,t} = F_t^i W_Q^{sa,i}, \quad
K_{sa}^{i,t} = F_t^i W_K^{sa,i}, \quad
V_{sa}^{i,t} = F_t^i W_V^{sa,i},
$$
$$
A_{sa}^{i,t} = \mathrm{softmax}\!\left(\frac{Q_{sa}^{i,t} K_{sa}^{i,t\top}}{\sqrt{d}}\right), \quad
F_t^{i,sa} = A_{sa}^{i,t} V_{sa}^{i,t}.
$$
It then performs **cross-memory attention** with the reference features $R_{t-1}^i$:
$$
Q_{mem}^{i,t} = F_t^{i,sa} W_Q^{mem,i}, \quad
K_{mem}^{i,t} = R_{t-1}^i W_K^{mem,i}, \quad
V_{mem}^{i,t} = R_{t-1}^i W_V^{mem,i},
$$
$$
A_{mem}^{i,t} = \mathrm{softmax}\!\left(\frac{Q_{mem}^{i,t} K_{mem}^{i,t\top}}{\sqrt{d}}\right), \quad
R^{i,t} = A_{mem}^{i,t} V_{mem}^{i,t},
$$
followed by
$$
F_t^{i'} = \mathrm{FFN}(F_t^{i,sa} + R^{i,t}).
$$
This yields the refined shallow and high-level features **$F_t^{2'}$** and **$F_t^{4'}$** [2507.22465].

Memory writing is also explicit. Each stored element is produced by a memory encoder that fuses the refined feature with the predicted mask:
$$
E_t^i = \mathrm{MemEnc}_i(F_t^{i''}, M_{\text{pred}}), \qquad i \in \{2,4\}.
$$
The update rule is
$$
R_t^i = \mathrm{Concat}(R_{t-1}^i[-(N-1):], E_t^i),
$$
with updates performed **every $k$ frames**, where **$k=1$ by default**. The **first frame** does not perform memory read or interaction and is used only to initialize the banks. The design uses **full softmax across all reference tokens** and **no sparsification** such as top-$k$ selection [2507.22465].

This organization makes the memory hierarchy functional rather than merely representational. The shallow memory supplies fine local evidence for later alignment, while the semantic memory supplies object-level context for later global integration.

## 4. Heterogeneous interaction mechanism

The heterogeneous interaction mechanism is the second core component of HMHI-Net. It operates on the refined shallow and high-level features,
$$
X^l \equiv F_t^{2'}, \qquad X^h \equiv F_t^{4'},
$$
and explicitly models their different roles. The design is bidirectional: **PLAM** performs shallow-to-high refinement, and **SGIM** performs high-to-shallow refinement [2507.22465].

### PLAM: Pixel-guided Local Alignment

PLAM is intended to inject local, fine-grained detail from shallow features into high-level features while preserving spatial coherence. It first aligns the shallow features to the high-level resolution and channel dimensionality:
$$
F_t^{2\_\mathrm{tmp}} = \mathrm{Linear}(\mathrm{ConvReLU}(F_t^{2'})).
$$
Here, **ConvReLU** downsamples from $H_2W_2$ to $H_4W_4$, and **Linear** maps $C_2 \to C_4$.

The aligned shallow feature is then concatenated with the high-level feature:
$$
U = \mathrm{Concat}(F_t^{4'}, F_t^{2\_\mathrm{tmp}}) \in \mathbb{R}^{H_4W_4 \times 2C_4}.
$$
PLAM applies **channel attention** in squeeze-excitation style:
$$
s = \mathrm{GAP}(U), \qquad
g_c = \sigma(W_2 \,\mathrm{ReLU}(W_1 s)), \qquad
U_c = U \odot g_c,
$$
and then **spatial attention** with pooled descriptors:
$$
P = \mathrm{Concat}(\mathrm{AvgPool}(U_c), \mathrm{MaxPool}(U_c)),
$$
$$
g_s = \sigma(\mathrm{Conv}(P)), \qquad
U_s = U_c \odot g_s.
$$
The output is
$$
F_t^{4''} = \mathrm{FFN}(U_s).
$$

The stated purpose of this sequence is locality-preserving alignment. Shallow detail is introduced into semantic tokens, but gated so that unrelated local structure is not indiscriminately fused into high-level representations.

### SGIM: Semantic-guided Global Integration

SGIM propagates global semantic context from high-level features into shallow features so that shallow tokens gain stronger object-level semantics. High-level channels are first aligned:
$$
F_t^{4\_\mathrm{tmp}} = \mathrm{Linear}(F_t^{4'}).
$$
The shallow feature then undergoes self-attention:
$$
Q_{sa}^{l} = F_t^{2'} W_Q^{sa,l}, \quad
K_{sa}^{l} = F_t^{2'} W_K^{sa,l}, \quad
V_{sa}^{l} = F_t^{2'} W_V^{sa,l},
$$
$$
A_{sa}^{l} = \mathrm{softmax}\!\left(\frac{Q_{sa}^{l} K_{sa}^{l\top}}{\sqrt{d}}\right), \quad
F_t^{2\_sa} = A_{sa}^{l} V_{sa}^{l}.
$$
Global semantic integration is then performed by cross-attention from shallow queries to high-level keys and values:
$$
Q_{ca}^{l} = F_t^{2\_sa} W_Q^{ca,l}, \quad
K_{ca}^{h} = F_t^{4\_\mathrm{tmp}} W_K^{ca,h}, \quad
V_{ca}^{h} = F_t^{4\_\mathrm{tmp}} W_V^{ca,h},
$$
$$
A_{ca} = \mathrm{softmax}\!\left(\frac{Q_{ca}^{l} K_{ca}^{h\top}}{\sqrt{d}}\right), \quad
G = A_{ca} V_{ca}^{h},
$$
$$
F_t^{2''} = \mathrm{FFN}(F_t^{2\_sa} + G).
$$

The paper characterizes the balance as **heterogeneity-aware**. PLAM preserves locality and alignment when injecting shallow detail into coarse semantics, whereas SGIM uses global cross-attention so each shallow token can access broad semantic context despite different spatial resolutions and token cardinalities. The bidirectional interaction is therefore not symmetric; it is specialized to the different statistics of shallow and high-level features [2507.22465].

## 5. Training objective, inference pipeline, and implementation

HMHI-Net is optimized with a **mixture of segmentation losses** applied frame-wise within each training clip:
- **Binary cross entropy** $L_{bce}$
- **Focal loss** $L_{foc}$
- **Dice loss** $L_{dice}$

The per-frame loss is
$$
L_{seg} = \lambda_{bce} L_{bce} + \lambda_{foc} L_{foc} + \lambda_{dice} L_{dice},
$$
and the final objective over **$T_{\text{train}}$** frames is
$$
L_{total} = \frac{1}{T_{\text{train}}} \sum_{t=1}^{T_{\text{train}}} L_{seg}^{(t)}.
$$
The role of these terms is explicit in the formulation: focal loss addresses **class imbalance**, and Dice loss improves **overlap/contours** [2507.22465].

The unsupervised cues are limited and clearly specified. **Optical flow** provides motion information complementary to appearance and is injected at all encoder levels through
$$
F_t^i = I_t^i + O_t^i.
$$
The predicted mask **$M_{\text{pred}}$** is written into memory and used to guide subsequent frames, but **no explicit entropy regularization**, **no contrastive objective**, and **no explicit pseudo-label generation beyond the predicted masks stored in memory** are described [2507.22465].

The inference pipeline is sequential:

1. Resize frames to **$512 \times 512$** and compute optical flow.
2. Encode $I_t$ and $O_t$ into multi-scale features.
3. For **$i=2$ and $4$**, apply self-attention and cross-attention with memory to obtain **$F_t^{i'}$**.
4. Apply **PLAM** and **SGIM** to obtain **$F_t^{4''}$** and **$F_t^{2''}$**.
5. Decode **$[F_t^1, F_t^{2''}, F_t^3, F_t^{4''}]$** to predict **$M_{\text{pred}}$**.
6. Write **$E_t^2$** and **$E_t^4$** into memory and update **$R_t^2$** and **$R_t^4$**.
7. Repeat for the next frame.

The reported implementation uses **SegFormer mit\_b1** as backbone by default, with **mit\_b2, mit\_b3, and Swin-Tiny** also evaluated. Training uses **AdamW**, with **pretraining lr = $6 \times 10^{-5}$** for **150 epochs on YouTube-VOS**, and **fine-tuning lr = $1 \times 10^{-4}$** with **CosineAnnealingLR until convergence**. The experiments use **4 × NVIDIA RTX 4090**, and the memory parameters are **$T=5$** and **$k=1$** for both training and inference. The paper reports **$\approx 26.2$ FPS** and **$\approx 60.8$M parameters** for HMHI-Net with **mit\_b1**, compared with **$\approx 36.7$M** for the baseline and **$\approx 47.7$M** for Multi-Mem; **batch size and FLOPs are not reported** [2507.22465].

## 6. Benchmarks, quantitative results, and ablations

HMHI-Net is evaluated on both **UVOS** and **video saliency detection (VSOD)** benchmarks. For UVOS, the metrics are **region similarity $J$ (IoU)**, **boundary $F$**, and their average **$J\&F$**. For VSOD, the metrics are **MAE**, **max F-measure $F_m$**, **enhanced alignment $E_m$**, and **structure measure $S_m$** [2507.22465].

### UVOS results

| Dataset | Metric | Result |
|---|---|---|
| DAVIS-16 | $J\&F$ / $J$ / $F$ | 89.8% / 88.6% / 91.0% |
| FBMS | $J$ | 86.9% |
| YouTube-Objects | $J$ | 76.2% |

The model is reported to achieve **state-of-the-art** UVOS performance, with improvements versus recent state of the art of approximately **+1.6% $J\&F$ on DAVIS-16**, **+3.5% $J$ on FBMS**, and **+1.5% $J$ on YouTube-Objects** [2507.22465].

### VSOD results

| Dataset | Metrics | Result |
|---|---|---|
| DAVSOD | MAE / $F_m$ / $E_m$ / $S_m$ | 0.054 / 0.801 / 0.896 / 0.847 |
| DAVIS-16 | MAE / $F_m$ / $E_m$ / $S_m$ | 0.009 / 0.947 / 0.990 / 0.951 |
| ViSal | MAE / $F_m$ / $E_m$ / $S_m$ | 0.012 / 0.962 / 0.991 / 0.960 |
| FBMS | MAE / $F_m$ / $E_m$ / $S_m$ | 0.030 / 0.946 / 0.977 / 0.930 |

Across these datasets, HMHI-Net achieves **best or runner-up scores** [2507.22465].

The ablation results are important because they directly support the paper’s design claims. Memory layer selection shows the **largest gain from shallow layer-2 memory (+0.8% $J\&F$)**, followed by a **modest gain from layer-3 (+0.6%)**, while **layer-4 only** yields only a **marginal gain (+0.2%)**. This is presented as empirical support for the necessity of shallow memory. In module ablations, **hierarchical memory alone** outperforms **single-level memory**; **PLAM (S2H)** adds **+1.0%** on DAVIS-16, **SGIM (H2S)** adds **+1.3%**, and the full HMHI-Net adds **+1.4%**. The paper further reports that **swapping modules**, thereby violating the intended heterogeneity, **degrades performance** [2507.22465].

Robustness is also reported along several axes. HMHI-Net yields **consistent gains** across **mit\_b1, mit\_b2, mit\_b3, and Swin-Tiny**, and using **both image and flow** performs best, although even **single-modality input** benefits from HMHI-Net relative to the baseline. This suggests that the claimed contribution is not restricted to a single backbone choice or to the two-stream setting alone.

## 7. Relation to prior work, limitations, and prospective extensions

HMHI-Net is situated at the intersection of memory-augmented video segmentation and motion-appearance fusion. In **semi-supervised VOS**, methods such as **STM**, **STCN**, **AOT**, **XMem**, **Cutie**, and **SAM2** are identified as representative memory-based systems that emphasize spatio-temporal memory and, in later variants, memory selection or compression. In **UVOS**, methods such as **TGFormer**, **PMN**, **DPA**, and **GSA** are described as introducing memory but mostly at high semantic levels, which the HMHI-Net paper argues explains their marginal gains. A separate line of work, including **MATNet**, **RTNet**, **HFAN**, **TMO**, and **SimulFlow**, relies on motion-appearance fusion without memory and is described as limited by short-term flow cues [2507.22465].

The paper’s principal claim is therefore a design correction rather than a general endorsement of “more memory.” The point is that UVOS requires memory at the **right representational levels**, coupled through interaction mechanisms that respect heterogeneity. This suggests that the contribution is as much about feature stratification as about temporal storage.

The reported limitations are concrete. The use of **dual banks** and **bidirectional interactions** adds **computation and storage overhead**, and the authors identify **token pruning** and **memory compression** as possible efficiency directions. Failure cases remain in **extremely complex scenes with rapid scale changes or severe occlusions**, where cross-resolution alignment may still be difficult. Proposed extensions include generalization to **other backbones and tasks**, including **semi-/fully supervised VOS** and **video saliency**, as well as the integration of richer unsupervised signals such as **self-supervised contrastive objectives** or **entropy regularization** [2507.22465].

A common source of confusion is nominal rather than conceptual: HMHI-Net refers specifically to the UVOS model introduced in **“Shallow Features Matter: Hierarchical Memory with Heterogeneous Interaction for Unsupervised Video Object Segmentation”** [2507.22465]. It is distinct from similarly named systems in unrelated domains. Within video segmentation, its defining proposition is that **shallow features are not auxiliary cues but indispensable memory elements** when dense object masks must be inferred without annotation-derived priors.

Source: https://www.emergentmind.com/topics/hmhi-net