---
title: 'HieraMamba: Hierarchical Video Grounding'
url: https://www.emergentmind.com/topics/hieramamba
type: topic
---

# HieraMamba: Hierarchical Video Grounding

HieraMamba is a hierarchical state-space architecture for **video temporal grounding**, also called **natural language query localization** in untrimmed video. In its canonical form, the model takes video features \(V = \{v_i\}_{i=1}^{L_V} \in \mathbb{R}^{L_V \times D_v}\) and query word features \(Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}\), and learns a mapping \(f(V,Q)\rightarrow (t_s,t_e)\) that predicts the start and end timestamps of the moment answering the query. Its defining contribution is a learned multi-scale temporal hierarchy built with **Anchor-MambaPooling (AMP)** blocks, which summarize local windows into compact anchor tokens while preserving fine temporal detail and linear-time sequence processing via Mamba-style selective scanning [2510.23043].

## 1. Problem setting and conceptual scope

HieraMamba addresses a specific long-video grounding problem: localizing the temporal span in an untrimmed video that matches a natural-language query. The stated difficulty is dual. First, long videos require **long-range temporal reasoning** over sequences that may span minutes to hours. Second, grounding queries vary in temporal granularity: some refer to broad events such as “prepping ingredients,” while others refer to fine events such as “stove on.” The model is therefore designed to preserve both broad context and temporally faithful local evidence [2510.23043].

The method is positioned against four common compromises in prior long-video grounding systems: **naive downsampling**, **fixed-length pooling**, **fixed sliding windows**, and **quadratic-cost Transformer attention**. The paper presents HieraMamba as avoiding all four. Rather than forming a coarse representation by blind subsampling, it constructs a temporal pyramid through learned compression. This suggests that the model’s central claim is not merely improved efficiency, but a particular balance between scalability and temporal fidelity.

The name “HieraMamba” is sometimes conflated with other hierarchical Mamba variants in vision, language, or time series. In the literature provided here, however, **HieraMamba** refers specifically to the video temporal grounding architecture based on AMP blocks and contrastive supervision, not to image super-resolution, deraining, stock forecasting, or hyperbolic language embeddings [2510.23043].

## 2. Hierarchical temporal representation and AMP blocks

The core representation is a multi-scale temporal pyramid
\[
\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.
\]
At layer \(l\), an AMP block takes \(A^{(l)}\) and outputs a refined same-level sequence \(\tilde V^{(l)}\) together with next-level anchors \(A^{(l+1)}\). For the first layer, the paper writes \(A^{(0)} = V^{(0)}\in\mathbb{R}^{L_0\times D_v}\), produces \(\tilde V^{(0)}\in\mathbb{R}^{L_0\times D_v}\), and forms \(A^{(1)}\in\mathbb{R}^{L_1\times D_v}\) with \(L_1=\lceil L_0/s\rceil\), where \(s\) is the stride [2510.23043].

Anchor construction begins by creating one anchor every \(s\) frames. Given \(V^{(0)}\in\mathbb{R}^{L\times D}\), the model forms
\[
A\in\mathbb{R}^{M\times D},\qquad M=\lfloor L/s\rfloor.
\]
Each anchor is initialized by pooling over its local window of \(s\) frames. The supplementary ablation evaluates **mean pooling**, **max pooling**, **attention pooling**, and **gated pooling**, and reports that **mean pooling** works best. The resulting anchor tokens are not processed separately. Instead, they are interleaved with frame tokens:
\[
\hat V = [a_0,\,v_0,\dots,v_{s-1},\,a_1,\,v_s,\dots,v_{2s-1},\,\dots] \in \mathbb{R}^{(L+M)\times D}.
\]
The paper states that this deterministic layout preserves temporal order, allows anchors to broadcast coarse context to nearby frames, and allows frames to refine anchors during the scan.

Within AMP, global modeling is handled by **Hydra**, described as a bidirectional Mamba-style model, while local precision is handled by a lightweight local Transformer with a representative window size of \(5\). This produces an explicit division of labor: bidirectional selective scanning captures long-range context in linear time, and narrow-window attention captures fine local patterns. RMSNorm, residual connections, and a learnable sigmoid gate are applied across the global encoding, local encoding, and FFN substages. The refined representation then yields both the next-level anchors and the same-level output retained for grounding [2510.23043].

The hierarchy is bottom-up rather than feature-pyramid-style top-down. The paper does not describe explicit top-down skip fusion across scales. Instead, it propagates anchors upward and later uses all refined outputs \(\{\tilde V^{(l)}\}\) jointly in cross-modal fusion and decoding. A plausible implication is that the hierarchy is intended less as a general multi-resolution backbone and more as a learned compression chain specialized for temporal localization.

## 3. Selective state spaces, hierarchy construction, and grounding head

HieraMamba motivates its sequence modeling with the standard state-space formulation. The continuous-time form is
\[
\frac{d\boldsymbol{h}(t)}{dt} = \mathbf{A}\boldsymbol{h}(t) + \mathbf{B}\boldsymbol{x}(t), \qquad \boldsymbol{y}(t) = \mathbf{C}\boldsymbol{h}(t) + \mathbf{D}\boldsymbol{x}(t),
\]
and the paper gives a Mamba selective scan update as
\[
\tilde{\boldsymbol{y}}_k = \Delta_k \cdot \left( \mathbf{A} \tilde{\boldsymbol{y}}_{k-1} + \mathbf{B}_k \odot \boldsymbol{x}_k \right), \qquad \boldsymbol{y}_k = \mathbf{C}_k \cdot \tilde{\boldsymbol{y}}_k,
\]
where \(\mathbf{B}_k\), \(\mathbf{C}_k\), and \(\Delta_k\) are input-conditioned. In HieraMamba, this selective scanning is instantiated through Hydra so that both forward and backward temporal evidence can influence grounding [2510.23043].

Stacking \(L\) AMP blocks yields the temporal hierarchy. The effective stride may follow
\[
S^{(l)} = s^{\,l-1},
\]
so token counts decrease geometrically with depth. The supplementary specifies **8 AMP layers** for Ego4D and TACoS, and **9 AMP layers** for MAD. These depths correspond to the need to represent progressively broader temporal context in long videos.

For language conditioning, the query is first embedded by a frozen text model: a **CLIP text encoder** for Ego4D and MAD, or **300-d GloVe** for TACoS. A stack of standard Transformers refines the text embeddings to \(E\in\mathbb{R}^{L_Q\times D_q}\). The multi-scale video pyramid and text features are then fused using cross-modal attention,
\[
X_{\text{fused}} = \text{CrossAttention}(\{\tilde V^{(l)}\}_{l=0}^{L-1}, E).
\]

Temporal localization is performed by a lightweight convolutional decoder following ActionFormer-style localization. At each level \(l\) and token \(t\), the model predicts a confidence score \(p_t^{(l)}\) and normalized boundary offsets \(\boldsymbol{\delta}_t^{(l)}=(\delta^s,\delta^e)\), with each head implemented by three 1D convolutions. Given effective stride \(S^{(l)}\), each token yields
\[
\hat{\mathbf{y}} = \bigl(S^{(l)}(t - \delta^s),\; S^{(l)}(t + \delta^e)\bigr).
\]
All proposals from all times and levels are ranked by confidence and merged with **Soft-NMS**, after which the final output is the top-\(k\) predicted moments [2510.23043].

## 4. Training objectives and representation shaping

HieraMamba is trained with standard dense localization losses together with two contrastive objectives:
\[
\mathcal{L} = \mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{reg}} + \mathcal{L}_{\text{contrast}},
\]
where \(\mathcal{L}_{\text{cls}}\) uses **Focal Loss** and \(\mathcal{L}_{\text{reg}}\) uses **Distance-IoU loss**. The contrastive term combines **Anchor-Conditioned Contrastive (ACC)** and **Segment-Pooled Contrastive (SPC)** [2510.23043].

ACC is self-supervised and applied at every hierarchy level. For anchor \(\boldsymbol{a}_i^{(l+1)}\), the positive set is all \(s\) refined tokens inside its temporal window,
\[
\mathcal{P}_i^{(l)} = \{\tilde{\boldsymbol{v}}_t^{(l)} \mid t \in [is,\, is+s)\},
\]
and the loss is a multi-positive InfoNCE:
\[
\mathcal{L}_{\mathrm{acc}\!\bigl(\boldsymbol{a}_i^{(l+1)}\bigr)} =
-\log
\frac{
\displaystyle \sum_{\boldsymbol{p} \in \mathcal{P}_i^{(l)}} \exp\!\bigl(\boldsymbol{a}_i^{(l+1)} \cdot \boldsymbol{p} \,/\, \tau\bigr)
}{
\displaystyle \sum_{\boldsymbol{c} \in \mathcal{P}_i^{(l)} \cup \mathcal{N}_i^{(l)}} \exp\!\bigl(\boldsymbol{a}_i^{(l+1)} \cdot \boldsymbol{c} \,/\, \tau\bigr)
}.
\]
Aggregated across anchors and layers,
\[
\mathcal{L}_{\text{ACC}} = \sum_{l=0}^{L-1} \sum_i \mathcal{L}_{\text{acc}(\boldsymbol{a}_i^{(l+1)})}.
\]
The stated intuition is that anchors should summarize their own windows while remaining distinct from distant unrelated windows.

SPC is supervised and uses ground-truth segments. For a segment \(g_m=[t_{\text{start}},t_{\text{end}})\) at layer \(l\), the tokens inside the interval are pooled into a prototype
\[
\boldsymbol{z}_{\text{seg}^{(l)}} = \operatorname{Pool}\!\bigl\{\tilde{\boldsymbol{v}}_t^{(l)} \;\big\vert\; t\in g_m\bigr\},
\]
using mean pooling. The corresponding loss is
\[
\mathcal{L}_{\mathrm{spc}^{(l)}\!\bigl(\boldsymbol{z}_{\mathrm{seg}^{(l)}}\bigr)} =
-\log
\frac{
\displaystyle \sum_{\boldsymbol{p} \in \mathcal{P}_{\mathrm{seg}^{(l)}}} \exp\!\bigl(\boldsymbol{z}_{\mathrm{seg}^{(l)}} \cdot \boldsymbol{p} \,/\, \tau\bigr)
}{
\displaystyle \sum_{\boldsymbol{c} \in \mathcal{P}_{\mathrm{seg}^{(l)}} \cup \mathcal{N}_{\mathrm{seg}^{(l)}}} \exp\!\bigl(\boldsymbol{z}_{\mathrm{seg}^{(l)}} \cdot \boldsymbol{c} \,/\, \tau\bigr)
}.
\]
Aggregated across levels,
\[
\mathcal{L}_{\text{SPC}} = \sum_{l=0}^{L-1} \mathcal{L}_{\text{spc}^{(l)}}.
\]

The joint contrastive objective is
\[
\mathcal{L}_{\text{contrast}} = \lambda_{\text{ACC}}\mathcal{L}_{\text{ACC}} + \lambda_{\text{SPC}}\mathcal{L}_{\text{SPC}}.
\]
Dataset-specific weights are reported as \((10,1)\) for Ego4D, \((1,0.1)\) for TACoS, and \((0.5,0.6)\) for MAD. The paper’s interpretation is that **ACC** enforces hierarchy consistency and local structural supervision, whereas **SPC** supplies semantic supervision aligned with annotated moments.

## 5. Empirical performance, efficiency, and ablations

HieraMamba reports standard **Recall@k at tIoU threshold \(\theta\)** with \(k\in\{1,5\}\) and \(\theta\in\{0.3,0.5\}\), and defines average recall as the average of R@1 IoU 0.3, R@1 IoU 0.5, R@5 IoU 0.3, and R@5 IoU 0.5. On **Ego4D-NLQ**, it achieves **18.81** for R@1@0.3, **13.04** for R@1@0.5, **40.82** for R@5@0.3, **29.96** for R@5@0.5, and **25.66** average recall. The paper contrasts this with table values of **24.44** for DeCafNet, **23.08** for SnAG, and **21.81** for RGNet, yielding gains of **+1.22**, **+2.58**, and **+3.85**, respectively [2510.23043].

On **MAD-v1**, HieraMamba reports **11.26 / 7.22 / 23.49 / 16.81** with **14.70** average recall, compared with **14.46** for DeCafNet. On **MAD-v2**, it reports **14.72**, **9.00**, **28.50**, **19.97**, and **18.05** average recall, exceeding **15.25** for SnAG and **14.87** for RGNet. On **TACoS**, it achieves **59.59**, **48.99**, **83.75**, **74.28**, and **66.65** average recall, compared with **64.96** average recall for OSGNet. The paper emphasizes MAD as a particularly strong test because the videos are hour-scale.

The efficiency analysis is reported on **MAD-v2**, where videos average around 100 minutes. The paper states that HieraMamba achieves the **highest accuracy** while requiring roughly **2.5× fewer FLOPs** than SnAG (Global), and that SnAG (Global) only marginally improves over SnAG (Local) while using nearly **3×** the computation. This is presented as evidence that learned hierarchical compression can improve the accuracy-efficiency frontier for long-video grounding [2510.23043].

Ablations clarify which components are responsible for the reported behavior. On Ego4D-NLQ, removing interleaving reduces average recall from **25.66** to **24.40**; removing bidirectional scan drops it to **23.29**; removing local encoding drops it to **24.63**; and removing gates drops it to **24.80**. Contrastive ablations report **24.68** without ACC or SPC, **25.22** with ACC only, **25.23** with SPC only, and **25.66** with both. Supplementary results show **mean pooling 24.68**, **max pooling 24.65**, **attention pooling 24.46**, and **gated pooling 24.37**. Another supplementary result shows **pooled SPC 25.23** versus **unpooled SPC 24.05**, which the paper interprets as support for segment-level pooling rather than forcing all in-segment tokens into one identical cluster.

## 6. Nomenclature, related “hierarchical Mamba” models, and recurrent misconceptions

A recurring source of confusion is that “HieraMamba” is often used loosely to refer to any Mamba model with some hierarchical element. The literature here shows that this is inaccurate. **Hi-Mamba** is a hierarchical Mamba for efficient **single-image super-resolution** based on a **Hierarchical Mamba Block (HMB)** and a **Direction Alternation Hierarchical Mamba Group (DA-HMG)**; it is not a video grounding model [2410.10140]. **RAMba** augments Mamba with **Hierarchical Sparse Attention (HSA)** for long-context random access; despite the “hierarchical” terminology, it is not named HieraMamba and is aimed at long-context retrieval rather than video grounding [2504.16795].

Other works use “hierarchical Mamba” for still different problems. **Hierarchical Mamba (HiM)** combines a Mamba2 backbone with **Poincaré** or **Lorentz** hyperbolic embeddings for hierarchy-aware language representations, focusing on ontological reasoning rather than temporal localization [2505.18973]. **HIGSTM** denotes **Hierarchical Information-Guided Spatio-Temporal Mamba** for stock forecasting, with index-guided decomposition and dynamic/global graph learning [2503.11387]. **MPHM** denotes **Multi-Prior Hierarchical Mamba** for image deraining, where a U-shaped restoration backbone uses a **Hierarchical Mamba Module (HMM)** and progressive prior injection [2511.13113]. **HyMamba** is a hyperspectral tracking model with cross-depth and temporal spectral-state propagation, but it is not a Hiera-style multiscale backbone [2509.08265].

These distinctions matter because the term “hierarchical” is used in multiple senses across the Mamba literature: multi-scale token pyramids, cross-block directional scheduling, graph-structured aggregation, hyperbolic output geometry, cross-depth recurrence, or sparse chunk retrieval. In the strict sense defined by the paper bearing the exact name, **HieraMamba** denotes a long-video temporal grounding system whose hierarchy is built from anchor-based learned temporal compression rather than from a generic vision or language backbone [2510.23043].

## 7. Limitations, transferability, and historical significance

The HieraMamba paper explicitly notes two limitations. First, it uses **frozen video backbones**, which make the system modular and efficient but prevent end-to-end optimization of visual representations for grounding. Second, anchor generation uses a **fixed temporal stride**, so anchors are created at a fixed cadence rather than adaptively based on content. The authors propose adaptive anchor allocation and end-to-end backbone training as future directions [2510.23043].

The broader significance of HieraMamba lies in how it reframes hierarchical sequence modeling for video. Instead of relying on heavy global attention or aggressive downsampling, it forms a representation hierarchy through learned anchor summaries refined by bidirectional selective scanning and local attention. This suggests a specific design principle: coarse context need not be obtained by discarding temporal structure, provided the coarse units are themselves learned and contrastively constrained.

A related historical point is that hierarchical Mamba variants have proliferated rapidly across domains, but they differ sharply in what is meant by hierarchy. The super-resolution line emphasizes **scale hierarchy and directional scheduling** [2410.10140]; long-context language modeling emphasizes **two-level chunk retrieval** [2504.16795]; structured language embeddings emphasize **hierarchy in output geometry** [2505.18973]; deraining emphasizes **network-stage and intra-block hierarchy** [2511.13113]. Against this background, HieraMamba occupies a distinct position: it is a hierarchical temporal encoder specialized for long-video grounding, and its main innovation is the combination of **Anchor-MambaPooling**, **interleaved anchor-frame scanning**, and **ACC/SPC supervision** to preserve both temporal fidelity and long-range context [2510.23043].

Source: https://www.emergentmind.com/topics/hieramamba