HieraMamba: Hierarchical Video Grounding

• HieraMamba is a hierarchical video temporal grounding model that uses multi-scale AMP blocks to compress and refine temporal features.
• It employs interleaved anchor-frame scanning and bidirectional selective scanning to preserve fine local details and long-range context.
• The architecture integrates cross-modal attention and dual contrastive objectives to improve localization accuracy and efficiency in long videos.

HieraMamba is a hierarchical state-space architecture for video temporal grounding, also called natural language query localization in untrimmed video. In its canonical form, the model takes video features $$V = \{v_i\}_{i=1}^{L_V} \in \mathbb{R}^{L_V \times D_v}$$ and query word features $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$, and learns a mapping $f(V,Q)\rightarrow (t_s,t_e)$ that predicts the start and end timestamps of the moment answering the query. Its defining contribution is a learned multi-scale temporal hierarchy built with Anchor-MambaPooling (AMP) blocks, which summarize local windows into compact anchor tokens while preserving fine temporal detail and linear-time sequence processing via Mamba-style selective scanning (An et al., 27 Oct 2025).

1. Problem setting and conceptual scope

HieraMamba addresses a specific long-video grounding problem: localizing the temporal span in an untrimmed video that matches a natural-language query. The stated difficulty is dual. First, long videos require long-range temporal reasoning over sequences that may span minutes to hours. Second, grounding queries vary in temporal granularity: some refer to broad events such as “prepping ingredients,” while others refer to fine events such as “stove on.” The model is therefore designed to preserve both broad context and temporally faithful local evidence (An et al., 27 Oct 2025).

The method is positioned against four common compromises in prior long-video grounding systems: naive downsampling, fixed-length pooling, fixed sliding windows, and quadratic-cost Transformer attention. The paper presents HieraMamba as avoiding all four. Rather than forming a coarse representation by blind subsampling, it constructs a temporal pyramid through learned compression. This suggests that the model’s central claim is not merely improved efficiency, but a particular balance between scalability and temporal fidelity.

The name “HieraMamba” is sometimes conflated with other hierarchical Mamba variants in vision, language, or time series. In the literature provided here, however, HieraMamba refers specifically to the video temporal grounding architecture based on AMP blocks and contrastive supervision, not to image super-resolution, deraining, stock forecasting, or hyperbolic language embeddings (An et al., 27 Oct 2025).

2. Hierarchical temporal representation and AMP blocks

The core representation is a multi-scale temporal pyramid

$$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$

At layer $l$, an AMP block takes $A^{(l)}$ and outputs a refined same-level sequence $\tilde V^{(l)}$ together with next-level anchors $A^{(l+1)}$. For the first layer, the paper writes $A^{(0)} = V^{(0)}\in\mathbb{R}^{L_0\times D_v}$, produces $\tilde V^{(0)}\in\mathbb{R}^{L_0\times D_v}$, and forms $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$0 with $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$1, where $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$2 is the stride (An et al., 27 Oct 2025).

Anchor construction begins by creating one anchor every $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$3 frames. Given $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$4, the model forms

$$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$5

Each anchor is initialized by pooling over its local window of $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$6 frames. The supplementary ablation evaluates mean pooling, max pooling, attention pooling, and gated pooling, and reports that mean pooling works best. The resulting anchor tokens are not processed separately. Instead, they are interleaved with frame tokens: $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$7 The paper states that this deterministic layout preserves temporal order, allows anchors to broadcast coarse context to nearby frames, and allows frames to refine anchors during the scan.

Within AMP, global modeling is handled by Hydra, described as a bidirectional Mamba-style model, while local precision is handled by a lightweight local Transformer with a representative window size of $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$8. This produces an explicit division of labor: bidirectional selective scanning captures long-range context in linear time, and narrow-window attention captures fine local patterns. RMSNorm, residual connections, and a learnable sigmoid gate are applied across the global encoding, local encoding, and FFN substages. The refined representation then yields both the next-level anchors and the same-level output retained for grounding (An et al., 27 Oct 2025).

The hierarchy is bottom-up rather than feature-pyramid-style top-down. The paper does not describe explicit top-down skip fusion across scales. Instead, it propagates anchors upward and later uses all refined outputs $$Q = \{w_j\}_{j=1}^{L_Q} \in \mathbb{R}^{L_Q \times D_q}$$9 jointly in cross-modal fusion and decoding. A plausible implication is that the hierarchy is intended less as a general multi-resolution backbone and more as a learned compression chain specialized for temporal localization.

3. Selective state spaces, hierarchy construction, and grounding head

HieraMamba motivates its sequence modeling with the standard state-space formulation. The continuous-time form is

$f(V,Q)\rightarrow (t_s,t_e)$0

and the paper gives a Mamba selective scan update as

$f(V,Q)\rightarrow (t_s,t_e)$1

where $f(V,Q)\rightarrow (t_s,t_e)$2, $f(V,Q)\rightarrow (t_s,t_e)$3, and $f(V,Q)\rightarrow (t_s,t_e)$4 are input-conditioned. In HieraMamba, this selective scanning is instantiated through Hydra so that both forward and backward temporal evidence can influence grounding (An et al., 27 Oct 2025).

Stacking $f(V,Q)\rightarrow (t_s,t_e)$5 AMP blocks yields the temporal hierarchy. The effective stride may follow

$f(V,Q)\rightarrow (t_s,t_e)$6

so token counts decrease geometrically with depth. The supplementary specifies 8 AMP layers for Ego4D and TACoS, and 9 AMP layers for MAD. These depths correspond to the need to represent progressively broader temporal context in long videos.

For language conditioning, the query is first embedded by a frozen text model: a CLIP text encoder for Ego4D and MAD, or 300-d GloVe for TACoS. A stack of standard Transformers refines the text embeddings to $f(V,Q)\rightarrow (t_s,t_e)$7. The multi-scale video pyramid and text features are then fused using cross-modal attention,

$f(V,Q)\rightarrow (t_s,t_e)$8

Temporal localization is performed by a lightweight convolutional decoder following ActionFormer-style localization. At each level $f(V,Q)\rightarrow (t_s,t_e)$9 and token $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$0, the model predicts a confidence score $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$1 and normalized boundary offsets $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$2, with each head implemented by three 1D convolutions. Given effective stride $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$3, each token yields

$$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$4

All proposals from all times and levels are ranked by confidence and merged with Soft-NMS, after which the final output is the top-$$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$5 predicted moments (An et al., 27 Oct 2025).

4. Training objectives and representation shaping

HieraMamba is trained with standard dense localization losses together with two contrastive objectives: $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$6 where $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$7 uses Focal Loss and $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$8 uses Distance-IoU loss. The contrastive term combines Anchor-Conditioned Contrastive (ACC) and Segment-Pooled Contrastive (SPC) (An et al., 27 Oct 2025).

ACC is self-supervised and applied at every hierarchy level. For anchor $$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$9, the positive set is all $l$0 refined tokens inside its temporal window,

$l$1

and the loss is a multi-positive InfoNCE: $l$2 Aggregated across anchors and layers,

$l$3

The stated intuition is that anchors should summarize their own windows while remaining distinct from distant unrelated windows.

SPC is supervised and uses ground-truth segments. For a segment $l$4 at layer $l$5, the tokens inside the interval are pooled into a prototype

$l$6

using mean pooling. The corresponding loss is

$l$7

Aggregated across levels,

$l$8

The joint contrastive objective is

$l$9

Dataset-specific weights are reported as $A^{(l)}$0 for Ego4D, $A^{(l)}$1 for TACoS, and $A^{(l)}$2 for MAD. The paper’s interpretation is that ACC enforces hierarchy consistency and local structural supervision, whereas SPC supplies semantic supervision aligned with annotated moments.

5. Empirical performance, efficiency, and ablations

HieraMamba reports standard Recall@k at tIoU threshold $A^{(l)}$3 with $A^{(l)}$4 and $A^{(l)}$5, and defines average recall as the average of R@1 IoU 0.3, R@1 IoU 0.5, R@5 IoU 0.3, and R@5 IoU 0.5. On Ego4D-NLQ, it achieves 18.81 for R@[email protected], 13.04 for R@[email protected], 40.82 for R@[email protected], 29.96 for R@[email protected], and 25.66 average recall. The paper contrasts this with table values of 24.44 for DeCafNet, 23.08 for SnAG, and 21.81 for RGNet, yielding gains of +1.22, +2.58, and +3.85, respectively (An et al., 27 Oct 2025).

On MAD-v1, HieraMamba reports 11.26 / 7.22 / 23.49 / 16.81 with 14.70 average recall, compared with 14.46 for DeCafNet. On MAD-v2, it reports 14.72, 9.00, 28.50, 19.97, and 18.05 average recall, exceeding 15.25 for SnAG and 14.87 for RGNet. On TACoS, it achieves 59.59, 48.99, 83.75, 74.28, and 66.65 average recall, compared with 64.96 average recall for OSGNet. The paper emphasizes MAD as a particularly strong test because the videos are hour-scale.

The efficiency analysis is reported on MAD-v2, where videos average around 100 minutes. The paper states that HieraMamba achieves the highest accuracy while requiring roughly 2.5× fewer FLOPs than SnAG (Global), and that SnAG (Global) only marginally improves over SnAG (Local) while using nearly 3× the computation. This is presented as evidence that learned hierarchical compression can improve the accuracy-efficiency frontier for long-video grounding (An et al., 27 Oct 2025).

Ablations clarify which components are responsible for the reported behavior. On Ego4D-NLQ, removing interleaving reduces average recall from 25.66 to 24.40; removing bidirectional scan drops it to 23.29; removing local encoding drops it to 24.63; and removing gates drops it to 24.80. Contrastive ablations report 24.68 without ACC or SPC, 25.22 with ACC only, 25.23 with SPC only, and 25.66 with both. Supplementary results show mean pooling 24.68, max pooling 24.65, attention pooling 24.46, and gated pooling 24.37. Another supplementary result shows pooled SPC 25.23 versus unpooled SPC 24.05, which the paper interprets as support for segment-level pooling rather than forcing all in-segment tokens into one identical cluster.

6. Nomenclature, related “hierarchical Mamba” models, and recurrent misconceptions

A recurring source of confusion is that “HieraMamba” is often used loosely to refer to any Mamba model with some hierarchical element. The literature here shows that this is inaccurate. Hi-Mamba is a hierarchical Mamba for efficient single-image super-resolution based on a Hierarchical Mamba Block (HMB) and a Direction Alternation Hierarchical Mamba Group (DA-HMG); it is not a video grounding model (Qiao et al., 2024). RAMba augments Mamba with Hierarchical Sparse Attention (HSA) for long-context random access; despite the “hierarchical” terminology, it is not named HieraMamba and is aimed at long-context retrieval rather than video grounding (Hu et al., 23 Apr 2025).

Other works use “hierarchical Mamba” for still different problems. Hierarchical Mamba (HiM) combines a Mamba2 backbone with Poincaré or Lorentz hyperbolic embeddings for hierarchy-aware language representations, focusing on ontological reasoning rather than temporal localization (Patil et al., 25 May 2025). HIGSTM denotes Hierarchical Information-Guided Spatio-Temporal Mamba for stock forecasting, with index-guided decomposition and dynamic/global graph learning (Yan et al., 14 Mar 2025). MPHM denotes Multi-Prior Hierarchical Mamba for image deraining, where a U-shaped restoration backbone uses a Hierarchical Mamba Module (HMM) and progressive prior injection (Yu et al., 17 Nov 2025). HyMamba is a hyperspectral tracking model with cross-depth and temporal spectral-state propagation, but it is not a Hiera-style multiscale backbone (Gao et al., 10 Sep 2025).

These distinctions matter because the term “hierarchical” is used in multiple senses across the Mamba literature: multi-scale token pyramids, cross-block directional scheduling, graph-structured aggregation, hyperbolic output geometry, cross-depth recurrence, or sparse chunk retrieval. In the strict sense defined by the paper bearing the exact name, HieraMamba denotes a long-video temporal grounding system whose hierarchy is built from anchor-based learned temporal compression rather than from a generic vision or language backbone (An et al., 27 Oct 2025).

7. Limitations, transferability, and historical significance

The HieraMamba paper explicitly notes two limitations. First, it uses frozen video backbones, which make the system modular and efficient but prevent end-to-end optimization of visual representations for grounding. Second, anchor generation uses a fixed temporal stride, so anchors are created at a fixed cadence rather than adaptively based on content. The authors propose adaptive anchor allocation and end-to-end backbone training as future directions (An et al., 27 Oct 2025).

The broader significance of HieraMamba lies in how it reframes hierarchical sequence modeling for video. Instead of relying on heavy global attention or aggressive downsampling, it forms a representation hierarchy through learned anchor summaries refined by bidirectional selective scanning and local attention. This suggests a specific design principle: coarse context need not be obtained by discarding temporal structure, provided the coarse units are themselves learned and contrastively constrained.

A related historical point is that hierarchical Mamba variants have proliferated rapidly across domains, but they differ sharply in what is meant by hierarchy. The super-resolution line emphasizes scale hierarchy and directional scheduling (Qiao et al., 2024); long-context language modeling emphasizes two-level chunk retrieval (Hu et al., 23 Apr 2025); structured language embeddings emphasize hierarchy in output geometry (Patil et al., 25 May 2025); deraining emphasizes network-stage and intra-block hierarchy (Yu et al., 17 Nov 2025). Against this background, HieraMamba occupies a distinct position: it is a hierarchical temporal encoder specialized for long-video grounding, and its main innovation is the combination of Anchor-MambaPooling, interleaved anchor-frame scanning, and ACC/SPC supervision to preserve both temporal fidelity and long-range context (An et al., 27 Oct 2025).