Anchor-MambaPooling (AMP)

• AMP is a hierarchical video encoding block that refines fine-grained features while compressing them into coarse anchor tokens for multi-scale temporal analysis.
• It employs selective state-space scanning and interleaved anchor-frame processing to maintain temporal fidelity and reduce computational complexity.
• The design leverages dual contrastive losses and layered processing to optimize local detail retention and global event discrimination for video temporal grounding.

Anchor-MambaPooling (AMP) is the core hierarchical video encoding block in HieraMamba. It is introduced for video temporal grounding in long untrimmed videos, with the stated purpose to simultaneously refine video features at the current temporal resolution and compress them into a smaller set of anchor tokens that are passed to the next, coarser level. The design constructs a hierarchical multi-scale representation of a video stream by using Mamba-style selective state-space scanning over an interleaved sequence of fine-grained tokens and coarse anchor tokens. In the nearby design space, Attention Mamba provides a pooling-based compression mechanism inside attention for a Mamba-centered time-series model, but it does not present an anchor-token method in the usual sense (An et al., 27 Oct 2025, Xiong et al., 2 Apr 2025).

1. Problem setting and design rationale

AMP is motivated by two coupled difficulties in video temporal grounding: long videos require long-range temporal reasoning, and queries demand different temporal granularities. The method is introduced against a background in which standard approaches are described as often failing by naively downsampling and losing temporal evidence, using fixed-length pooling that compresses indiscriminately, relying on fixed sliding windows that can fragment events at window boundaries, or using transformer attention whose quadratic cost forces aggressive resolution reduction. AMP is presented as a way to form a feature pyramid “learned through token-level compression rather than naive downsampling” (An et al., 27 Oct 2025).

The method is explicitly positioned against several standard temporal reduction strategies. Versus standard temporal downsampling, AMP does not just drop frames; it creates anchor tokens that summarize windows and then lets those anchors interact with their constituent frames through selective scan. Versus fixed-window pooling, anchor formation is initialized from local windows but then refined through bidirectional Mamba-style scanning and local refinement. Versus strided convolutions, the reduction is not purely mechanical, because AMP uses explicit summary tokens and retains both refined same-scale features and compressed next-scale anchors. Versus transformer token compression, the central claim is linear-time complexity through Mamba/Hydra selective scan.

This suggests that AMP should be understood less as a simple pooling operator than as a hierarchical compression-and-refinement mechanism. A plausible implication is that its contribution lies in preserving temporal fidelity and semantic richness while still producing a usable multiscale representation for downstream localization.

2. Block structure and hierarchical representation

For the initial video feature sequence,

$V^{(0)} = V \in \mathbb{R}^{L_0 \times D_v},$

the paper sets

$A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$

An AMP block outputs a refined sequence at the same scale and an anchor set for the next level. For the first block, these are

$$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$

with

$L_1 = \lceil L_0 / s \rceil.$

For any layer $l>0$, the block receives $A^{(l)}$ and returns refined tokens $\tilde V^{(l)}$ and further-downsampled anchors $A^{(l+1)}$. Stacking AMP blocks forms the pyramid

$$\mathcal{V}_{\text{pyr}} = \{\tilde V^{(0)},\, \tilde V^{(1)},\, \dots,\, \tilde V^{(L-1)}\}.$$

Each $\tilde V^{(l)}$ corresponds to a different temporal granularity (An et al., 27 Oct 2025).

Anchor tokens are the compressed summary tokens produced inside AMP. Given first-level features $A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$0 and temporal stride $A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$1, the method instantiates one anchor every $A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$2 frames, yielding $A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$3 with $A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$4. Each anchor is initialized by pooling over its local window of $A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$5 frames. The supplementary evaluates mean pooling, max pooling, attention pooling, and gated pooling; mean pooling performs best. The stated reason anchors preserve semantic richness and temporal structure is twofold: they are refined by the same selective scan as neighboring fine-grained tokens, and they are placed deterministically in temporal order immediately before the frames they summarize.

A distinctive design choice is interleaving anchors with fine-grained tokens into a single ordered sequence:

$A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$6

This layout is described as maintaining temporal order and enabling bidirectional information flow: anchors can broadcast coarse context to nearby frames, and frames can refine anchors during scanning.

The interleaved sequence is processed by a global encoder based on Hydra, adopted as a bidirectional Mamba-like state-space model. After global encoding, AMP adds a lightweight local encoder: a windowed Transformer with a narrow attention window. The block further includes RMSNorm before each substage, residual connections, a learnable sigmoid gate for feature fusion, and a final feed-forward network. The gate is used instead of unconditional residual addition so the model can adaptively decide how much information to propagate as representations become more abstract. The hierarchy is built recursively, with actual stacks of 8 layers for Ego4D, 8 layers for TACoS, and 9 layers for MAD.

3. Mathematical formulation

AMP is presented within a state-space framework. The paper first gives the classical continuous-time state-space model:

$A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$7

After discretization, it gives

$A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$8

For Mamba’s selective scan, the formulation is

$A^{(0)} = V^{(0)} \in \mathbb{R}^{L_0 \times D_v}.$9

Here $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$0 is the input token at step $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$1, $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$2 is the internal state or output accumulator, $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$3 and $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$4 are input-conditioned modulation terms, and $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$5 is the token-dependent step size (An et al., 27 Oct 2025).

Within AMP itself, the principal structural equations are the anchor-generation and hierarchy equations. Given $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$6 and stride $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$7, the method creates $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$8 anchors $$\tilde V^{(0)} \in \mathbb{R}^{L_0 \times D_v}, \qquad A^{(1)} \in \mathbb{R}^{L_1 \times D_v},$$9, each initialized by pooling over its $L_1 = \lceil L_0 / s \rceil.$0-frame local window. The interleaving equation,

$L_1 = \lceil L_0 / s \rceil.$1

is the core sequence construction: it preserves chronology while allowing a single scan to mix coarse summaries and local evidence. The hierarchical outputs,

$L_1 = \lceil L_0 / s \rceil.$2

followed by

$L_1 = \lceil L_0 / s \rceil.$3

formalize the rule that each layer preserves a refined same-scale stream and emits a compressed next-scale stream. No scale is thrown away; all refined scales are kept for grounding.

The paper does not give a closed-form AMP complexity equation, but repeatedly states the key property: linear-time complexity inherited from Mamba/Hydra scans. In this formulation, global temporal reasoning over full sequences is retained without quadratic attention cost.

4. Contrastive objectives for anchor quality

AMP is trained with two contrastive losses that are tied specifically to the hierarchical anchor representations. The first is the Anchor-Conditioned Contrastive (ACC) loss, described as self-supervised and layer-wise. At layer $L_1 = \lceil L_0 / s \rceil.$4, AMP outputs anchors $L_1 = \lceil L_0 / s \rceil.$5 and refined sequence tokens $L_1 = \lceil L_0 / s \rceil.$6. For each anchor $L_1 = \lceil L_0 / s \rceil.$7, the positive set $L_1 = \lceil L_0 / s \rceil.$8 contains all $L_1 = \lceil L_0 / s \rceil.$9 tokens in its own temporal window, and the negative set $l>0$0 is constructed from distant anchors, with a temporal margin to avoid treating nearby semantically similar windows as negatives. The single-anchor loss is

$l>0$1

and the total ACC loss is

$l>0$2

The stated effect is compactness and distinctiveness: an anchor must match all the tokens it summarizes while staying separate from distant unrelated regions (An et al., 27 Oct 2025).

The second objective is the Segment-Pooled Contrastive (SPC) loss, which is supervised by the ground-truth query span. For an annotated segment $l>0$3, the refined tokens inside the segment are pooled into a single segment prototype using mean pooling:

$l>0$4

Positives are all in-segment tokens and negatives are all out-of-segment tokens. The layer-wise SPC objective is

$l>0$5

with

$l>0$6

The paper explicitly argues that pooling is necessary because a ground-truth interval can contain heterogeneous sub-actions; pooling forms a holistic event prototype without forcing every in-segment token to collapse to the exact same point.

The joint contrastive objective is

$l>0$7

and the final training loss is

$l>0$8

The supplementary gives $l>0$9 for Ego4D, $A^{(l)}$0 for TACoS, and $A^{(l)}$1 for MAD. Together, ACC and SPC are described as preserving local detail, global discrimination, and query relevance.

5. Function within HieraMamba and empirical profile

AMP is the video-side hierarchical encoder in HieraMamba. The pipeline is: a frozen video backbone extracts clip-level video features $A^{(l)}$2; a frozen text backbone extracts query token embeddings $A^{(l)}$3; a transformer-based text encoder refines $A^{(l)}$4 into text embeddings $A^{(l)}$5; a stack of $A^{(l)}$6 AMP blocks processes the video into the pyramid $A^{(l)}$7; the multi-scale video pyramid and text features are fused by cross-attention; and a lightweight convolutional decoder predicts temporal localization. The paper does not say anchors are directly used as proposal tokens for final scoring. Rather, the decoder consumes the refined pyramid features, although those features are shaped by anchor interaction throughout AMP (An et al., 27 Oct 2025).

At each scale $A^{(l)}$8, $A^{(l)}$9 is fed to a classification head producing confidence $\tilde V^{(l)}$0 and a regression head producing normalized offsets $\tilde V^{(l)}$1. Each token yields a proposal

$\tilde V^{(l)}$2

and all proposals across scales are ranked and merged with Soft-NMS. This makes the multiscale structure generated by AMP directly relevant to multi-scale localization.

The clearest isolation of AMP comes from the component ablations.

Variant	Avg. Recall	$\tilde V^{(l)}$3
HieraMamba (Full)	25.66	—
w/o Interleaving	24.40	-1.26
w/o Bidirectional Scan	23.29	-2.37
w/o Local Encoding	24.63	-1.03
w/o Gates	24.80	-0.86

These results show that interleaving matters for anchor-frame exchange, bidirectional scan matters most, local encoding helps preserve fine structure, and gates help adaptive fusion. The supplementary anchor-generation ablation reports 24.68 for mean pooling, 24.65 for max pooling, 24.46 for attention pooling, and 24.37 for gated pooling, with mean pooling as the best initialization. The contrastive-loss ablation reports 24.68 with neither ACC nor SPC, 25.22 with ACC only, 25.23 with SPC only, and 25.66 with both; the supplementary further reports 25.23 for pooled SPC and 24.05 for unpooled SPC, while the base model is 24.68.

The efficiency experiment on MAD-v2 reports that HieraMamba improves average recall over SnAG (Global) by +2.52% while requiring roughly 2.5× fewer FLOPs. Benchmark results reported for systems built around AMP are 25.66 average recall on Ego4D-NLQ versus 24.44 for DeCafNet, 18.05 on MAD-v2 versus 15.25 for SnAG and 14.87 for RGNet, and 66.65 on TACoS versus 64.96 for OSGNet. In the paper’s interpretation, these outcomes indicate that the hierarchical anchor-based video encoder improves localization across sparse events, hour-scale movie grounding, and compositional cooking actions.

6. Relation to pooling-based Mamba compression and limitations

A nearby reference point for AMP is Attention Mamba, a hybrid time-series forecasting architecture that combines an Adaptive Pooling block with a bidirectional Mamba block. The similarity is architectural rather than task-level: both designs couple Mamba-based sequence modeling with compressed global representatives. In Attention Mamba, two linear projections first obtain Query and Key; the Adaptive Pooling block computes Weights; the bidirectional Mamba block computes the refined Value; and the final combination is

$\tilde V^{(l)}$4

The pooling mechanism is defined by

$\tilde V^{(l)}$5

with $\tilde V^{(l)}$6, followed by

$\tilde V^{(l)}$7

and

$\tilde V^{(l)}$8

The paper states that both adaptive pooling operators downscale the last two dimensions of Query and Key to a one quarter level, yielding $\tilde V^{(l)}$9. The intended role is to accelerate attention computation while preserving global information (Xiong et al., 2 Apr 2025).

The relation to AMP is strongest at the level of compressed summaries. Attention Mamba is clearly pooling-based and a form of token or feature reduction, but it is not an anchor-token method in the explicit sense. There are no explicit anchor vectors, no learned anchor bank, no top-$A^{(l+1)}$0 token selection, no Nyström-style landmarks, and no adaptive routing among representative tokens. The representatives are obtained by adaptive average and max pooling over $A^{(l+1)}$1 and $A^{(l+1)}$2, then projected back to full size. For this reason, Attention Mamba is better characterized as a pooled-summary Mamba-attention hybrid than an anchor-token Mamba pooling method.

The distinction helps clarify common misconceptions. AMP is not equivalent to naive downsampling, fixed-window pooling, or strided convolution, because it retains refined same-scale features and compressed next-scale anchors while allowing anchor-token and frame-token interaction inside a shared sequence. Conversely, not every Mamba model with pooling is an AMP model. Attention Mamba is conceptually adjacent because it compresses $A^{(l+1)}$3 into global summaries, but its final operation is gating-like rather than anchor-based sequence-to-anchor attention.

The limitations explicitly attached to AMP are a fixed temporal stride for anchor generation and reliance on precomputed clip embeddings from frozen backbones. The authors suggest adaptive anchor generation as future work. The nearby limitation in Attention Mamba is analogous in spirit: despite the title phrase “adaptive pooling,” the pooling ratio is fixed to one quarter, and the authors suggest a dynamic input-dependent adaptive pooling scheme for future work. Taken together, these points indicate that the broader design space around AMP includes both anchor-based hierarchical compression and pooled-summary compression, but only the former uses explicit anchor tokens as the organizing representation.