Adaptive Multiscale Time Pyramid Network

• The module AMTPN provides a multiscale temporal representation by combining TPP, AFF, and TCE to capture short-, mid-, and long-term dependencies in video data.
• The adaptive feature fusion (AFF) dynamically weighs scale-specific features, enabling precise anomaly detection under weak supervision.
• Integration of AMTPN in DAMS enhances spatiotemporal localization, as evidenced by improved AP and AUC performance on XD-Violence and UCF-Crime datasets.

Searching arXiv for the specified papers and closely related work to ground the article. Adaptive Multiscale Time Pyramid Network (AMTPN) is the temporal modeling core used in the Dual-Branch Adaptive Multiscale Spatiotemporal Framework (DAMS) for weakly supervised video anomaly detection. It is introduced as a learnable, multiscale, and context-aware temporal representation module intended to replace rigid, single-scale temporal processing. In DAMS, AMTPN operates on backbone video features before CBAM and classification, and is designed to capture short-, mid-, and long-term temporal dependencies, adaptively reweight the utility of different scales for each video, and enhance the fused representation with long-range temporal context (An et al., 28 Jul 2025).

1. Conceptual role and problem setting

AMTPN is defined in the context of video anomaly detection, where the objective is spatio-temporal localization of abnormal events under weak supervision. The motivating claim is that anomalies are temporally heterogeneous: some are brief and sharp, whereas others evolve slowly over long ranges. A fixed temporal receptive field is therefore treated as insufficient. AMTPN is introduced to address this limitation through multiscale feature extraction, adaptive scale weighting, and contextual enhancement (An et al., 28 Jul 2025).

Within DAMS, the main processing path is summarized as

$$F_{\text{vad}} = \text{ClsHead}\big(\text{CBAM}(\text{AMTPN}(F_{\text{backbone}}))\big).$$

This placement makes AMTPN the first dedicated temporal refinement block after backbone feature extraction. Its output is described as a temporally enriched representation that is multiscale, adaptive, and contextual. The intended effect is improved weakly supervised anomaly localization, especially when abnormality is subtle, temporally irregular, or duration-varying.

The module’s internal information flow is given as

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$

with input

$X \in \mathbb{R}^{B \times C \times T}.$

Here, the temporal dimension $T$ is the axis along which multiscale modeling is performed.

2. Three-stage cascade architecture

AMTPN is explicitly organized as a three-level cascade composed of Temporal Pyramid Pooling (TPP), Adaptive Feature Fusion (AFF), and Temporal Context Enhancement (TCE). The paper characterizes this composition as enabling multigrained representation and dynamically weighted reconstruction of temporal features (An et al., 28 Jul 2025).

Stage	Core operation	Function
TPP	multiscale temporal pooling and Conv1D	extracts features at multiple receptive-field sizes
AFF	lightweight attention over scales	learns dynamic scale weights
TCE	context-enhancing refinement	injects long-range temporal context

Temporal Pyramid Pooling is the multiscale feature extraction stage. It creates a temporal pyramid by generating feature maps at different receptive-field sizes. Small scales are described as capturing fine-grained local changes, while large scales capture broader temporal patterns and long-term semantics. The paper visualizes this as a progression from high-frequency detail at small scales to more abstract semantic activation at larger scales.

Adaptive Feature Fusion is the stage that makes the module adaptive rather than merely multibranch. The paper states that the scale-specific outputs are not equally useful for every video, so AFF learns dynamic weighting instead of fixed or uniform fusion. In the paper’s interpretation, abrupt anomalies can drive higher weight on smaller scales, whereas prolonged or context-dependent anomalies can shift emphasis toward larger scales.

Temporal Context Enhancement is the final refinement stage. It is applied after scale fusion and is described as a self-attention-style mechanism for modeling dependencies across the aggregated temporal features. Its stated role is to propagate salient abnormal cues across neighboring and distant timestamps, suppress irrelevant or noisy segments, and sharpen temporal discrimination.

3. Formal specification of the AMTPN operators

The TPP stage defines the multiscale feature set as

$$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), \ldots, \phi_{s_K}(X)\}.$$

For each temporal scale $s_k$,

$$\phi_{s_k}(X) = \text{ReLU}\big(\text{BN}(\text{Conv1D}(\mathcal{P}_{s_k}(X)))\big),$$

where

$$\mathcal{P}_{s_k}(X)=\text{AvgPool1D}(X; s_k, 1, \lfloor s_k/2 \rfloor).$$

The interpretation given in the paper is precise: AvgPool1D uses kernel size $s_k$, stride $1$, and padding $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$0, thereby preserving temporal length while varying the receptive field. The implementation uses the scale set

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$1

for both XD-Violence and UCF-Crime (An et al., 28 Jul 2025).

AFF computes content-dependent scale weights. For each scale-specific feature,

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$2

The resulting embeddings are concatenated and transformed into normalized weights:

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$3

where $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$4, $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$5, and

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$6

The fused representation is then

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$7

A compact form given by the paper is

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$8

TCE is specified in two forms. One is

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s \phi_s(X) \rightarrow \text{TCE}\!\left(\sum_s w_s \phi_s(X)\right),$$9

where $X \in \mathbb{R}^{B \times C \times T}.$0 is global average pooling over the temporal dimension, $X \in \mathbb{R}^{B \times C \times T}.$1 and $X \in \mathbb{R}^{B \times C \times T}.$2 are learnable projection matrices, $X \in \mathbb{R}^{B \times C \times T}.$3 is sigmoid, and $X \in \mathbb{R}^{B \times C \times T}.$4 is element-wise multiplication. The algorithmic pseudocode also presents a Transformer-like form:

$X \in \mathbb{R}^{B \times C \times T}.$5

$X \in \mathbb{R}^{B \times C \times T}.$6

$X \in \mathbb{R}^{B \times C \times T}.$7

$X \in \mathbb{R}^{B \times C \times T}.$8

This dual presentation suggests that TCE should be understood functionally as a context-enhancing refinement stage rather than reduced to a single narrow implementation reading.

4. Multigrained representation, adaptivity, and contextualization

The paper states that AMTPN produces a multigrained representation by combining multiple temporal receptive fields, adaptive scale weighting, and contextual recalibration (An et al., 28 Jul 2025). The intended rationale is that anomalies differ not only in appearance but also in temporal grain: some are visible over a few frames, some emerge across longer trajectories, and some require surrounding context for interpretation.

The term “adaptive” is attached specifically to content-aware scale selection. AFF learns which scales matter more for a particular sample instead of imposing uniform contributions across branches. The term “multiscale” refers to the pyramid of receptive fields created by TPP. The term “context-aware” refers to the addition of TCE after fusion, which injects long-range temporal dependencies into the already aggregated feature.

The paper further frames AMTPN with an information-theoretic objective:

$X \in \mathbb{R}^{B \times C \times T}.$9

This is used to motivate the claim that the module should preserve task-relevant information about anomaly labels while limiting unnecessary information loss. A Complementarity Index is also defined:

$T$0

with

$T$1

and

$T$2

This quantity is used to motivate the position that multiscale features are synergistic rather than merely redundant. A plausible implication is that the paper treats scale interaction as an information-combination problem, not only an architectural design choice.

5. Coupling with CBAM and relevance to weakly supervised VAD

AMTPN is not identical to the full main branch of DAMS; it is coupled with CBAM, which follows it in the temporal pipeline. The paper states the branch as

$T$3

after which CBAM refines the result through sequential channel and temporal/spatial attention (An et al., 28 Jul 2025):

$T$4

Channel attention is defined as

$T$5

and temporal attention as

$T$6

For 1D temporal sequences, the tensor is unsqueezed to 2D, CBAM is applied, and the result is squeezed back.

The division of labor between the two modules is explicit. AMTPN determines what temporal scales to extract and how to combine them; CBAM determines which channels and temporal positions are most informative after multiscale temporal processing. The paper treats them as complementary: AMTPN improves temporal modeling, whereas CBAM improves feature calibration and localization.

This interaction is especially relevant under weak supervision, where only video-level labels are available and frame-level anomaly labels are absent. The paper argues that AMTPN helps temporal localization by encoding multiple time scales, adaptively emphasizing the most relevant scale per sample, and contextualizing features so that anomaly evidence is not isolated to a single snippet. The stated motivation is particularly tied to XD-Violence and UCF-Crime, where anomaly duration varies widely.

6. Empirical findings, ablations, and relation to earlier time-pyramid models

The paper reports clear gains from introducing AMTPN into the baseline DAMS branch. From Table 6, the baseline yields 77.07 AP / 78.24 AUC, whereas Baseline + AMTPN yields 83.01 AP / 87.45 AUC (An et al., 28 Jul 2025). This is presented as evidence that AMTPN alone improves performance substantially over the baseline.

The component ablations in Table 7 further distinguish the contributions of TPP, AFF, and TCE:

Configuration	Reported result
Baseline + TPP + TCE	79.35 / 82.18
Baseline + AFF + TPP	82.02 / 89.43
Baseline + AFF + TCE	82.77 / 91.15
Full AMTPN (TPP + AFF + TCE)	84.00 / 94.67

The paper identifies AFF as the most influential AMTPN submodule. Removing AFF caused the largest drop among components, specifically 12.49% on UCF-Crime and 4.65% on XD-Violence. Removing TCE caused a smaller but still meaningful drop of 3.52% on UCF-Crime and 1.23% on XD-Violence. These results are used to support three linked claims: TPP provides the multiscale foundation, AFF is the critical adaptive mechanism, and TCE supplies an additional contextual gain.

The full DAMS framework reaches 84.00% AP on XD-Violence and 94.67% AUC on UCF-Crime. The paper attributes much of this to AMTPN’s ability to capture anomalies at different temporal granularities, produce smoother and more coherent temporal anomaly scores, and better detect both transient and long-duration anomalies.

A frequent source of confusion is the similarity between AMTPN and the earlier Multi-scale Transformer Pyramid Network (MTPNet) for multivariate time series forecasting (Zhang et al., 2023). The two are related at the level of broad design intent: both reject rigid single-scale temporal modeling and both use temporal pyramids to address heterogeneous dependencies. However, they are not the same architecture. MTPNet is designed for forecasting multivariate time series $T$7, uses a dimension invariant embedding, transformer encoder-decoder pairs across unconstrained patch sizes $T$8, and aggregates multi-scale latent states for prediction. AMTPN, by contrast, operates on video feature sequences $T$9 and is built from TPP, AFF, and TCE rather than DI embedding and transformer pyramid decoding. This suggests that AMTPN should be understood as a video-anomaly-specific adaptive temporal pyramid module, not as a renaming of the forecasting-oriented MTPNet.