Dual-Branch Adaptive Multiscale Spatiotemporal Framework

• DAMS is a weakly supervised video anomaly detection method that addresses multiscale temporal dependencies, visual-semantic heterogeneity, and sparse frame annotations.
• It integrates a main spatiotemporal branch for adaptive multiscale modeling with a CLIP-driven semantic branch that provides high-level vision-language priors and pseudo-labels.
• The fused framework improves anomaly localization and detection accuracy, as evidenced by enhanced AP on XD-Violence and AUC on UCF-Crime datasets.

The Dual-Branch Adaptive Multiscale Spatiotemporal Framework (DAMS) is a weakly supervised video anomaly detection method that targets three intertwined difficulties in real-world anomaly detection: multiscale temporal dependencies, visual-semantic heterogeneity, and scarcity of frame-level annotations. It formulates anomaly detection as spatio-temporal localization of abnormal events and decomposes representation learning into two complementary branches: a main spatiotemporal branch for adaptive multiscale temporal modeling and discriminative feature refinement, and a CLIP-driven semantic branch that injects high-level vision-language priors through semantic alignment and pseudo-labeling. The two branches are fused so that low-level dynamics and high-level semantics reinforce each other, yielding what the paper describes as a complete inference chain from underlying spatio-temporal features to high-level semantic concepts (An et al., 28 Jul 2025).

1. Problem setting and design rationale

DAMS is introduced in the context of weakly supervised video anomaly detection, where only video-level supervision is available during training and frame-level localization must be inferred indirectly (An et al., 28 Jul 2025). The design rationale is organized around four bottlenecks.

First, fixed-scale temporal modeling is treated as insufficient. Prior methods relying on single-path architectures or predefined temporal pyramids are described as unable to adapt flexibly to anomalies of different durations. Some anomalies are transient, whereas others evolve slowly, so a rigid temporal receptive field may miss short bursts or long-range dependencies.

Second, single-dimensional feature processing is described as inadequate because temporal modeling alone does not determine which channels and which temporal positions matter most, especially in cluttered surveillance video. Third, weak supervision limits localization quality, since anomalous segments must be inferred from video-level labels, often producing noisy predictions and poor instance selection. Fourth, semantics are treated as underutilized in visual-only pipelines, motivating the introduction of CLIP to inject semantic knowledge learned from large-scale vision-language pretraining.

Within this formulation, DAMS is explicitly intended to create a more complete anomaly representation: one branch learns adaptive multiscale spatiotemporal structure, and the other provides semantic guidance through cross-modal alignment. A plausible implication is that the framework is designed not merely to improve classification accuracy, but to improve the localization behavior of weakly supervised models by coupling temporal precision with semantic priors.

2. Dual-branch architecture and complementarity

The architecture consists of two parallel pathways (An et al., 28 Jul 2025). The main spatiotemporal branch takes extracted video features and processes them through a temporal convolutional transformer backbone, followed by the Adaptive Multiscale Temporal Pyramid Network (AMTPN) and a Convolutional Block Attention Module (CBAM). Its role is to model fine-grained motion, temporal context, and discriminative feature salience.

The CLIP-guided semantic branch leverages CLIP’s pretrained vision and text encoders to establish a shared embedding space between visual snippets and anomaly-related textual descriptions. It generates frame-level pseudo-labels and supports multi-scale instance selection and semantic alignment. Its role is to provide high-level priors that compensate for missing frame annotations.

The paper characterizes the two branches as orthogonally complementary. The main branch is strong at temporal localization and feature refinement, while the CLIP branch injects semantic structure and weak supervision signals. Their fusion is presented as improving robustness and interpretability. A common misconception would be to regard the CLIP component as a frozen classifier appended to a conventional detector. The framework explicitly rejects that reading: CLIP is not simply used as a frozen classifier, but as a source of semantic supervision and instance-level guidance.

3. Main spatiotemporal branch: AMTPN and CBAM

The main temporal contribution is the Adaptive Multiscale Temporal Pyramid Network, defined as a three-stage cascade consisting of Temporal Pyramid Pooling (TPP), Adaptive Feature Fusion (AFF), and Temporal Context Enhancement (TCE) (An et al., 28 Jul 2025). The AMTPN input is a video feature sequence

$X \in \mathbb{R}^{B \times C \times T}$

with temporal scales $S = \{s_1, s_2, ..., s_k\}$. Its overall pipeline is given as

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$

Temporal Pyramid Pooling extracts hierarchical temporal representations at multiple resolutions: $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$ where

$$\phi_{s_k}(X) = \text{ReLU}(\text{BN}(\text{Conv1D}(\mathcal{P}_{s_k}(X))))$$

and

$$\mathcal{P}_{s_k}(X) = \text{AvgPool1D}(X; s_k, 1, \lfloor s_k/2 \rfloor).$$

The reported temporal pyramid scales are $[1,3,9,27]$ for both datasets. Small scales capture local short-term cues, while large scales capture longer-term patterns and semantic context.

Adaptive Feature Fusion replaces uniform multiscale aggregation with data-dependent weighting. The normalized weights are written as

$$w = \text{Softmax}(\text{Conv1D}(\text{AdaptiveAvgPool1D}(\phi_{s_1}(X))))$$

and the fusion as

$$\mathcal{F}_{\text{fused}} = \text{Conv1D}\left(\sum_{i=1}^K w_i \cdot \phi_{s_i}(X)\right).$$

In the algorithmic description, each scale feature is globally averaged, processed by a two-layer MLP with ReLU, concatenated, and passed through a softmax to produce fusion weights. The ablation study identifies AFF as the most important internal AMTPN component, supporting the claim that adaptive scale weighting is central when anomalies vary in duration and density.

Temporal Context Enhancement is a context refinement module expressed in channel-attention style: $$\text{TCE}(\mathcal{F}_{\text{agg}}) = \mathcal{F}_{\text{agg}} \odot \sigma(W_2 \cdot \text{ReLU}(W_1 \cdot \text{GAP}(\mathcal{F}_{\text{agg}}))).$$ It learns channel-wise importance weights from global temporal context, suppressing uninformative channels and amplifying channels that better discriminate anomalies. The algorithmic description also uses linear projections $S = \{s_1, s_2, ..., s_k\}$0, scaled dot-product attention, residual connections, layer normalization, and an FFN. This suggests a transformer-style self-attentive context-enhancement step, although the paper’s explicit formulation remains centered on context refinement.

After AMTPN, CBAM refines features across both channel and temporal dimensions. Given

$S = \{s_1, s_2, ..., s_k\}$1

the module applies

$S = \{s_1, s_2, ..., s_k\}$2

Channel attention is defined as

$S = \{s_1, s_2, ..., s_k\}$3

and temporal attention as

$S = \{s_1, s_2, ..., s_k\}$4

For 1D temporal processing, the paper uses

$S = \{s_1, s_2, ..., s_k\}$5

CBAM is justified as maximizing information entropy across channel and spatial dimensions and is also expressed through the mutual-information objective

$S = \{s_1, s_2, ..., s_k\}$6

4. CLIP-driven semantic branch and cross-branch fusion

The parallel branch introduces contrastive language-visual pretraining into weakly supervised anomaly detection (An et al., 28 Jul 2025). It computes similarity between a frame $S = \{s_1, s_2, ..., s_k\}$7 and textual descriptions $S = \{s_1, s_2, ..., s_k\}$8 of anomaly classes via CLIP encoders $S = \{s_1, s_2, ..., s_k\}$9 and $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$0: $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$1 The score is converted into a pseudo-label distribution: $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$2 These pseudo-labels are precomputed and used during training, providing frame-level semantic hints without additional annotation.

The branch also includes a multiscale instance selection and alignment mechanism. Its stated function is to identify the most semantically informative instances across scales and align them with textual anomaly descriptions. In effect, it acts as a semantic filter that highlights frame snippets whose CLIP similarity suggests anomaly-like content, and these pseudo-supervised instances guide the main branch.

Information fusion between the two branches is described as an attention-driven integration strategy based on mutual information maximization. The bidirectional interaction links bottom-up spatiotemporal representations from AMTPN plus CBAM with top-down semantic guidance from CLIP. The paper argues that this matters because AMTPN plus CBAM provides precise temporal localization and feature discrimination, whereas CLIP provides semantic priors that reduce ambiguity in weak supervision. A plausible implication is that the fusion is intended to reduce confusion between background motion and genuinely abnormal events in settings where appearance alone is subtle or context-dependent.

5. Training objectives and optimization

DAMS uses a multi-task learning framework with three losses, automatically balanced by uncertainty-based weighting (An et al., 28 Jul 2025).

The pseudo-label supervised loss uses CLIP-generated pseudo-labels with focal loss: $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$3 Pseudo-labels are obtained through thresholding: $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$4

The video-level classification loss aggregates the most anomalous frame scores using top-$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$5 pooling: $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$6 where $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$7 are the top $$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$8 frame scores and

$$X \rightarrow \{\phi_s(X)\} \rightarrow \sum_s w_s\phi_s(X) \rightarrow \text{TCE}\left(\sum_s w_s\phi_s(X)\right).$$9

The binary video classification loss is

$$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$0

To increase separation between normal and anomalous representations, the framework uses triplet contrastive loss: $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$1 Here, $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$2 is the anomaly anchor, $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$3 is a positive embedding derived from CLIP pseudo-labels, and $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$4 is a negative embedding from normal samples.

The total loss is

$$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$5

This formulation is used to automatically balance semantic supervision, video-level classification, and metric learning without manual tuning.

The implementation details reported in the paper are: NVIDIA vGPU-48GB hardware, Optuna hyperparameter tuning, temporal pyramid scales $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$6, maximum training iterations of 5000, validation every 100 steps, automatic mixed precision enabled, training batch size 30, test batch size 5 for XD-Violence and 10 for UCF-Crime, 4 data-loading workers, and 10-crop augmentation at test time.

6. Datasets, empirical results, and analytical findings

The empirical evaluation is conducted on XD-Violence and UCF-Crime (An et al., 28 Jul 2025). XD-Violence contains 4,754 untrimmed videos, 217 hours total, and 6 violent categories: fighting, shooting, riot, abuse, explosion, and car accidents. Its split is 3,954 training videos and 800 testing videos, with frame-level start-end annotations used at test time and AP as the metric. UCF-Crime contains 1,900 untrimmed videos, 128 hours total, and 13 anomalous event types plus the normal class. Its split is 1,610 training videos and 290 testing videos, with frame-level AUC as the metric.

On XD-Violence, the reported performance is 84.00% AP. The listed comparison methods are MGFN (I3D-RGB) at 80.11%, Wu et al. (I3D-RGB) at 78.64%, RTSM at 77.81%, MSL at 78.28%, NG-MIL at 78.51%, and ST-HTAM at 78.06%. The paper’s discussion states that DAMS is 4.81 AP points better than MGFN (79.19/80.11 depending on the table/version reported).

On UCF-Crime, the reported performance is 94.67% AUC. The listed comparison methods are MGFN (I3D-RGB) at 86.98%, CRFD at 89.89%, GLFE at 86.12%, NG-MIL at 85.63%, BN-WVAD at 84.29%, TDS-Net at 84.50%, and ST-HTAM at 81.42%. The paper describes this as a very large gain over prior weakly supervised approaches under the reported protocol.

The ablation studies attribute measurable gains to each major component. For the full framework, the baseline is 77.07 AP and 78.24 AUC; adding CBAM only yields 80.60 and 85.29; adding AMTPN only yields 83.01 and 87.45; and full DAMS yields 84.00 and 94.67. This indicates that AMTPN contributes more than CBAM alone, while the combination performs best.

For internal AMTPN modules, the baseline is 77.07 and 78.24; TCE + TPP gives 79.35 and 82.18; AFF + TPP gives 82.77 and 91.15; AFF + TCE gives 82.02 and 89.43; and full AMTPN gives 84.00 and 94.67. The paper reports AFF as the most impactful AMTPN module, and removing it causes the largest drop.

For CBAM submodules, the baseline is 77.07 and 78.24; spatial attention only gives 83.63 and 88.73; channel attention only gives 82.88 and 90.73; and full CBAM gives 84.00 and 94.67. Both channel and spatial/temporal attention therefore contribute, with their joint use performing best.

For the loss terms, $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$7 gives 79.00 and 71.83; $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$8 gives 80.67 and 91.41; and $$\mathcal{F}_{\text{multi}} = \{\phi_{s_1}(X), \phi_{s_2}(X), ..., \phi_{s_K}(X)\},$$9 gives 84.00 and 94.67. The reported interpretation is that pseudo-label supervision helps substantially, triplet loss is especially important for discriminability, and the full three-loss combination is best.

The qualitative analyses include temporal anomaly score curves, feature evolution visualizations, and pyramid scale behavior. DAMS is shown to produce sharper peaks on anomalous segments and lower scores on normal frames than the baseline. After adaptive multiscale fusion, features become more centralized, less redundant, and more temporally coherent. Lower scales capture short-term local details, while higher scales capture long-term semantic patterns. These analyses are used to support the interpretation that DAMS improves both localization and representation stability.

7. Limitations and prospective extensions

The paper explicitly identifies two limitations (An et al., 28 Jul 2025). First, precomputed CLIP pseudo-labels may introduce domain bias and reduce adaptability to novel anomaly types. Second, computational overhead may limit real-time deployment.

The future directions proposed by the authors are end-to-end trainable versions of the architecture, more efficient attention mechanisms, semi-supervised learning to better leverage unlabeled data, and additional modalities beyond CLIP, including audio and motion cues.

Taken together, DAMS is presented as a dual-branch weakly supervised video anomaly detection framework that couples an AMTPN plus CBAM spatiotemporal branch with a CLIP-guided semantic branch. Its central claim is that anomaly understanding benefits from the combination of bottom-up spatiotemporal precision and top-down semantic priors, and the reported results on UCF-Crime and XD-Violence are used to substantiate that position (An et al., 28 Jul 2025).