PEL4VAD: Weakly Supervised Video Anomaly Detection

• PEL4VAD is a weakly supervised video anomaly detection framework that uses video-level labels to refine temporal context and semantic subclass discrimination.
• It employs a four-stage pipeline combining Temporal Context Aggregation and Prompt-Enhanced Learning to align video features with knowledge-based prompts.
• Evaluated on benchmarks like UCF-Crime and ShanghaiTech, PEL4VAD achieves competitive performance with fewer parameters and improved efficiency.

Searching arXiv for the primary PEL4VAD paper and closely related weakly-supervised video anomaly detection work to ground the article. PEL4VAD denotes the weakly supervised video anomaly detection framework introduced in "Learning Prompt-Enhanced Context Features for Weakly-Supervised Video Anomaly Detection" (Pu et al., 2023). It is designed for the setting in which untrimmed videos are labeled only at the video level, without frame-level annotations during training. The method addresses two constraints identified in prior weakly supervised anomaly detection research: the use of multi-branch architectures to model local and global temporal relations separately, which increases parameters and computational costs, and the reliance on binary Multiple Instance Learning (MIL)-based classification loss, whose coarse-grained interclass separability neglects fine-grained discriminability within anomalous classes. PEL4VAD responds with a four-stage pipeline combining Temporal Context Aggregation (TCA), Prompt-Enhanced Learning (PEL), MIL-based optimization, and score smoothing, with the explicit goal of efficient context modeling and enhanced semantic discriminability (Pu et al., 2023).

1. Problem setting and conceptual motivation

Weakly supervised video anomaly detection assumes that only bag-level supervision is available: videos are labeled as normal or anomalous, while precise snippet- or frame-level anomaly boundaries are unavailable during training. In this regime, prior research had already used graph convolution networks and self-attention mechanisms alongside MIL-based classification loss to model temporal relations and learn discriminative features, but often through separate branches for local and global dependencies. PEL4VAD is formulated as a response to the inefficiency of that pattern and to the semantic limitation of treating anomaly as a single binary category (Pu et al., 2023).

The framework therefore targets two distinct objectives. The first is temporal modeling efficiency: local and global contextual cues should be captured without parallel temporal branches. The second is subclass-level semantic discrimination: anomalous subclasses should be encouraged to separate from one another rather than being collapsed into one monolithic anomalous label. In the terminology of the paper, these are handled respectively by TCA and PEL.

A useful way to situate the method is to view it as a hybrid of context aggregation and cross-modal supervision. MIL remains the supervisory backbone, but prompt-based semantic priors are injected to refine representation geometry. This suggests that PEL4VAD is not a replacement for MIL-based weak supervision; rather, it augments MIL with an explicit mechanism for anomaly subclass separability.

2. End-to-end pipeline

PEL4VAD is organized in four stages, as summarized in the paper’s framework overview (Pu et al., 2023).

First, an untrimmed video is split into non-overlapping 16-frame snippets. Each snippet $x_i$ is mapped to a $D$-dimensional embedding through a pre-trained I3D backbone. This stage produces the raw snippet feature sequence $X$.

Second, the snippet embeddings are processed by the Temporal Context Aggregation module. TCA computes a single similarity matrix and reuses it to derive both global and local context features, which are then adaptively fused into a context-enhanced representation $X^c$.

Third, Prompt-Enhanced Learning operates on these context features. A two-layer MLP projects $X^c$ into a mid-level embedding $X^e$. At this point, class-specific knowledge-based prompts $p_c$, constructed from ConceptNet and encoded by a pre-trained CLIP text encoder, are aligned with $X^e$ through a cross-modal alignment loss.

Fourth, a classifier consisting of a causal convolution followed by a sigmoid produces snippet-level anomaly scores $S$. During training, a bag-level MIL loss $L_{MIL}$ encourages higher scores in true anomaly videos. At test time, score-smoothing post-processing reduces spurious peaks (Pu et al., 2023).

This decomposition is central to the identity of PEL4VAD. TCA handles temporal dependency structure with low overhead, while PEL injects semantic priors into the learned feature space. The classifier and MIL objective then convert those features into weakly supervised anomaly scores.

3. Temporal Context Aggregation

The TCA module is the principal mechanism for temporal relation modeling. Its defining property is the reuse of one similarity matrix $D$0 to capture both global and local temporal dependencies without parallel branches (Pu et al., 2023).

Given a sequence $D$1, the similarity matrix is computed as

$D$2

where $D$3 is a learned linear projection into a $D$4-dimensional latent space.

Global context features are obtained by

$D$5

where $D$6 is another linear projection. Local context features are derived by applying a temporal mask $D$7 that zeroes out all but a $D$8-sized temporal window around each snippet. The module then performs adaptive fusion through

$D$9

extended to the matrix form

$X$0

After a residual connection, normalization, and a small linear layer, the fused output becomes the context-enhanced feature $X$1.

The intended efficiency gain follows directly from this construction. Only one attention map is learned, and $X$2 is a single scalar or small vector, so parameter count and FLOPS remain very low while both local and global cues are captured. In ablation, the TCA module alone uses 1.21 M parameters and 241 M FLOPS, while heavier temporal models cited in the paper include RTFM at 24.7 M parameters and MTCM at 13.3 M (Pu et al., 2023). This makes TCA not merely a feature extractor but an efficiency claim about temporal modeling under weak supervision.

4. Prompt-Enhanced Learning and semantic priors

The PEL module is introduced to address the limitation of binary MIL-based supervision, which provides only coarse-grained interclass separability and neglects fine-grained discriminability within anomalous classes (Pu et al., 2023). Its operation depends on prompt construction, context separation, and cross-modal alignment.

For each anomaly subclass $X$3, and for the normal category, the method retrieves $X$4 top relations from ConceptNet, with examples including IsA, UsedFor, and HasProperty. The connected concepts are filtered by relevance score, and the remaining concept words $X$5 are encoded by a pre-trained CLIP text encoder. The resulting prompt embedding is

$X$6

To avoid diluting anomaly signals, the method performs context separation. Snippet scores $X$7 are converted into foreground and background activations $X$8 and $X$9 through a scaled-softmax over $X^c$0 and $X^c$1, respectively. These activations are then pooled against the mid-level MLP output $X^c$2 to obtain video-level representations $X^c$3 and $X^c$4.

Cross-modal alignment is then imposed between video-level features and prompt embeddings through a cosine-softmax:

$X^c$5

where $X^c$6 is cosine similarity and $X^c$7 is a temperature. The alignment loss is given as

$X^c$8

According to the paper, this Kullback–Leibler divergence loss pushes true pairs such as $X^c$9 closer and mismatched pairs apart. The explicit consequence is that anomaly subclasses are separated in feature space rather than collapsed into a single anomalous superclass.

In practical terms, the paper states that this mechanism allows the framework to distinguish, for example, “robbery” from “road accident” or “fighting” from “shooting” more reliably than binary MIL alone (Pu et al., 2023). A plausible implication is that the prompt mechanism functions as an external semantic regularizer over video representations, with ConceptNet and CLIP supplying the knowledge prior.

5. Optimization, supervision, and inference

The total training objective is a weighted sum of the MIL classification loss and the prompt alignment loss:

$X^c$0

Here, $X^c$1 is a bag-level binary cross-entropy defined on top-$X^c$2 pooled snippet scores, with top-$X^c$3 used for abnormal videos and $X^c$4 for normal videos. The coefficient $X^c$5 is dataset-dependent: it is set to 1 for UCF-Crime and XD-Violence, and 9 for ShanghaiTech (Pu et al., 2023).

The classifier attached to the learned features is deliberately simple. A causal convolution followed by sigmoid produces snippet-level anomaly scores $X^c$6. These scores serve both as the basis for MIL optimization and as inputs to the context-separation procedure used by the prompt module.

At inference time, the model outputs snippet-level anomaly scores, and a simple score-smoothing post-processing step is applied to reduce spurious peaks. The paper abbreviates this as SS and reports that it adds another approximately 0.4–0.5% in AUC or AP in ablation (Pu et al., 2023).

This training and inference design clarifies an important point about PEL4VAD. The system is weakly supervised rather than unsupervised, and its prompt component is auxiliary rather than standalone. MIL remains the optimization core, while prompt alignment improves semantic structure in the learned representation.

6. Benchmarks, ablations, and reported performance

PEL4VAD is evaluated on three benchmarks: UCF-Crime, XD-Violence, and ShanghaiTech (Pu et al., 2023). The paper reports state-of-the-art or highly competitive performance while using far fewer parameters for temporal modeling.

Benchmark	Reported result	Comparative note
UCF-Crime	AUC = 86.76%, FAR 0.43%	prior best $X^c$7 by UR-DMU
XD-Violence	AP = 85.59%, FAR 0.57%	prior best 83.54% (multimodal), 81.66% (UR-DMU)
ShanghaiTech	AUC = 98.14%, FAR 0.00%	a new record on this split

The benchmark descriptions provided in the paper are also informative. UCF-Crime contains 13 real anomalies and 1,900 videos. XD-Violence is described as containing 6 violence types and being multi-source. ShanghaiTech is characterized as a crowd-scenes dataset.

Ablation results attribute distinct roles to the major components. TCA alone outperforms heavier temporal models while using 1.21 M parameters and 241 M FLOPS. Adding PEL contributes an extra approximately 1% gain on UCF-Crime and approximately 2% on XD-Violence. Score smoothing yields a further approximately 0.4–0.5% in AUC or AP (Pu et al., 2023).

The paper also notes that the approach significantly improves the detection accuracy of certain anomaly sub-classes. This is consistent with the intended function of the prompt-alignment loss: to enforce fine-grained separability among anomalous subclasses rather than only normal-versus-abnormal separation.

7. Interpretation, significance, and practical implications

PEL4VAD occupies a specific methodological position within weakly supervised video anomaly detection. It preserves the standard MIL formulation but modifies two components that had become limiting in prior work: temporal context modeling and semantic discrimination. The TCA module reduces architectural redundancy by reusing a single similarity matrix for both global and local dependencies, while the PEL module introduces knowledge-based prompts to impose subclass-aware structure on the representation space (Pu et al., 2023).

Several misconceptions are precluded by the design itself. PEL4VAD is not a fully supervised detector, because frame-level annotations are absent during training. It is not a pure prompt-learning system, because the classifier and bag-level MIL loss remain essential. It is also not a multi-branch temporal architecture in the conventional sense criticized by the paper, because local and global context are derived from the same similarity matrix rather than from separate branches.

The practical implications identified in the paper concern security and surveillance scenarios. By grounding anomaly scores in human-readable concepts derived from knowledge-based prompts, the system is described as reducing both false-alarm noise and missed-event risk, while making predictions more interpretable and more trustworthy for downstream operators (Pu et al., 2023). This suggests a broader significance: prompt-based priors can function not only as semantic labels but also as a mechanism for restructuring feature spaces in weakly supervised detection problems where binary supervision is intrinsically coarse.

The publicly released codebase is available at https://github.com/yujiangpu20/PEL4VAD, which further situates PEL4VAD as a reproducible research artifact in the ongoing literature on weakly supervised video anomaly detection (Pu et al., 2023).