LAVAD: Training-Free Video Anomaly Detection

• LAVAD Framework is a training-free video anomaly detection method that leverages pre-trained vision-language models and LLMs to localize anomalies.
• It transforms video frames into text descriptions and employs cross-modal similarity cleaning to mitigate caption noise and improve anomaly scoring.
• Temporal aggregation with prompt engineering refines anomaly scores, achieving competitive performance on benchmarks like UCF-Crime and XD-Violence.

LAVAD (LAnguage-based, training-free Video Anomaly Detection) is a methodology for video anomaly detection that eschews the traditional requirement of domain-specific model training by exploiting frozen, pre-trained vision-LLMs (VLMs) and LLMs at inference time only. LAVAD achieves competitive performance in localizing anomalous events in surveillance videos by transforming the visual domain into temporally-aggregated text and employing prompt-based anomaly scoring, combined with cross-modal similarity cleaning and refinement. All model components operate without any task-specific retraining or data collection, fundamentally distinguishing LAVAD from established unsupervised, weakly supervised, or one-class anomaly detection methods (Zanella et al., 2024).

1. Problem Motivation and Conceptual Architecture

Conventional video anomaly detection (VAD) frameworks depend on training deep models to learn a mapping from video frames (or frame sequences) to anomaly scores $a_i = f(I_i)$. These pipelines are sensitive to shifts in environment, require substantial annotated data, and routinely incur considerable deployment costs due to the need for domain adaptation.

LAVAD circumvents these limitations through a training-free formulation, leveraging:

• Off-the-shelf VLMs for dense per-frame video captioning,
• A systematic prompt engineering and temporal aggregation strategy with LLMs for anomaly scoring,
• Cross-modal similarity-based caption cleaning and anomaly score refinement,
• An end-to-end pipeline that applies solely pretrained, frozen models.

The high-level stages are summarized as:

1. Video-to-Text Captioning via VLM;
2. Caption Cleaning by cross-modal similarity;
3. Temporal Summary Generation and Prompt-based Anomaly Scoring via LLM;
4. Score Refinement using text-video embedding alignment.

2. Video-to-Text Captioning and Caption Cleaning

Each test video $V = [I_1, ..., I_M]$ is discretized by sampling one frame every 16 frames. The BLIP-2 model family (variants such as flan-t5-xl, flan-t5-xxl, opt-6.7b) serves as the captioner, producing raw textual descriptions $C = [C_1, ..., C_M]$ for all sampled frames.

To suppress noisy or misaligned captions, LAVAD utilizes ImageBind’s aligned encoders ($E_I$ for images, $E_T$ for text). For each frame $I_i$, the method computes $\hat{C}_i = C_{j^*}$, where $$j^* = \arg\max_j \cos\left(E_I(I_i), E_T(C_j)\right)$$. This cross-modal cleaning step selects the caption whose semantic embedding most closely matches the frame, mitigating VLM-induced captioning artifacts.

3. Prompting Mechanism and Temporal Aggregation via LLM

LAVAD interleaves three distinct prompt templates:

• Context Prompt ($P_C$): Frames the detection problem in a forensic context, e.g., "If you were a law enforcement agency, how would you rate the scene described on a scale from 0 to 1, with 0 representing a standard scene and 1 denoting a scene with suspicious activities?"
• Format Prompt ($P_F$): Demands a machine-parsable answer from the LLM: "Please provide the response in the form of a Python list and respond with only one number in the provided list below [0, 0.1, ..., 1.0] without any textual explanation. It should begin with ‘[’ and end with ‘]’."
• Summary Prompt ($V = [I_1, ..., I_M]$0): Asks the LLM to produce a brief summary of events from a local temporal window: "Please summarize what happened in few sentences, based on the following temporal description of a scene. Do not include any unnecessary details or descriptions."

For each test frame, N=10 captions are sampled uniformly from a $V = [I_1, ..., I_M]$1 second window around $V = [I_1, ..., I_M]$2, forming a snippet. The cleaned captions of the snippet are concatenated and input to the LLM with $V = [I_1, ..., I_M]$3, producing a temporally-aggregated description $V = [I_1, ..., I_M]$4. This is then scored via $V = [I_1, ..., I_M]$5, yielding $V = [I_1, ..., I_M]$6.

4. Mathematical Pipeline and Cross-Modal Refinement

Let $V = [I_1, ..., I_M]$7, $V = [I_1, ..., I_M]$8, $V = [I_1, ..., I_M]$9, and $C = [C_1, ..., C_M]$0 be the temporal summary for snippet $C = [C_1, ..., C_M]$1 around $C = [C_1, ..., C_M]$2.

Mathematical Definitions

1. Caption Cleaning: $C = [C_1, ..., C_M]$3, where

$C = [C_1, ..., C_M]$4

1. Temporal Summary ($C = [C_1, ..., C_M]$5): The LLM receives $C = [C_1, ..., C_M]$6 concatenated with the sequence of snippet captions.
2. Initial Anomaly Score:

$C = [C_1, ..., C_M]$7

1. Score Refinement:
   • Compute snippet embedding $C = [C_1, ..., C_M]$8 and summary embeddings $C = [C_1, ..., C_M]$9
   • Similarity weights: $E_I$0
   • For the $E_I$1 most similar summaries (by cosine similarity), the final score:

   $E_I$2

• Anomaly label is set if $E_I$3; ROC AUC across candidate $E_I$4 is reported.

End-to-End Pseudocode (Condensed)

$E_I$5

5. Experimental Results and Comparative Analysis

Experiments utilize:

• UCF-Crime: 1,900 surveillance videos with 13 anomaly types, separate train/test splits.
• XD-Violence: 4,754 videos covering 6 violence classes, with train and test splits.

Implementation employs BLIP-2 variants for captioning, Llama-2-13B-chat for prompting, and ImageBind for multimodal embeddings.

Main Results

Dataset	Metric	LAVAD	SOTA (Unsupervised)	SOTA (Training-Free)
UCF-Crime	Frame AUC (%)	80.28	GCL (79.84)	CLIP/ImageBind (~53–56), LLaVA-1.5 (72.84)
XD-Violence	Frame AUC (%)	85.36	RareAnom (68.33)	LLaVA-1.5 (79.62)
XD-Violence	AP (%)	62.01	–	LLaVA-1.5 (50.26)

Ablation studies indicate:

• Caption cleaning improves AUC by 3.8 points (UCF-Crime).
• Temporal summary and refinement contribute additive ~7.5-point improvements each.
• Prompt impersonation ("If you were a law enforcement agency") provides a 0.96-point gain.
• Ensemble captioners outperform single-model variants.

6. Ablation Studies and Parameter Sensitivity

The effect of each LAVAD component was isolated:

• Without caption cleaning: AUC falls from 80.28% to 76.48%.
• Without temporal summary (using only cleaned captions + refinement): Drops to 72.70%.
• Without score refinement: Drops to 72.79%.
• Prompt variations: Best performance with context framing + impersonation.
• Window and neighbor size: T=10 s and K=10 neighbors yield optimum results; performance degrades for smaller/larger windows or neighbor counts.

Performance proved robust across datasets with different anomaly types and densities.

7. Limitations and Implications

LAVAD is limited to the expressiveness and semantic range of the underlying captioner and LLM; failure cases may occur for subtle, highly context-dependent anomalies. No explicit handling exists for frame-wise temporal coherence outside LLM summary aggregation. All evaluation was conducted on surveillance datasets; a plausible implication is that domain transferability hinges on the adequacy of VLM-generated scene descriptions (Zanella et al., 2024).

No user studies, latency, or runtime performance metrics are reported. All components operate in a genuinely "training-free" regime for VAD, precluding adaptation to new anomaly types absent modification of upstream vision-LLMs or LLMs. The cross-modal similarity cleaning/refinement pipeline critically depends on the alignment fidelity of embedding spaces. Potential extensions include adaptation to other event localization tasks and improved anomaly score calibration.

• "Harnessing LLMs for Training-free Video Anomaly Detection" (Zanella et al., 2024)