VAD-Instruct50k: Multimodal VAD Benchmark

• VAD-Instruct50k is a large-scale multimodal benchmark that integrates sparse, precise temporal annotations with LLM-generated explanations for video anomaly detection.
• It leverages efficient single-frame supervision and trimmed event clips from UCF-Crime and XD-Violence to reduce annotation costs while enabling accurate event localization.
• The dataset advances unbiased anomaly detection and explainable reasoning by standardizing open-ended instruction tuning with rich, human-centric dialogue pairs.

VAD-Instruct50k is a large-scale, multimodal instruction-tuning benchmark designed for open-ended video anomaly detection (VAD) with a focus on precise temporal supervision and rich, explainable annotations. Developed as part of the Holmes-VAD framework, it addresses the limitations of existing unsupervised and weakly supervised VAD methods, such as biased detection in the presence of challenging or unseen events and the lack of interpretability. VAD-Instruct50k uniquely combines efficient single-frame labeling, trimmed event-based video segments, and LLM-generated natural language explanations to enable accurate anomaly localization and comprehensive human-centric reasoning (Zhang et al., 2024).

1. Motivation and Rationale

The development of VAD-Instruct50k was motivated by two critical limitations in existing VAD research: the biased anomaly space resulting from coarse video-level labels, and the lack of interpretability owing to black-box model design. Traditional unsupervised and weakly supervised pipelines suffer from frequent false alarms on uncommon but normal activities due to the absence of reliable frame-level supervision. Moreover, even when localized, anomalous events lack corresponding human-readable explanations, hindering transparent deployment and effective debugging. VAD-Instruct50k addresses these gaps by providing sparse yet reliable temporal supervision and constructing richly annotated clip–text pairs suitable for both detection and open-ended explanatory tasks.

2. Data Collection and Annotation Paradigm

VAD-Instruct50k was sourced from the largest weakly supervised VAD datasets: UCF-Crime and XD-Violence. The dataset comprises 810 abnormal and 800 normal videos from UCF-Crime, together with 1,905 abnormal and 2,032 normal videos from XD-Violence, yielding a total of 5,547 untrimmed, manually filtered videos. Each anomalous event is annotated with exactly one timestamp (single-frame “glance” annotation), producing on average 2.35 annotations per video (approximately 13,000 annotated frames). This paradigm sharply reduces annotation cost relative to dense frame-by-frame labeling, while allowing valid event localization.

Around each annotated frame set $\mathcal{G} = \{g_i\}_{i=1}^{n}$, a lightweight VAD network $\phi_s$ is pseudo-labeled to predict dense anomaly scores $\{s_t\}$. For each $g_i$, temporal clip proposals $(s_i, e_i)$ are formed by extending outward from $g_i$ until $s_t$ falls below a fraction $\alpha$ of $s_{g_i}$. This process generates a set $E = \{ (s_i, e_i, y_i)\}_{i=1}^{N_e}$ of event clips, where $\phi_s$0 denotes the anomaly class or “Normal.” Randomly sampled windows are drawn from normal videos to provide negative examples. Clips are paired with semantic captions $\phi_s$1 produced by Video-LLaVA, augmented where available by SurveillanceVision annotations.

3. Instruction Synthesis and Data Structure

Dialogue-style instruction-tuning data is synthesized by pairing user queries $\phi_s$2 (e.g., “Are there any unexpected or unusual events in the video clip?”) with a “task” prompt $\phi_s$3 that concatenates the clip caption and its anomaly label for reference. Feeding $\phi_s$4 to an LLM (Llama3-Instruct-70B) produces assistant responses that both identify and explain the anomaly, resulting in samples of the form:

$\phi_s$5

VAD-Instruct50k integrates three modalities: (1) raw untrimmed videos with single-frame timestamps, (2) trimmed, labeled clips with dense captions, and (3) dialogue pairs in JSON-like format. Abnormal events are annotated among 13 classes for UCF-Crime (e.g., “Explosion,” “Fighting,” “Car Accident”) and 6 classes for XD-Violence, while normal clips are treated uniformly with descriptive captions. A diverse set of prompts $\phi_s$6 ensures broad instruction coverage.

A statistical overview is provided below:

Statistic	Value	Source
# of untrimmed videos	5,547	(Zhang et al., 2024)
Avg. single-frame annotations/video	2.35 → ~13,000 frames	(Zhang et al., 2024)
# of trimmed event clips $\phi_s$7	~13,000–14,000	(Zhang et al., 2024)
# of instruction items	51,567	(Zhang et al., 2024)
Avg. words/assistant response	44.83	(Zhang et al., 2024)

4. Annotation Quality Control and Evaluation

A semi-automatic, expert-in-the-loop pipeline emphasizes reliability at each stage. Low-quality source videos are eliminated via manual inspection. Single-frame event annotations undergo a two-round review: independent annotation followed by adjudication to remove errors and redundancy. Instructional outputs from the LLM are filtered to exclude low-fidelity or irrelevant responses.

The dataset’s viability for interpretable VAD is validated by training the Holmes-VAD temporal sampler on pseudo-labeled frames and fine-tuning a multimodal LLM using the full instruction corpus. Detection performance is benchmarked using frame-level AUC for UCF-Crime and AP for XD-Violence. Explanation quality is quantitatively assessed via human evaluation across three axes—Judgement Accuracy (JA), Content Perception (CP), and Anomaly Explanatory power (AE)—on 86 clips rated independently by 10 annotators.

5. Unique Contributions and Advancements

VAD-Instruct50k’s key innovations are its integration of sparse, glance-level temporal supervision with LLM-generated explanations, and its support for unified anomaly detection and reasoning. This structure enables the following advances:

• Unbiased Anomaly Detection: Temporal samplers trained on single-frame cues are shown to resist false positives on rare but normal behavior, addressing the systematic bias of prior VAD models.
• Explainable Reasoning: Instruction-tuned multimodal LLMs can answer both “when” an anomaly occurs and “why” it is anomalous, yielding high utility for real-world, safety-critical surveillance contexts.
• Standardized Open-Vocabulary Benchmark: With 5,547 videos and over 51,000 instruction items, VAD-Instruct50k establishes a uniform testbed for future research on explainable, open-ended VAD, spanning both detection accuracy and human-centric interpretability.

6. Prospective Research Directions

The design of VAD-Instruct50k creates a foundation for multiple research directions, including zero- and few-shot anomaly reasoning, transparent justification in live surveillance alerting, and the integration of world knowledge into automated video security systems. Its data format and methodology facilitate further exploration of multimodal LLM capabilities for high-stakes video analysis and provide a strong baseline for community-driven benchmarking and future innovation in interpretable VAD (Zhang et al., 2024).