Evaluation of Vision-LLMs in Surveillance Video

Abstract: The widespread use of cameras in our society has created an overwhelming amount of video data, far exceeding the capacity for human monitoring. This presents a critical challenge for public safety and security, as the timely detection of anomalous or criminal events is crucial for effective response and prevention. The ability for an embodied agent to recognize unexpected events is fundamentally tied to its capacity for spatial reasoning. This paper investigates the spatial reasoning of vision-LLMs (VLMs) by framing anomalous action recognition as a zero-shot, language-grounded task, addressing the embodied perception challenge of interpreting dynamic 3D scenes from sparse 2D video. Specifically, we investigate whether small, pre-trained vision--LLMs can act as spatially-grounded, zero-shot anomaly detectors by converting video into text descriptions and scoring labels via textual entailment. We evaluate four open models on UCF-Crime and RWF-2000 under prompting and privacy-preserving conditions. Few-shot exemplars can improve accuracy for some models, but may increase false positives, and privacy filters -- especially full-body GAN transforms -- introduce inconsistencies that degrade accuracy. These results chart where current vision--LLMs succeed (simple, spatially salient events) and where they falter (noisy spatial cues, identity obfuscation). Looking forward, we outline concrete paths to strengthen spatial grounding without task-specific training: structure-aware prompts, lightweight spatial memory across clips, scene-graph or 3D-pose priors during description, and privacy methods that preserve action-relevant geometry. This positions zero-shot, language-grounded pipelines as adaptable building blocks for embodied, real-world video understanding. Our implementation for evaluating VLMs is publicly available at: https://github.com/pascalbenschopTU/VLLM_AnomalyRecognition

• The paper introduces a novel zero-shot anomaly detection method by reframing video analysis as a language-driven inference task.
• The methodology generates textual descriptions from video frames and applies Natural Language Inference for flexible, prompt-based classification.
• Results highlight benefits from few-shot prompting while noting that privacy filters can degrade accuracy due to video inconsistencies.

Evaluation of Vision-LLMs in Surveillance Video

Introduction

The paper "Evaluation of Vision-LLMs in Surveillance Video" (2510.23190) addresses the critical challenge of detecting anomalous or criminal events from vast surveillance video datasets. Due to the overwhelming volume of video data, manual monitoring becomes impractical, necessitating advanced automated systems capable of real-world video understanding. This work investigates the capabilities of Vision-LLMs (VLMs) for zero-shot anomaly detection, assessing their ability to recognize unexpected events through spatial reasoning grounded in language descriptions.

Spatial Reasoning in Vision-LLMs

The authors propose a novel approach where small, pre-trained vision-LLMs serve as anomaly detectors by transforming video content into textual descriptions and leveraging textual entailment for label scoring. Such models focus on spatially salient events while struggling with complex scenarios due to noisy spatial cues and identity obfuscation. This enables a promising pathway for zero-shot anomaly detection, especially in scenarios characterized by sparse data labels.

Figure 1: Few-shot prompting impact on models accuracy level.

Methodology

The core methodology involves reconfiguring anomaly classification as a language-grounded reasoning task rather than conventional pixel-label mapping. The framework operates in two main stages: generating textual descriptions from video input and applying zero-shot classification through Natural Language Inference (NLI). The innovative element here is the reframing of video understanding tasks that bypass extensive model retraining, relying instead on the inherent world knowledge of LLMs.

Derivation and Practical Implications

The framework derives from a sequence of RGB frames $X=(x_t)_{t=1}^{T}$, generating descriptions using vision-LLM $F_{\theta}$ and scoring with NLI classifier $g_{\phi}$. Practical advantages include true zero-shot recognition flexibility, where new anomaly types can simply be added to textual label sets, and modular architecture permitting seamless model upgrades.

Figure 2: Images from UCF-Crime dataset used for few-shot prompting.

Experimental Setup

Experiments utilize benchmark datasets like UCF-Crime and RWF-2000, evaluating models' zero-shot capabilities under varied experimental conditions, including few-shot prompting and privacy filters. Metrics such as class-averaged Top-1 accuracy reveal insights into models' ability to adapt and generalize across these conditions. The experimental design ensures that any performance differences arise from either prompting or privacy transformations rather than extraneous factors.

Figure 3: Example A.

Results

Prompting Effects

Analysis of prompting techniques demonstrates that few-shot examples can enhance accuracy but may induce higher false-positive rates. This trade-off underscores the complexity of fine-tuning model sensitivity while maintaining high precision in anomaly detection tasks.

Figure 4: All classes compared over prompting experiments, the few-shot examples include: Fighting, RoadAccidents, Shooting, and Stealing.

Privacy Filters

Privacy-preserving filters are critical for real-world deployment yet result in accuracy degradation, with GAN-based full-body anonymization causing the most significant disruption due to introducing video inconsistencies. Models like VideoLLaMA-3 indicate potential in reducing false positives under these conditions, highlighting a direction for future research to improve video consistency.

Conclusions

The investigation into small vision-LLMs for zero-shot anomaly detection elucidates key considerations in model prompting and privacy-preservation strategies. While such models show efficacy in simple tasks such as fight detection, there remains substantial room for improvement in handling complex scenarios autonomously. Future work may emphasize enhancing privacy methods for increased temporal consistency and optimizing model configurations to balance sensitivity and precision effectively. This work contributes a foundational understanding of the applicability and limitations of vision-LLMs in dynamic, real-world video analysis settings.