Human-Centric Video Anomaly Detection Through Spatio-Temporal Pose Tokenization and Transformer (2408.15185v2)

Abstract: Video Anomaly Detection (VAD) presents a significant challenge in computer vision, particularly due to the unpredictable and infrequent nature of anomalous events, coupled with the diverse and dynamic environments in which they occur. Human-centric VAD, a specialized area within this domain, faces additional complexities, including variations in human behavior, potential biases in data, and substantial privacy concerns related to human subjects. These issues complicate the development of models that are both robust and generalizable. To address these challenges, recent advancements have focused on pose-based VAD, which leverages human pose as a high-level feature to mitigate privacy concerns, reduce appearance biases, and minimize background interference. In this paper, we introduce SPARTA, a novel transformer-based architecture designed specifically for human-centric pose-based VAD. SPARTA introduces an innovative Spatio-Temporal Pose and Relative Pose (ST-PRP) tokenization method that produces an enriched representation of human motion over time. This approach ensures that the transformer's attention mechanism captures both spatial and temporal patterns simultaneously, rather than focusing on only one aspect. The addition of the relative pose further emphasizes subtle deviations from normal human movements. The architecture's core, a novel Unified Encoder Twin Decoders (UETD) transformer, significantly improves the detection of anomalous behaviors in video data. Extensive evaluations across multiple benchmark datasets demonstrate that SPARTA consistently outperforms existing methods, establishing a new state-of-the-art in pose-based VAD.

• The paper introduces PoseWatch, a transformer-based architecture using novel spatio-temporal pose and relative pose tokenization for human-centric video anomaly detection.
• PoseWatch employs a Unified Encoder Twin Decoders (UETD) transformer core with decoders for current sequence reconstruction and future sequence prediction, enhancing anomaly detection via self-supervised learning.
• Evaluations show PoseWatch outperforms state-of-the-art methods on benchmark datasets, achieving high AUC-ROC scores and offering a privacy-preserving solution for surveillance and healthcare applications.

An Overview of PoseWatch: A Transformer-based Architecture for Human-centric Video Anomaly Detection

The presented paper introduces a cutting-edge architecture called PoseWatch, specifically designed to address the complexities of human-centric Video Anomaly Detection (VAD). As VAD continues to evolve as a pivotal area of research within the broader field of computer vision, it brings forth a myriad of challenges primarily due to the unpredictable nature of anomalous events, in addition to privacy concerns associated with human subjects. The paper explores these challenges head-on by employing human pose as a high-level feature, a promising approach to mitigate common pitfalls such as privacy issues and data biases.

PoseWatch stands out in its adoption of a novel Spatio-Temporal Pose and Relative Pose (ST-PRP) tokenization strategy coupled with a transformer-based architecture named the Unified Encoder Twin Decoders (UETD). This approach notably enhances human motion representation over time, which is pivotal for both VAD and broader human behavior analysis tasks. The authors propose an ambitious model that ingeniously integrates NLP techniques with computer vision methodologies to propel the domain of human behavior analysis forward.

Key Methodologies and Architecture

The paper details the conceptual underpinnings of the PoseWatch framework, which is premised on two innovative components:

1. Spatio-Temporal Pose and Relative Pose (ST-PRP) Tokenization: This mechanism accentuates the spatial and temporal dimensions of human poses, enriched further by relative motion information. This serves to maximize the efficiency of self-attention within the transformer, offering a robust tokenization paradigm for sophisticated pose-based tasks. Evaluations indicate the tokenization strategy substantially enhances the model's capabilities in distinguishing normal versus anomalous behaviors, as evidenced by the extensive ablation paper results.
2. Unified Encoder Twin Decoders (UETD) Transformer Core: This component features a dual-branch architecture encompassing a shared encoder and two differently purposed decoders: a Current Target Decoder (CTD) and a Future Target Decoder (FTD). The CTD is responsible for reconstructing the current sequence, while the FTD is tasked with predicting future sequences. The UETD architecture elegantly fuses these insights to bolster anomaly detection accuracy, employing novel self-supervised learning techniques.

Empirical Analysis and Results

PoseWatch underwent extensive evaluation across multiple benchmark datasets, including SHT, HR-SHT, and the Charlotte Anomaly Dataset (CHAD). The architecture consistently outperformed state-of-the-art methods in pose-based VAD, achieving impressive Area Under the Receiver Operating Characteristic Curve (AUC-ROC) scores. For instance, the model secured an average AUC-ROC score of 80.67%, establishing itself as a robust and generalizable anomaly detection framework.

Moreover, PoseWatch demonstrated superior performance in terms of Equal Error Rate (EER), striking a commendable balance between false positive and false negative rates. This is crucial for real-world applications where minimization of both types of errors is vital to ensure reliability and efficiency.

Implications and Future Developments

The implications of PoseWatch's architecture are twofold. Practically, it offers a credible solution for privacy-preserving anomaly detection in video datasets, which can be particularly beneficial for surveillance, healthcare, and traffic monitoring applications. Theoretically, the integration of NLP-inspired methods into computer vision sets a precedent for future interdisciplinary approaches that aim to tackle complex real-world challenges.

Looking forward, the research opens avenues for further exploration into self-supervised learning frameworks in VAD and other related domains. As transformer-based architectures continue to gain momentum, these methodologies could be fine-tuned and expanded to encompass wider applications, ultimately leading to more intelligent, adaptive, and ethically-aligned AI systems. Future work could explore optimizing the model's latency, scaling to higher-dimensional datasets, and integrating additional data modalities to further enhance anomaly detection capabilities.