Long Movie Clip Classification with State-Space Video Models (2204.01692v3)

Abstract: Most modern video recognition models are designed to operate on short video clips (e.g., 5-10s in length). Thus, it is challenging to apply such models to long movie understanding tasks, which typically require sophisticated long-range temporal reasoning. The recently introduced video transformers partially address this issue by using long-range temporal self-attention. However, due to the quadratic cost of self-attention, such models are often costly and impractical to use. Instead, we propose ViS4mer, an efficient long-range video model that combines the strengths of self-attention and the recently introduced structured state-space sequence (S4) layer. Our model uses a standard Transformer encoder for short-range spatiotemporal feature extraction, and a multi-scale temporal S4 decoder for subsequent long-range temporal reasoning. By progressively reducing the spatiotemporal feature resolution and channel dimension at each decoder layer, ViS4mer learns complex long-range spatiotemporal dependencies in a video. Furthermore, ViS4mer is $2.63\times$ faster and requires $8\times$ less GPU memory than the corresponding pure self-attention-based model. Additionally, ViS4mer achieves state-of-the-art results in $6$ out of $9$ long-form movie video classification tasks on the Long Video Understanding (LVU) benchmark. Furthermore, we show that our approach successfully generalizes to other domains, achieving competitive results on the Breakfast and the COIN procedural activity datasets. The code is publicly available at: https://github.com/md-mohaiminul/ViS4mer.

Introduction to ViS4mer

A significant challenge in computer vision is understanding long movie clips. To tackle this, a new computational framework known as ViS4mer has been introduced, which is designed to classify long-form video content efficiently. Unlike traditional video recognition models that handle short videos typically up to 10 seconds in duration, ViS4mer is built specifically for complex tasks such as classifying relationships among characters, predicting the genre, and more, in videos that are much longer.

ViS4mer Architecture

ViS4mer melds the advantages of self-attention mechanisms found in Transformers with the computational efficiency of state-space sequence models, notably the structured state-space sequence (S4) layer. This approach results in reduced computational cost and lower memory usage while retaining the ability to consider the broad temporal context within videos.

The architecture employs a standard Transformer encoder for extracting spatial features and an innovative temporal S4 decoder that operates at multiple scales. By progressively scaling down the resolution and channel dimensions, ViS4mer effectively compresses video information. It maintains hierarchy in the learned spatiotemporal features, crucial for understanding the extended context within long clips.

Performance and Efficiency

When tested on the Long Video Understanding (LVU) benchmark, ViS4mer demonstrated remarkable proficiency, achieving state-of-the-art results on six of nine tasks. It has been shown to process video data 2.63 times faster and requires eight times less GPU memory compared to models relying solely on self-attention. Moreover, a transition away from self-attention also enables the model to maintain performance even as the length of the input sequence increases.

Generalization and Applications

Beyond movie understanding, ViS4mer's utility extends to other long-range video domains. It has been evaluated on procedural activity datasets such as Breakfast and COIN, showcasing competitive results. This suggests its applicability can extend to numerous real-world scenarios, offering a versatile tool for analyzing extensive video content in various contexts.

In sum, ViS4mer presents an efficient, powerful, and scalable framework to push the boundaries of long-form video understanding, offering compelling prospects for future advancements in computer vision and machine learning.