The THUMOS Challenge on Action Recognition for Videos "in the Wild" (1604.06182v1)

Abstract: Automatically recognizing and localizing wide ranges of human actions has crucial importance for video understanding. Towards this goal, the THUMOS challenge was introduced in 2013 to serve as a benchmark for action recognition. Until then, video action recognition, including THUMOS challenge, had focused primarily on the classification of pre-segmented (i.e., trimmed) videos, which is an artificial task. In THUMOS 2014, we elevated action recognition to a more practical level by introducing temporally untrimmed videos. These also include `background videos' which share similar scenes and backgrounds as action videos, but are devoid of the specific actions. The three editions of the challenge organized in 2013--2015 have made THUMOS a common benchmark for action classification and detection and the annual challenge is widely attended by teams from around the world. In this paper we describe the THUMOS benchmark in detail and give an overview of data collection and annotation procedures. We present the evaluation protocols used to quantify results in the two THUMOS tasks of action classification and temporal detection. We also present results of submissions to the THUMOS 2015 challenge and review the participating approaches. Additionally, we include a comprehensive empirical study evaluating the differences in action recognition between trimmed and untrimmed videos, and how well methods trained on trimmed videos generalize to untrimmed videos. We conclude by proposing several directions and improvements for future THUMOS challenges.

• The paper introduces the THUMOS Challenge benchmark, shifting focus from trimmed to untrimmed videos for realistic action recognition.
• It details dual tasks for action classification and temporal detection, evaluated with metrics such as mAP and IoU thresholds.
• Experimental results show that integrating deep learning with traditional features enhances detection and classification performance in challenging video contexts.

The THUMOS Challenge on Action Recognition for Videos "in the Wild"

The THUMOS Challenge, introduced in 2013, aims to advance the field of action recognition in realistic, unsegmented videos. This competition has played a significant role in shifting focus from the recognition of pre-segmented (or trimmed) videos to untrimmed videos that accurately reflect real-world data. This essay summarizes the given paper, detailing the nuances of THUMOS, its implementation, the dataset, methods, results, and future directions for research.

Overview of the Benchmark and Tasks

The THUMOS challenge provides a comprehensive benchmark for action classification and temporal detection in untrimmed videos. It is split into two principal tasks:

1. Action Classification: This involves predicting whether a particular action appears in a video. It considers each video segment as a binary classification task.
2. Temporal Detection: This requires identifying the temporal boundaries of an action within a video segment, making the task more complex than simple classification.

Introduced to the community with the 2013 THUMOS dataset based on UCF101, THUMOS'14 and subsequent editions incorporated untrimmed videos. The dataset from THUMOS'14 introduced background videos to present more challenging scenarios where the context may be similar but the action may not be present.

Data Collection and Annotation

The dataset, primarily collected from YouTube, incorporates positive and background videos. Positive videos contain the specified action, while background videos resemble the scene but lack the specific action, added to challenge the robustness of the action recognizers. A detailed annotation workflow ensures accuracy and consistency:

• Positive Videos: Videos first filtered using Freebase topics and search keywords, and then manually annotated to confirm the presence of action.
• Background Videos: Manually verified to ensure absence of all 101 actions across various sub-categories.

Temporal annotations are provided for action boundaries, distinguishing between clear and ambiguous instances. These annotations are crucial for temporal detection tasks as they enable precise evaluation.

Evaluation Protocols and Metrics

For both tasks, the evaluation protocols utilize precision metrics:

• Action Classification: Evaluated using mean Average Precision (mAP) over all classes, reflecting the accuracy of the classifier across multiple classes.
• Temporal Detection: The mAP evaluates the intersection-over-union (IoU) of detected time intervals with actual action intervals at multiple thresholds, enforcing robustness in the detection accuracy.

Methods and Approaches

In THUMOS’15, various methods were employed by different teams for both classification and detection tasks. A trend towards using deep learning, particularly Convolutional Neural Networks (CNN), emerged. Complementing these, traditional features like Improved Dense Trajectories (iDT) also remained prevalent.

Classification Methods

Deep learning features derived from networks like VGGNet, GoogleNet, and the two-stream CNN model were widely used. Techniques such as Vector of Locally Aggregated Descriptors (VLAD) pooling enhanced the representations extracted by these networks. Fusion techniques, including average fusion and logistic regression fusion, were employed to combine multiple features effectively.

Temporal Detection

Temporal detection methods relied heavily on iDT features encoded with Fisher Vectors. Sliding window techniques and different pooling strategies helped optimize the detection of action intervals. Context features (background information) were evaluated separately and combined with action features to assess their influence on detection performance.

Experimental Results

Action Classification

Experiments showed that classifiers trained on trimmed videos experienced a significant drop in performance when tested on untrimmed videos if no adjustments were made. However, using sliding window approaches with max or average pooling preserved performance, illustrating a robust adaptation to untrimmed data. The importance of context was reaffirmed, with models trained with both action and context representations outperforming those trained on action alone.

Temporal Detection

Temporal detection experiments highlighted that using appropriate window lengths and incorporating context information improved mAP scores. Temporal non-maximum suppression helped refine detection results.

Implications and Future Directions

The THUMOS benchmark has clearly influenced the landscape of video action recognition. Several future directions were suggested to expand this research area:

• Inclusive Datasets: Expanding the dataset to include a broader range of action categories and ensuring higher diversity within action instances.
• Deep Semantic Understanding: Moving beyond detection to semantically rich annotations involving objects, actions, scenes, attributes, and their interrelationships.
• Weakly Supervised Learning: Emphasizing scenarios where training data lacks frame-level annotations, pushing for models that can infer actions from less detailed, more realistic data scenarios.
• Textual Summaries and Q&A: Integrating textual descriptions and generating Q&A pairs to assess comprehensive understanding.

Conclusion

By introducing untrimmed videos and challenging background contexts, THUMOS has pushed the envelope for action recognition in naturalistic settings. The comprehensive datasets, coupled with detailed annotations and robust evaluation protocols, provide a strong foundation for advancing both theoretical understanding and practical applications of action recognition in videos. Future iterations of THUMOS promise to integrate more sophisticated and semantically rich datasets, driving forward the capabilities of machine learning models in understanding and interpreting human actions in video data.