A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization (2101.00545v3)

Abstract: Weakly supervised temporal action localization is a challenging vision task due to the absence of ground-truth temporal locations of actions in the training videos. With only video-level supervision during training, most existing methods rely on a Multiple Instance Learning (MIL) framework to predict the start and end frame of each action category in a video. However, the existing MIL-based approach has a major limitation of only capturing the most discriminative frames of an action, ignoring the full extent of an activity. Moreover, these methods cannot model background activity effectively, which plays an important role in localizing foreground activities. In this paper, we present a novel framework named HAM-Net with a hybrid attention mechanism which includes temporal soft, semi-soft and hard attentions to address these issues. Our temporal soft attention module, guided by an auxiliary background class in the classification module, models the background activity by introducing an "action-ness" score for each video snippet. Moreover, our temporal semi-soft and hard attention modules, calculating two attention scores for each video snippet, help to focus on the less discriminative frames of an action to capture the full action boundary. Our proposed approach outperforms recent state-of-the-art methods by at least 2.2% mAP at IoU threshold 0.5 on the THUMOS14 dataset, and by at least 1.3% mAP at IoU threshold 0.75 on the ActivityNet1.2 dataset. Code can be found at: https://github.com/asrafulashiq/hamnet.

• The paper introduces HAM-Net, a hybrid attention mechanism that integrates soft, semi-soft, and hard attention modules to capture complete action boundaries.
• It overcomes traditional MIL limitations by modeling background activities and reducing over-reliance on the most discriminative snippets.
• Experimental results show HAM-Net achieves over 2.2% mAP improvement on THUMOS14 and notable gains on ActivityNet1.2.

Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localization

This paper introduces a novel approach to addressing the challenging task of Weakly-Supervised Temporal Action Localization (WTAL) using a framework named HAM-Net. The primary objective of WTAL is to predict the temporal boundaries of actions within a video using only video-level labels during the training phase. The absence of frame-level annotations makes this task particularly challenging, as traditional methods rely heavily on strong supervision.

Motivation and Background

Traditional WTAL methods often employ a Multiple Instance Learning (MIL) framework. While MIL is effective at identifying discriminative frames where an action is highly evident, it struggles with capturing the entire temporal extent of actions. Moreover, these methods inadequately model background activities, which are crucial for accurate foreground action localization. The paper identifies two key shortcomings of the MIL-based approach: (1) the tendency to focus only on the most discriminative snippets, thereby ignoring complete action boundaries; and (2) inefficiencies in modeling context and background, leading to incorrect action boundary predictions.

The HAM-Net Framework

HAM-Net introduces a hybrid attention mechanism to combat the identified limitations of traditional MIL strategies. The core innovation involves the utilization of three types of temporal attention modules: soft, semi-soft, and hard attentions.

1. Temporal Soft Attention: This module is informed by an auxiliary background class to adequately model background activities. It assigns an "action-ness" score to each snippet, helping the model to differentiate between action and non-action segments.
2. Semi-Soft and Hard Attentions: These modules focus on overcoming the partial attention issue. The semi-soft attention drops highly discriminative snippets, maintaining focus on less obvious, yet important frames to capture complete action boundaries. The hard attention provides an additional layer of scrutiny by assigning binary attention values, effectively creating a map of less discriminative segments which includes both foreground and background snippets.

Performance and Evaluation

The effectiveness of HAM-Net is demonstrated on two prominent datasets: THUMOS14 and ActivityNet1.2. HAM-Net outperforms existing state-of-the-art WTAL methods, achieving at least a 2.2% improvement in mean Average Precision (mAP) at an Intersection over Union (IoU) threshold of 0.5 on THUMOS14, and at least a 1.3% mAP improvement at an IoU threshold of 0.75 on ActivityNet1.2. These results underscore the robustness of HAM-Net in detecting both start and end boundaries of action instances within complex and untrimmed videos.

Implications and Future Directions

The introduction of a hybrid attention mechanism constitutes a significant enhancement in the field of WTAL, fostering improved action boundary detection. Practically, this can enhance a range of applications including video analytics, content moderation, and surveillance systems where precise action detection with limited annotation cost is critical.

Theoretically, the concept of hybrid attention can be extended and adapted to other domains where delineating boundary information is crucial, such as in temporal segmentation tasks or event boundary detection in natural language processing. Future research may explore the integration of HAM-Net with Transformer-based models to enhance its capability further, or employ unsupervised learning paradigms to minimize supervision even more.

In conclusion, the proposed HAM-Net framework provides a sophisticated solution to a complex problem in video analysis, demonstrating significant improvement over existing methods by effectively leveraging multiple attention mechanisms to capture the entirety of action instances with limited supervision.