TriDet: Temporal Action Detection with Relative Boundary Modeling (2303.07347v2)

Abstract: In this paper, we present a one-stage framework TriDet for temporal action detection. Existing methods often suffer from imprecise boundary predictions due to the ambiguous action boundaries in videos. To alleviate this problem, we propose a novel Trident-head to model the action boundary via an estimated relative probability distribution around the boundary. In the feature pyramid of TriDet, we propose an efficient Scalable-Granularity Perception (SGP) layer to mitigate the rank loss problem of self-attention that takes place in the video features and aggregate information across different temporal granularities. Benefiting from the Trident-head and the SGP-based feature pyramid, TriDet achieves state-of-the-art performance on three challenging benchmarks: THUMOS14, HACS and EPIC-KITCHEN 100, with lower computational costs, compared to previous methods. For example, TriDet hits an average mAP of $69.3\%$ on THUMOS14, outperforming the previous best by $2.5\%$, but with only $74.6\%$ of its latency. The code is released to https://github.com/sssste/TriDet.

• The paper introduces a novel Trident-head that models action boundaries via relative probability distributions for precise temporal localization.
• It deploys a Scalable-Granularity Perception layer to improve feature discrimination while minimizing computational overhead.
• TriDet achieves state-of-the-art performance on benchmarks like THUMOS14 and HACS, boosting mAP and reducing latency compared to prior methods.

An Examination of TriDet: Temporal Action Detection with Relative Boundary Modeling

The paper "TriDet: Temporal Action Detection with Relative Boundary Modeling" introduces a novel approach to the task of Temporal Action Detection (TAD) in untrimmed videos. This task requires precise localization and categorization of actions within videos, where action boundaries are often ambiguous. TriDet is presented as a one-stage framework that aims to mitigate this challenge through innovative boundary modeling techniques and efficient feature processing.

Core Contributions

1. Trident-head for Boundary Localization: The foundation of TriDet is its Trident-head, which departs from conventional boundary prediction strategies. Unlike methods that rely on segment-level global features or instant-level regressions, the Trident-head models action boundaries via a relative probability distribution approach. It uses start, end, and center-offset heads that estimate relative probabilities across adjacent time bins, which contributes to more reliable boundary detection. This approach leverages statistical modeling to enhance boundary localization accuracy, offering superior performance at various Intersection over Union (IoU) thresholds.
2. Scalable-Granularity Perception (SGP) Layer: Another key innovation is the SGP layer, proposed within the TriDet's feature pyramid network. The SGP layer addresses the rank loss problem plaguing self-attention mechanisms where inter-snippet feature similarity is high. By replacing self-attention with a convolutional-based architecture that includes an instant-level branch and a window-level branch, the SGP layer enhances feature discrimination and reduces computational overhead. This results in an improved temporal feature representation that efficiently aggregates information from different temporal granularities.
3. Competitive Performance: TriDet demonstrates state-of-the-art performance across multiple challenging benchmarks—THUMOS14, HACS, and EPIC-KITCHEN 100—achieving significant improvements in mean Average Precision (mAP) while maintaining lower computational costs compared to existing methods. For instance, on the THUMOS14 dataset, TriDet achieves an average mAP of 69.3%, outperforming previous best methods by 2.5% with a latency reduction to 74.6%. These empirical results underscore the effectiveness of TriDet's architectural enhancements and boundary modeling strategies.

Implications and Future Directions

The implications of this work are both practical and theoretical. Practically, TriDet can be readily employed in various applications, from video surveillance to sports analysis, where temporal precision is critical. Theoretically, the introduction of relative boundary modeling expands the toolkit for tackling fuzzy temporal boundaries, which could inspire further advancements in modeling tactics in TAD and related fields.

One potential direction for future research is to explore the adaptability of TriDet's architectural components, such as the Trident-head and SGP layer, across different modalities and with other types of temporal data (e.g., sensor data, audio streams). Additionally, deeper investigation into how these components interact with newer backbone networks may yield further improvements in efficiency and accuracy.

Overall, this paper provides a compelling approach to Temporal Action Detection, offering insights that could reshape prevailing methods for handling the inherent complexities of video-based temporal localization tasks.