Temporal Convolutional Networks for Action Segmentation and Detection (1611.05267v1)

Abstract: The ability to identify and temporally segment fine-grained human actions throughout a video is crucial for robotics, surveillance, education, and beyond. Typical approaches decouple this problem by first extracting local spatiotemporal features from video frames and then feeding them into a temporal classifier that captures high-level temporal patterns. We introduce a new class of temporal models, which we call Temporal Convolutional Networks (TCNs), that use a hierarchy of temporal convolutions to perform fine-grained action segmentation or detection. Our Encoder-Decoder TCN uses pooling and upsampling to efficiently capture long-range temporal patterns whereas our Dilated TCN uses dilated convolutions. We show that TCNs are capable of capturing action compositions, segment durations, and long-range dependencies, and are over a magnitude faster to train than competing LSTM-based Recurrent Neural Networks. We apply these models to three challenging fine-grained datasets and show large improvements over the state of the art.

• The paper introduces TCNs that effectively capture long-range dependencies and deliver robust performance in fine-grained video action segmentation and detection.
• It details two variants: ED-TCN with an encoder-decoder structure for efficient computation and Dilated TCN with dilated convolutions for a large receptive field.
• Empirical results on 50 Salads, MERL Shopping, and GTEA datasets show significant improvements in F1 score, accuracy, and segmentation precision over RNN models.

Temporal Convolutional Networks for Action Segmentation and Detection

Introduction

The paper introduces Temporal Convolutional Networks (TCNs), a novel class of temporal models designed for fine-grained action segmentation and detection in videos. This research is significant for applications in fields such as robotics, surveillance, and education, where understanding human actions over time is crucial. Current methodologies often split the task into two stages: extracting local spatiotemporal features from video frames and then utilizing a high-level temporal classifier. This approach, however, struggles with capturing long-range temporal dependencies and the detailed structure within individual action segments.

Temporal Convolutional Networks

TCNs address these limitations through a hierarchical structure of temporal convolutions. The paper presents two variants of TCNs:

1. Encoder-Decoder TCN (ED-TCN): This model uses a sequence of temporal convolutions, pooling, and upsampling layers. It effectively captures long-range dependencies with fewer layers but longer convolutional filters, making it efficient in terms of training time and memory usage.
2. Dilated TCN: Adapted from the WaveNet model for speech processing, this variant employs dilated convolutions and skip connections. Dilated convolutions allow the model to have a large receptive field with a deep stack of layers, effectively capturing complex temporal patterns.

Both TCNs are designed to perform computations simultaneously across all time-steps in a layer, rather than sequentially frame-by-frame, resulting in significant computational efficiency gains compared to traditional Recurrent Neural Networks (RNNs).

Experimental Evaluation

The authors evaluate TCNs on three challenging fine-grained datasets:

1. 50 Salads: This dataset consists of videos of users making salads, characterized by long durations and numerous action instances. The results demonstrate that TCNs significantly outperformed RNN-based models and other state-of-the-art methods, particularly in reducing over-segmentation errors and capturing long-range temporal dependencies.
2. MERL Shopping: This dataset involves action detection in a retail environment. Both causal (only past information) and acausal (past and future information) versions of TCNs were tested. The results show superior performance of TCNs over the baseline methods, with the acausal models providing the most accurate segmentations.
3. Georgia Tech Egocentric Activities (GTEA): An egocentric view dataset where users perform kitchen activities. TCNs again provided superior performance, highlighting their robustness across different types of video data and action granularities.

Key Metrics and Results

The paper uses various metrics for evaluating the models, including frame-wise accuracy, segmental F1 score, and mean Average Precision (mAP) with different criteria. Notably, the segmental F1 score introduced by the authors effectively captures both segmentation and detection performance, penalizing over-segmentation and accounting for minor temporal shifts:

• On the 50 Salads dataset, the ED-TCN achieved an F1 score of 76.5 (F1@10), and an accuracy of 73.4%, outperforming both Bi-LSTM and other state-of-the-art models.
• On the MERL Shopping dataset, the ED-TCN achieved an F1 score of 86.7 (F1@10), significantly higher than competing models.
• On the GTEA dataset, the ED-TCN achieved an F1 score of 72.2 (F1@10) with a frame-wise accuracy of 64.0%.

Implications and Future Directions

TCNs provide a powerful alternative to traditional RNNs for action segmentation and detection. The hierarchical convolutional approach effectively captures complex temporal patterns and long-range dependencies. The significant improvement in training speed further emphasizes the practical advantages of TCNs in terms of scalability and efficiency. Future research could explore the integration of TCNs with more sophisticated feature extraction methods and their application to other domains requiring temporal analysis, such as speech recognition or financial time-series forecasting.

By demonstrating the efficacy of TCNs in various benchmark datasets, the paper paves the way for further exploration and optimization of convolutional approaches to temporal modeling, potentially transforming methodologies in time-series analysis.