ASFormer: Transformer for Action Segmentation (2110.08568v1)

Abstract: Algorithms for the action segmentation task typically use temporal models to predict what action is occurring at each frame for a minute-long daily activity. Recent studies have shown the potential of Transformer in modeling the relations among elements in sequential data. However, there are several major concerns when directly applying the Transformer to the action segmentation task, such as the lack of inductive biases with small training sets, the deficit in processing long input sequence, and the limitation of the decoder architecture to utilize temporal relations among multiple action segments to refine the initial predictions. To address these concerns, we design an efficient Transformer-based model for action segmentation task, named ASFormer, with three distinctive characteristics: (i) We explicitly bring in the local connectivity inductive priors because of the high locality of features. It constrains the hypothesis space within a reliable scope, and is beneficial for the action segmentation task to learn a proper target function with small training sets. (ii) We apply a pre-defined hierarchical representation pattern that efficiently handles long input sequences. (iii) We carefully design the decoder to refine the initial predictions from the encoder. Extensive experiments on three public datasets demonstrate that effectiveness of our methods. Code is available at \url{https://github.com/ChinaYi/ASFormer}.

• The paper introduces ASFormer, which integrates local connectivity inductive biases via temporal convolutions to enhance frame-level action recognition.
• The model uses a hierarchical representation with progressively expanded self-attention receptive fields to efficiently process long video sequences.
• Experimental results on 50Salads, GTEA, and Breakfast datasets demonstrate significant improvements in frame accuracy and segmentation quality.

ASFormer: Transformer for Action Segmentation

This paper presents ASFormer, a Transformer-based model tailored for the action segmentation task, which distinguishes it from the more typical sequence-to-sequence problems like those in natural language processing. Action segmentation requires assigning an action label to each frame of a long, untrimmed video sequence, focusing heavily on the temporal relationships between frames rather than single image content. The authors highlight particular challenges when applying vanilla Transformers directly to this task, such as a lack of inductive biases, inefficiency in handling long sequences, and limitations of the encoder-decoder framework for refining predictions. To address these challenges, ASFormer introduces several innovations.

Key Innovations

1. Local Connectivity Inductive Priors:
   • ASFormer introduces local connectivity inductive biases by incorporating temporal convolution layers. This design leverages the high locality of features inherent in video data, where consecutive frames typically depict the same action, thereby aiding the learning process even with small training datasets.
2. Hierarchical Representation Pattern:
   • To efficiently handle long input sequences, ASFormer employs a hierarchical representation pattern. It constrains each self-attention layer's receptive field to a local window initially, which is gradually expanded in deeper layers. This local-to-global information integration improves convergence and reduces the computational burden on the self-attention layers.
3. Enhanced Decoder Design:
   • The decoder utilizes a cross-attention mechanism derived from initial encoder predictions, facilitating refined iterative predictions without disturbing the learned feature space, important for maintaining prediction accuracy and temporal coherence among action segments.

Experimental Results

The paper reports extensive tests across three standard datasets: 50Salads, GTEA, and Breakfast, showing the model's robustness and state-of-the-art performance in action segmentation tasks. Notable gains in frame accuracy, segmental edit scores, and segmental overlap F1 scores demonstrate the capabilities of ASFormer. For instance, ASFormer outperforms existing methods by delivering smoother predictions and better handling of long sequence dependencies, as noted in the experimental benchmarks.

Implications and Future Directions

ASFormer contributes a significant advancement in leveraging Transformer architectures for video-based action segmentation, marking a crucial shift from more conventional convolutional approaches. The explicit incorporation of local connectivity biases alongside a structured expansion of receptive fields addresses critical data-scarcity issues and computational limitations in long-sequence processing. Future explorations could further enhance ASFormer's scalability or extend its applications into similar domains, such as gesture recognition or complex event detection, where temporal dynamics are equally pivotal. The paper also opens pathways for integrating ASFormer into multimodal frameworks, potentially combining audio and video cues for comprehensive action understanding in real-world scenarios.

In summary, ASFormer stands as an effective adaptation of Transformer techniques for action segmentation, overcoming traditional limitations through thoughtful architectural enhancements. Its successful application demonstrates the benefit of combining innovative model design with tailored inductive biases and efficient hierarchical processing strategies.