Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (1801.07455v2)

Abstract: Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

• The paper introduces ST-GCN, which leverages graph convolution to model skeleton sequences as spatial-temporal graphs for improved human action recognition.
• The approach constructs graphs by designating body joints as nodes and linking them across frames to effectively capture dynamic movements.
• Experimental results on the Kinetics and NTU-RGB+D datasets demonstrate significant accuracy gains compared to traditional skeleton-based methods.

Spatial Temporal Graph CNNs for Skeleton-Based Action Recognition

In the paper "Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition" authored by Sijie Yan, Yuanjun Xiong, and Dahua Lin from The Chinese University of Hong Kong, a novel methodology is presented for recognizing human actions utilizing dynamic skeleton data. This approach seeks to address the inherent limitations found in prior methods by employing the robust capabilities of graph convolutional networks (GCNs) extended into the spatial-temporal domain, denoted as Spatial-Temporal Graph Convolutional Networks (ST-GCN).

Introduction and Motivation

Human action recognition is a pivotal aspect of video understanding with applications spanning surveillance, human-computer interaction, and sports analysis. Existing research has explored various modalities such as appearance, depth, optical flows, and body skeletons. However, modeling dynamic skeletons has historically been constrained by reliance on hand-crafted features or traversal rules, leading to limited expressive power and generalization.

Proposed Method: ST-GCN

The core contribution of this work lies in the development of ST-GCN, which models skeleton sequences as graphs. Each node in these graphs represents a human body joint, while edges are defined spatially based on natural joint connectivity and temporally by linking the same joints across consecutive frames. This approach leverages the inherent structure in human movements and utilizes graph neural networks to capture both spatial configurations and temporal dynamics.

Key aspects of ST-GCN include:

• Graph Construction: Nodes correspond to joints, and edges embody intra-frame joint connectivities and inter-frame temporal relationships.
• Graph Convolution: Inspired by traditional CNNs, the convolution operation is adapted to work on graph structures, maintaining locality by restricting filters to 1-neighbor subsets. This operation aggregates features from neighboring nodes and emphasizes the importance of joint relations.
• Partitioning Strategies: Several strategies were explored for partitioning neighboring nodes, including uniform labeling, distance partitioning, and spatial configuration partitioning. The latter provided the best performance by differentiating the roles of joints in dynamic contexts.

Experimental Evaluation

The performance of ST-GCN was rigorously evaluated on two large-scale datasets, Kinetics and NTU-RGB+D, which present diverse challenges. The results demonstrated substantial improvements over existing skeleton-based methods.

• Kinetics: On this unconstrained dataset derived from YouTube videos, ST-GCN demonstrated significant performance improvements, achieving a top-1 accuracy of 30.7% and a top-5 accuracy of 52.8%. This outperforms traditional feature encoding and other deep learning models like Deep LSTM and Temporal ConvNets.
• NTU-RGB+D: This dataset, captured in a constrained environment, saw ST-GCN excel with top-1 accuracies of 81.5% in cross-subject evaluation and 88.3% in cross-view evaluation, surpassing the state-of-the-art methods such as PA-LSTM, ST-LSTM+TS, and C-CNN+MTLN.

Implications and Future Directions

Theoretical Implications: The ST-GCN framework provided a generic, scalable approach to model dynamic skeletons without dependence on hand-crafted features. These findings underscore the potential of graph-based approaches in capturing complex spatial-temporal dependencies in human actions.

Practical Implications: The adaptability of ST-GCN across different datasets and scenarios showcases its robustness. The incorporation of skeleton data in combination with other modalities like RGB frames and optical flow can enhance the performance of action recognition systems, opening pathways for multi-modal analysis.

Future Developments: Future research could extend ST-GCN by:

1. Exploring different graph convolutional architectures or graph attention mechanisms.
2. Incorporating contextual information such as objects and scenes into the spatial-temporal graphs, enhancing the model's ability to interpret more complex actions.
3. Addressing cross-domain generalization to ensure performance across varied datasets without the need for extensive retraining.

Overall, this paper represents a substantial advancement in skeleton-based action recognition, showcasing the efficacy of spatial-temporal graph convolutional networks and their potential to redefine approaches in this domain.