Skeleton-based Group Activity Recognition via Spatial-Temporal Panoramic Graph (2407.19497v2)

Abstract: Group Activity Recognition aims to understand collective activities from videos. Existing solutions primarily rely on the RGB modality, which encounters challenges such as background variations, occlusions, motion blurs, and significant computational overhead. Meanwhile, current keypoint-based methods offer a lightweight and informative representation of human motions but necessitate accurate individual annotations and specialized interaction reasoning modules. To address these limitations, we design a panoramic graph that incorporates multi-person skeletons and objects to encapsulate group activity, offering an effective alternative to RGB video. This panoramic graph enables Graph Convolutional Network (GCN) to unify intra-person, inter-person, and person-object interactive modeling through spatial-temporal graph convolutions. In practice, we develop a novel pipeline that extracts skeleton coordinates using pose estimation and tracking algorithms and employ Multi-person Panoramic GCN (MP-GCN) to predict group activities. Extensive experiments on Volleyball and NBA datasets demonstrate that the MP-GCN achieves state-of-the-art performance in both accuracy and efficiency. Notably, our method outperforms RGB-based approaches by using only estimated 2D keypoints as input. Code is available at https://github.com/mgiant/MP-GCN

• The paper introduces a Multi-Person Panoramic Graph Convolutional Network (MP-GCN) that unifies human and object keypoints to improve group activity recognition.
• It demonstrates state-of-the-art performance with benchmarks like 96.2% accuracy on the Volleyball dataset and reduced computational cost.
• The approach fuses low-level skeletal features through a spatial-temporal convolution framework, addressing challenges of occlusion and background variation.

A Comprehensive Evaluation of Skeleton-based Group Activity Recognition with Spatial-Temporal Panoramic Graphs

The paper presents a novel approach to Group Activity Recognition (GAR) leveraging skeleton-based methods, which proposes a Spatial-Temporal Panoramic Graph to enhance the recognition performance. Existing methods heavily rely on the RGB modality, which encounters challenges like occlusion and background variation. By contrast, extracting keypoint information from human poses and integrating object keypoints significantly reduces computational overhead and enhances accuracy.

Research Contributions

The primary contribution of this paper is the introduction of a Multi-Person Panoramic Graph Convolutional Network (MP-GCN), which unifies intra-person, inter-person, and person-object relationship modeling through a spatial-temporal graph convolution framework. This analytic strategy addresses three critical gaps in previous methods:

1. Graph Structure Improvement: The paper advocates for the development of panoramic graphs, integrating both human and object keypoints in a single holistic framework. This graph structure not only compensates for the absence of objects in conventional skeleton data but also resolves the inadequacies in shared weight handling and inter-person modeling. The panoramic graph configuration captures complex human-object interactions, enhancing the feature extraction capabilities far beyond the limitations of traditional single-person skeletal graphs.
2. Efficiency and Performance Benchmarks: The proposed MP-GCN attains state-of-the-art performance on widely used datasets, including Volleyball, NBA, and Kinetics400. The research demonstrates that the MP-GCN approach outperforms existing RGB and pose-only based GAR methods. Particularly, performance metrics on the Volleyball dataset, achieving 96.2% Multi-class Classification Accuracy (MCA) and 84.6% Individual Mean Classification Accuracy (IMCA), validate its efficacy in both fully and weakly supervised settings.
3. Modular Network Architecture: Through early fusion of low-level features derived from joint, bone, joint motion, and bone motion inputs, followed by a hierarchical structure of graph convolution and temporal convolution networks, MP-GCN maintains performance with fewer parameters, showcasing robust efficiency and reduced computational cost.

Methodology Insights

The method begins with pose estimation, leveraging advanced tracking algorithms to capture skeleton dynamics. By integrating this data into a panoramic multi-person-object graph, the authors employ structural GCN to encapsulate spatial-temporal features effectively. This approach allows the simultaneous modeling of multiple participant interactions, facilitated by a rigorous intra-inter partitioning strategy.

Further, the research delineates a sophisticated tracking-based reassignment strategy to optimize identity assignments, ensuring consistently high data quality across frames, which mitigates common issues such as miss detection.

Implications and Potential Advances

This work has significant implications for practical applications in surveillance, sports analysis, and complex event understanding. Its robust performance under various conditions highlights the potential for robotics and AI systems to interpret dynamic human environments with greater contextual awareness.

Future avenues of exploration could involve enhancing object dynamic representations within the panoramic graph for real-time applications and extending scalability to larger groups. Furthermore, the integration of attention mechanisms for enhanced focus on vital interactions and roles within group activities presents another promising direction.

Conclusion

This paper contributes to the GAR domain by overcoming substantial limitations in existing models. By shifting from RGB-heavy approaches to an efficient skeleton-based model integrating keypoint-rich representations, the researchers broaden the scope and applicability of group activity recognition technology. The proposed MP-GCN model offers clear advantages in both recognition accuracy and computational efficiency, suggesting that such an integrated graph-based approach will play a pivotal role in the evolution of GAR systems.