View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition (1804.07453v3)

Abstract: Skeleton-based human action recognition has recently attracted increasing attention thanks to the accessibility and the popularity of 3D skeleton data. One of the key challenges in skeleton-based action recognition lies in the large view variations when capturing data. In order to alleviate the effects of view variations, this paper introduces a novel view adaptation scheme, which automatically determines the virtual observation viewpoints in a learning based data driven manner. We design two view adaptive neural networks, i.e., VA-RNN based on RNN, and VA-CNN based on CNN. For each network, a novel view adaptation module learns and determines the most suitable observation viewpoints, and transforms the skeletons to those viewpoints for the end-to-end recognition with a main classification network. Ablation studies find that the proposed view adaptive models are capable of transforming the skeletons of various viewpoints to much more consistent virtual viewpoints which largely eliminates the viewpoint influence. In addition, we design a two-stream scheme (referred to as VA-fusion) that fuses the scores of the two networks to provide the fused prediction. Extensive experimental evaluations on five challenging benchmarks demonstrate that the effectiveness of the proposed view-adaptive networks and superior performance over state-of-the-art approaches. The source code is available at https://github.com/microsoft/View-Adaptive-Neural-Networks-for-Skeleton-based-Human-Action-Recognition.

• The paper presents view adaptation modules integrated in VA-RNN and VA-CNN that automatically adjust observation viewpoints to boost recognition accuracy.
• It demonstrates enhanced performance through end-to-end training and a fusion strategy, achieving up to 95.0% accuracy on NTU RGB+D cross-view protocols.
• The approach incorporates data augmentation with random skeleton rotation to mitigate viewpoint variations, ensuring robust recognition in practical settings.

Overview of View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition

The paper "View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition" presents a significant advancement in the field of computer vision, specifically within the domain of skeleton-based human action recognition. Human action recognition is an essential aspect of computer vision, and it finds applications in a variety of areas, including video surveillance, human-computer interaction, and video analysis. However, one of the primary challenges in human action recognition is the variation in action representations when actions are captured from different viewpoints, which can significantly affect recognition performance. This paper introduces a novel approach to tackle this issue effectively.

Key Contributions

The paper proposes a view adaptation framework that includes two specifically designed neural network architectures: View-Adaptive Recurrent Neural Network (VA-RNN) and View-Adaptive Convolutional Neural Network (VA-CNN). These networks aim to improve skeleton-based action recognition by automatically adjusting the observation viewpoint in a data-driven manner.

1. View Adaptation Module: Instead of relying on pre-defined criteria for viewpoint alignment, the authors introduce a view adaptation module. This module is integrated into both VA-RNN and VA-CNN. It learns to determine optimal virtual observation viewpoints over the sequence of actions and subsequently transforms the skeleton data into these views, thus facilitating efficient feature learning by reducing intra-class variations due to diverse viewpoints.
2. End-to-End Training: The proposed networks are designed for end-to-end training, optimizing both the view adaptation and the main classification networks concurrently to enhance action recognition performance.
3. Two-Stream Fusion: The authors introduce a VA-fusion scheme that combines the predictions from VA-RNN and VA-CNN, leveraging the strengths of both architectures to achieve an even higher performance in action recognition tasks.
4. Data Augmentation for Robustness: The method incorporates a novel use of data augmentation techniques where random rotation of skeleton sequences is employed during training to improve model robustness and alleviate overfitting.

Numerical Results and Implications

The results presented are extensively evaluated on five challenging datasets, including NTU RGB+D, SYSU Human-Object Interaction, UWA3D, Northwestern-UCLA, and SBU Kinect Interaction datasets. The proposed approach demonstrates superior performance compared to state-of-the-art methods, significantly improving accuracy across these datasets. For instance, on the NTU RGB+D dataset, the VA-fusion achieved a noteworthy improvement over previous methods with a recognition accuracy of 89.4% and 95.0% for cross-subject and cross-view protocols, respectively.

These results signify the effectiveness of the view adaptation module in mitigating the effects of viewpoint variations, leading to improved learning of action-specific features and consequently enhancing recognition performance. The work paves the way for developing more generalized and robust models in skeleton-based action recognition, expanding its practical applicability.

Speculations on Future Developments and Practical Applications

Practically, this research can considerably impact environments where skeleton-based recognition is used, such as real-time surveillance systems, interactive gaming, and gesture-based control systems. The framework provides flexibility in recognizing actions from varying perspectives and can be adapted for more complex activities involving multi-person interactions.

Theoretically, the proposed view adaptation mechanisms can inspire further research into integrating viewpoint adaptation into different modalities beyond skeleton data, such as integrating it with RGB video data or depth sensor inputs. Future work might explore extending the current models to handle dynamic changes in the environment, such as variable lighting or occlusions, further, increasing their utility in real-world scenarios.

Overall, the proposed approach represents a significant step towards overcoming one of the critical challenges in action recognition by providing a methodology to automatically adapt observation viewpoints, thereby facilitating the development of more accurate and reliable human action recognition systems.