Skeleton-Based Human Action Recognition with Global Context-Aware Attention LSTM Networks (1707.05740v5)

Abstract: Human action recognition in 3D skeleton sequences has attracted a lot of research attention. Recently, Long Short-Term Memory (LSTM) networks have shown promising performance in this task due to their strengths in modeling the dependencies and dynamics in sequential data. As not all skeletal joints are informative for action recognition, and the irrelevant joints often bring noise which can degrade the performance, we need to pay more attention to the informative ones. However, the original LSTM network does not have explicit attention ability. In this paper, we propose a new class of LSTM network, Global Context-Aware Attention LSTM (GCA-LSTM), for skeleton based action recognition. This network is capable of selectively focusing on the informative joints in each frame of each skeleton sequence by using a global context memory cell. To further improve the attention capability of our network, we also introduce a recurrent attention mechanism, with which the attention performance of the network can be enhanced progressively. Moreover, we propose a stepwise training scheme in order to train our network effectively. Our approach achieves state-of-the-art performance on five challenging benchmark datasets for skeleton based action recognition.

• The paper presents a global context-aware attention mechanism in LSTM networks that isolates informative joints for improved action recognition.
• It employs a two-stream framework integrating fine-grained joint and coarse-grained body part attention to capture both spatial and temporal dependencies.
• Experimental results on datasets like NTU RGB+D demonstrate state-of-the-art performance and robust noise resistance.

Skeleton Based Human Action Recognition with Global Context-Aware Attention LSTM Networks

The paper presents a novel approach to skeleton-based human action recognition by introducing the Global Context-Aware Attention Long Short-Term Memory (GCA-LSTM) Networks. This architecture aims to enhance the attention capabilities of LSTM networks in processing 3D skeleton sequences by leveraging global contextual information to focus on informative skeletal joints.

Introduction and Motivation

In the domain of action recognition, accurately identifying human activities through 3D skeleton data holds significant importance due to its applications in surveillance, healthcare, and human-robot interaction. The LSTM network, known for managing sequential data due to its gate-based architecture, lacks explicit attention mechanisms needed for focusing on key joints that are most informative for a given action sequence. Informative joints offer significant cues while irrelevant ones may introduce noise, thereby necessitating the development of enhanced attention mechanisms.

GCA-LSTM Architecture

The GCA-LSTM aims to address this deficiency by integrating a global context memory cell that aids in iteratively refining attention across multiple iterations, thereby dynamically focusing on the most relevant joints.

1. Architecture Overview:
   • The network utilizes two layers of ST-LSTM (Spatio-Temporal LSTM) to encapsulate spatial dependencies among joints within a frame and temporal dependencies across frames.
   • A global context memory is maintained to store a representation of the entire action sequence, which feeds into every step to assist in attention scoring and refinement.
2. Attention Mechanism:
   • A recurrent attention mechanism allows iterative updates to the global context memory by computing attention scores that guide the focus on vital joints using both spatial and temporal characteristics.
   • Informativeness scores are computed at each step based on previous global context memory, aiding in filtering noise from irrelevant joints.
3. Two-Stream Framework:
   • The paper additionally presents a two-stream attention strategy, combining joint-level (fine-grained) attention with body part-level (coarse-grained) attention.
   • This dual attention approach enhances the recognition performance by also considering the holistic movements of body parts.

Experimental Results

Extensive experimentation across five challenging datasets, including NTU RGB+D and SYSU-3D, showcases the efficacy of the proposed approach. The GCA-LSTM demonstrates superior performance by achieving state-of-the-art results, with notable accuracy improvements attributed to its ability to concentrate on pertinent features via the context-aware attention mechanism.

1. Training Methodology:
   • Direct versus stepwise training methods were evaluated, revealing that a stepwise approach mitigates overfitting and accelerates convergence by incrementally optimizing network parameters.
2. Robustness:
   • The model exhibits robustness against noise in input data, as demonstrated through trials involving synthetic noise additions.
3. Comparative Performance:
   • The GCA-LSTM outperforms existing methods by substantial margins, confirming the advantage of integrating global context into the action recognition models.

Implications and Future Directions

The introduction of global context-aware attention manifests significant theoretical and practical implications. Theoretically, it presents a method to effectively incorporate long-range dependencies and dynamic attention adjustments into LSTM networks. Practically, the model's robustness and heightened accuracy constitute a promising advance for real-world applications.

Future research could explore extending these mechanisms into more complex hierarchical models or applying them to other types of data sequences, such as those in natural language processing, where context-aware attention may similarly enhance performance.

In summary, the paper successfully addresses the limitations of conventional LSTM networks for action recognition tasks by proposing an innovative architecture that leverages global context-aware attention, thereby setting a new performance benchmark in the field of skeleton-based action recognition.