RGB-D-based Action Recognition Datasets: A Survey (1601.05511v1)

Abstract: Human action recognition from RGB-D (Red, Green, Blue and Depth) data has attracted increasing attention since the first work reported in 2010. Over this period, many benchmark datasets have been created to facilitate the development and evaluation of new algorithms. This raises the question of which dataset to select and how to use it in providing a fair and objective comparative evaluation against state-of-the-art methods. To address this issue, this paper provides a comprehensive review of the most commonly used action recognition related RGB-D video datasets, including 27 single-view datasets, 10 multi-view datasets, and 7 multi-person datasets. The detailed information and analysis of these datasets is a useful resource in guiding insightful selection of datasets for future research. In addition, the issues with current algorithm evaluation vis-\'{a}-vis limitations of the available datasets and evaluation protocols are also highlighted; resulting in a number of recommendations for collection of new datasets and use of evaluation protocols.

• The paper categorizes 44 RGB-D datasets into three groups (single-view, multi-view, and multi-person) to clarify dataset configurations.
• It identifies challenges such as dataset saturation, limited diversity, and simplistic environments that hinder algorithm robustness.
• The study recommends enlarging dataset scope and adopting cross-dataset evaluations to promote realistic and generalizable action recognition.

A Comprehensive Survey of RGB-D-based Action Recognition Datasets

The paper "RGB-D-based Action Recognition Datasets: A Survey" provides a detailed assessment of the evolution, current state, and future directions of RGB-D action recognition datasets. It aims to encapsulate the essence of 44 datasets within the RGB-D domain by categorizing them into three distinct groups based on the number of perspectives captured: single-view, multi-view, and multiple-person interaction datasets.

Overview and Dataset Classification

The categorization employed in this paper facilitates a deeper understanding of dataset configurations:

1. Single-view Datasets: Comprising 27 datasets, these capture actions from a fixed angle. Key datasets like the MSR-Action3D highlight the inception of this field. The standard evaluation protocols associated with these datasets include Cross Subject (CS), Leave One Subject Out (LOSubO), and various cross-validation (CV) methodologies.
2. Multi-view Datasets: These 10 datasets offer a multi-perspective approach, ensuring robustness across varied camera angles. Notable examples include the Berkeley MHAD, which stands out with its multimodal capture setup, accommodating RGB-D, motion capture, and even audio modalities. The varying view angles bolster algorithm testing for real-world applications where viewpoint invariance is critical.
3. Multi-person Interaction Datasets: These datasets, including six comprehensive collections like the SBU Kinect Interaction, are centered around collaborative or interaction-based human activities. They capture intricate, nuanced interactive behaviors requiring advanced recognition algorithms capable of handling overlapping actions and multiple actors simultaneously.

Key Insights and Challenges

The paper outlines several essential insights and the overarching challenges within the RGB-D dataset landscape:

• Applicability and Action Diversity: With most datasets crafted for specific real-world applications, the paper urges the necessity of expanding dataset diversity. Many collections remain restricted, either by the type of actions represented or by their small sample sizes, limiting their utility in training generalized AI models.
• Environmental Complexity: The datasets vary significantly in their environmental complexity. Some contain factors like occlusion, cluttered backgrounds, and diverse execution styles, while others remain considerably simplistic. This discrepancy affects the robustness and the generalization capabilities of algorithm evaluations.
• Saturation and Evaluation Protocols: In multiple cases, datasets have reached a saturation point where algorithms achieve near-perfect accuracy. This saturation tends to obscure genuine algorithm improvements and underscores an essential need for standardized and rigorous evaluation protocols. The paper critiques existing practices and advocates for cross-dataset evaluation frameworks.

Recommendations for Future Research

The authors present several forward-looking recommendations aimed at overcoming the identified challenges:

• Enlarged Dataset Scope: Future datasets should aim to integrate varied action categories and encompass larger sample pools, enhancing their applicability and reliability.
• Cross-dataset Evaluation Protocols: Promoting the usage of cross-dataset evaluations can mitigate overfitting to specific environmental conditions and viewpoint setups, leading to broader algorithm robustness.
• Increased Environmental Complexity and Realism: To bridge the gap between controlled and real-world scenarios, dataset creation should simulate more realistic environments featuring dynamic backgrounds, non-fixed camera setups, and extensive interacting entities.

Conclusion and Implications

This survey presents an exhaustive examination of RGB-D-based action recognition datasets and serves as a vital resource for researchers intending to navigate this dynamic field. By emphasizing the strengths and addressing the current limitations, it provides a foundation for developing datasets and algorithms that align more closely with practical, scalable applications. The recommendations provided intend to guide the community toward fostering realistic datasets that will advance AI research in understanding and recognizing human actions more effectively in complex and interactive environments. As such, future work should embrace the proposed evaluation criteria to enhance cross-disciplinary progress and innovation.