Human Activity Recognition using RGB-Event based Sensors: A Multi-modal Heat Conduction Model and A Benchmark Dataset (2504.05830v1)

Abstract: Human Activity Recognition (HAR) primarily relied on traditional RGB cameras to achieve high-performance activity recognition. However, the challenging factors in real-world scenarios, such as insufficient lighting and rapid movements, inevitably degrade the performance of RGB cameras. To address these challenges, biologically inspired event cameras offer a promising solution to overcome the limitations of traditional RGB cameras. In this work, we rethink human activity recognition by combining the RGB and event cameras. The first contribution is the proposed large-scale multi-modal RGB-Event human activity recognition benchmark dataset, termed HARDVS 2.0, which bridges the dataset gaps. It contains 300 categories of everyday real-world actions with a total of 107,646 paired videos covering various challenging scenarios. Inspired by the physics-informed heat conduction model, we propose a novel multi-modal heat conduction operation framework for effective activity recognition, termed MMHCO-HAR. More in detail, given the RGB frames and event streams, we first extract the feature embeddings using a stem network. Then, multi-modal Heat Conduction blocks are designed to fuse the dual features, the key module of which is the multi-modal Heat Conduction Operation layer. We integrate RGB and event embeddings through a multi-modal DCT-IDCT layer while adaptively incorporating the thermal conductivity coefficient via FVEs into this module. After that, we propose an adaptive fusion module based on a policy routing strategy for high-performance classification. Comprehensive experiments demonstrate that our method consistently performs well, validating its effectiveness and robustness. The source code and benchmark dataset will be released on https://github.com/Event-AHU/HARDVS/tree/HARDVSv2

Human Activity Recognition using RGB-Event Based Sensors: Insights and Future Directions

The development of human activity recognition (HAR) has been a significant pursuit within the field of computer vision, largely due to the advancement of RGB-based cameras. However, constraints such as low light sensitivity and motion blur limit their efficacy in dynamic environments. In this context, the integration of event cameras, which mimic biological vision systems, presents a potential paradigm shift by offering high dynamic range and high temporal resolution without motion artifacts.

This paper introduces a cutting-edge approach that hybridizes RGB frames and data from event cameras to enhance HAR performance. The primary contributions include the introduction of the HARDVS 2.0 dataset, a large-scale multi-modal HAR benchmark, and the development of a novel recognition framework termed Multi-modal Heat Conduction Operation for HAR (MMHCO-HAR).

Key Contributions and Methodology

1. Dataset Innovation:
   • HARDVS 2.0 Dataset: The HARDVS 2.0 benchmark dataset fills a critical gap in resource availability for HAR research. It provides 300 diverse action categories across over 107,646 paired RGB and event video sequences. This broad representation supports comprehensive algorithmic evaluation under real-world variability such as varying illumination, motion speeds, and occlusions.
2. Novel Recognition Framework:
   • Multi-modal Heat Conduction Model: Inspired by the physics of heat conduction, the MMHCO-HAR facilitates effective feature extraction and fusion from RGB and event modalities. This is accomplished through a stem network that extracts multi-modal feature embeddings and Heat Conduction Operation (HCO) layers that adaptively merge these features.
   • Adaptive Fusion Strategy: A policy routing mechanism refines feature integration, dynamically selecting among complementary, discriminative, or specific feature fusion approaches based on input data characteristics.
3. Comprehensive Experimental Validation:
   • The proposed MMHCO-HAR framework demonstrates superior recognition accuracy on the HARDVS 2.0 dataset, surpassing previous methods reliant on RGB-only inputs. The multi-modal approach addresses deficiencies in feature richness and adaptability.

Implications and Future Work

The implications of this research are multifaceted, spanning theoretical advancements in multi-modal learning and practical enhancements in HAR applications. By leveraging the distinct advantages of RGB and event data, the approach significantly mitigates issues posed by traditional sensors. The robust modelling via physics-inspired operations promises improved interpretability and efficiency.

Moving forward, the research opens several avenues for exploration:

• Enhanced Temporal Modeling: Future studies could explore optimizing temporal sequence modeling, perhaps through hybrid architectures incorporating recurrent layers or attention mechanisms.
• Cross-Modal Learning: Integrating other modalities such as audio or textual cues might further enhance activity understanding.
• Transfer Learning Applications: Given the comprehensive nature of the dataset, it provides a fertile ground for developing transfer learning models applicable to related tasks such as gesture recognition and real-time surveillance.

In conclusion, this paper provides a meaningful advancement in HAR, demonstrating the potential of integrating event-based data streams for more effective activity recognition in challenging environments. The methodologies and dataset presented here will surely catalyze future research endeavors, driving the field towards more robust, practical solutions.