Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognition (2312.02185v1)

Abstract: Various types of sensors can be used for Human Activity Recognition (HAR), and each of them has different strengths and weaknesses. Sometimes a single sensor cannot fully observe the user's motions from its perspective, which causes wrong predictions. While sensor fusion provides more information for HAR, it comes with many inherent drawbacks like user privacy and acceptance, costly set-up, operation, and maintenance. To deal with this problem, we propose Virtual Fusion - a new method that takes advantage of unlabeled data from multiple time-synchronized sensors during training, but only needs one sensor for inference. Contrastive learning is adopted to exploit the correlation among sensors. Virtual Fusion gives significantly better accuracy than training with the same single sensor, and in some cases, it even surpasses actual fusion using multiple sensors at test time. We also extend this method to a more general version called Actual Fusion within Virtual Fusion (AFVF), which uses a subset of training sensors during inference. Our method achieves state-of-the-art accuracy and F1-score on UCI-HAR and PAMAP2 benchmark datasets. Implementation is available upon request.

• The paper proposes Virtual Fusion and its extension AFVF, enabling models to train with multi-sensor data and perform inference with a single sensor.
• It employs contrastive learning to capture correlations across sensor inputs, thereby overcoming limitations of traditional sensor fusion methods.
• Experimental results on benchmark datasets demonstrate superior accuracy, offering a cost-effective and privacy-preserving HAR solution.

Unleashing the Potential of Single-Sensor Activity Recognition with Virtual Fusion

Human Activity Recognition (HAR) is an area of technology intent on identifying people's activities through data acquired from various sensors. From enhancing sports performance analysis to improving health monitoring, HAR has a broad spectrum of applications. Traditional methods often use camera or wearable sensors, each with its advantages and disadvantages. Cameras can capture a wide range of movements, while wearables like accelerometers and gyroscopes are preferred for their privacy preservation and portability.

However, recognizing activities with a single sensor often presents challenges. For instance, a singular wearable might not provide enough context to distinguish between different activities accurately. Traditionally, combining multiple sensors—sensor fusion—has been a method to address this issue, improving accuracy. Despite its advantages, sensor fusion isn't without drawbacks, including its invasion of privacy, expensive setup, and complex operation.

This context sets the stage for the introduction of a novel approach named Virtual Fusion. This new method leverages the benefits of multiple sensors during the training phase of a HAR model without requiring anything more than a single sensor during the actual operation or inference. By employing a technique called contrastive learning, the model can exploit the inherent correlations among the data from different sensors.

The paper's experimental results showcased a significant advancement. Virtual Fusion led to an enhancement in accuracy using single-sensor inference. In some cases, the results even outperformed traditional sensor fusion, which relies on multiple sensors during both training and testing. This improvement opens doors to cost-effective, efficient, and privacy-respectful activity recognition.

To extend the capabilities of Virtual Fusion, the researchers also introduced a more general version named Actual Fusion within Virtual Fusion (AFVF). AFVF can work with any subset of the sensors used during training while still using a single or a few sensors for inference. This flexibility means that even if certain sensors are unavailable or unsuitable for deployment, one can still use the training data from those sensors to improve the performance of available sensors during operation.

The effectiveness of Virtual Fusion and AFVF was validated through extensive experiments on benchmark datasets. These experiments highlighted the state-of-the-art results accomplished by Virtual Fusion and its extension, proving its robustness compared to existing methods.

In conclusion, Virtual Fusion proposes a ground-breaking approach for HAR, maximizing the utility of multiple sensors in the learning phase while simplifying the actual application to a single sensor. Its extension, AFVF, allows for a versatile application with various sensor configurations during inference, ensuring adaptability and superior performance in real-world situations. This research indicates a promising direction for future development in HAR technologies, potentially leading to smarter and more integrated sensor-based applications for everyday life and specialized domains alike.