Remote Heart Rate Measurement from Highly Compressed Facial Videos: an End-to-end Deep Learning Solution with Video Enhancement (1907.11921v1)

Abstract: Remote photoplethysmography (rPPG), which aims at measuring heart activities without any contact, has great potential in many applications (e.g., remote healthcare). Existing rPPG approaches rely on analyzing very fine details of facial videos, which are prone to be affected by video compression. Here we propose a two-stage, end-to-end method using hidden rPPG information enhancement and attention networks, which is the first attempt to counter video compression loss and recover rPPG signals from highly compressed videos. The method includes two parts: 1) a Spatio-Temporal Video Enhancement Network (STVEN) for video enhancement, and 2) an rPPG network (rPPGNet) for rPPG signal recovery. The rPPGNet can work on its own for robust rPPG measurement, and the STVEN network can be added and jointly trained to further boost the performance especially on highly compressed videos. Comprehensive experiments are performed on two benchmark datasets to show that, 1) the proposed method not only achieves superior performance on compressed videos with high-quality videos pair, 2) it also generalizes well on novel data with only compressed videos available, which implies the promising potential for real world applications.

• The paper presents a novel end-to-end deep learning solution for remote heart rate measurement from highly compressed facial videos, addressing challenges posed by compression artifacts.
• The proposed framework utilizes a two-stage network: an enhancement network (STVEN) to improve video quality and an rPPG network (rPPGNet) to accurately recover heart rate signals.
• Evaluations show the method outperforms existing techniques on benchmark datasets, particularly under high compression, enabling more robust telemedicine and remote monitoring applications.

Overview of Remote Heart Rate Measurement from Highly Compressed Facial Videos

The paper presents a novel end-to-end deep learning methodology for enhancing and recovering remote photoplethysmography (rPPG) signals from highly compressed facial videos. This approach is significant for applications such as remote healthcare, where accurate heart activity assessments without contact are desirable. Traditional rPPG methods have often struggled to maintain accuracy under conditions of video compression, which is a prevalent issue given the widespread use of video coding techniques for efficient data storage and transmission. By proposing a two-stage framework comprised of a video enhancement network (STVEN) and a dedicated rPPG network (rPPGNet), the authors address these challenges effectively.

Methodology

The proposed framework operates in two primary stages:

1. Spatio-Temporal Video Enhancement Network (STVEN): This network focuses on enhancing the quality of the highly compressed videos. Through a meticulous process of learning to fill in the artifacts introduced by compression (operating across different compression bitrates), the STVEN refines video quality by reconstructing key high-level data necessary for rPPG signal extraction.
2. rPPGNet: The second component is a robust network designed to recover accurate heart rate signals from the video input. Equipped with a skin-based attention module and partition constraints, rPPGNet is capable of delivering not only average heart rate metrics but also more comprehensive heart rate variability information. This advancement over conventional methodologies, which typically focus merely on the average heart rate, marks an important contribution.

Numerical Results and Claims

The proposed system demonstrates its capability through experiments on benchmark datasets such as the OBF and MAHNOB-HCI datasets. The solution is shown to outperform existing methods, particularly under conditions where video data is highly compressed. For example, the model achieved high correlation coefficients (above 0.7) and reduced error metrics in scenarios involving low-bitrate video inputs, distinguishing these contributions from prior methods that were vulnerable to compression artifacts.

Implications and Speculations

The implications of this work are far-reaching in practical terms. By enhancing the fidelity of remote heart rate monitoring systems, especially in compressed media contexts, this model holds promise for broad deployment in telemedicine and continuous, unobtrusive health monitoring. The theoretical advancements presented in the integration of video enhancement with the rPPG analysis suggest a direction for further studies, particularly in refining the neural architecture to handle varied real-world conditions.

Additionally, the robust feature sharing between video enhancement and rPPG recovery tasks points to future explorations in multi-task learning frameworks, which could yield improved models flexible enough to handle fluctuating video qualities. In practice, adoption of such systems could reduce reliance on high-quality but resource-intensive video data, thereby democratically extending rPPG applications to settings with bandwidth constraints.

Future Developments

While the paper addresses a significant gap in the domain of remote physiological signal measurement, there remain avenues for exploration. Future work may delve into extending the model to other physiological measurement contexts, or further optimizing the network to balance performance with computational efficiency—a crucial consideration for deployment on less powerful devices common in telehealth scenarios.

In conclusion, this paper contributes a meaningful advancement in remote heart rate measurement technology, offering a viable pathway to robust, real-world rPPG applications in the face of video compression challenges.