Remote Bio-Sensing: Open Source Benchmark Framework for Fair Evaluation of rPPG (2307.12644v2)

Abstract: rPPG (Remote photoplethysmography) is a technology that measures and analyzes BVP (Blood Volume Pulse) by using the light absorption characteristics of hemoglobin captured through a camera. Analyzing the measured BVP can derive various physiological signals such as heart rate, stress level, and blood pressure, which can be applied to various applications such as telemedicine, remote patient monitoring, and early prediction of cardiovascular disease. rPPG is rapidly evolving and attracting great attention from both academia and industry by providing great usability and convenience as it can measure biosignals using a camera-equipped device without medical or wearable devices. Despite extensive efforts and advances in this field, serious challenges remain, including issues related to skin color, camera characteristics, ambient lighting, and other sources of noise and artifacts, which degrade accuracy performance. We argue that fair and evaluable benchmarking is urgently required to overcome these challenges and make meaningful progress from both academic and commercial perspectives. In most existing work, models are trained, tested, and validated only on limited datasets. Even worse, some studies lack available code or reproducibility, making it difficult to fairly evaluate and compare performance. Therefore, the purpose of this study is to provide a benchmarking framework to evaluate various rPPG techniques across a wide range of datasets for fair evaluation and comparison, including both conventional non-deep neural network (non-DNN) and deep neural network (DNN) methods. GitHub URL: https://github.com/remotebiosensing/rppg

• The paper presents an open-source benchmark framework for fair evaluation of rPPG methods, addressing key reproducibility challenges.
• It details a multi-layer approach covering data analysis, preprocessing, model training, and evaluation with performance metrics like MAE and RMSE.
• The framework supports both DNN and non-DNN approaches, improving comparability across diverse datasets and environmental conditions.

An Overview of Remote Bio-Sensing: Open Source Benchmark Framework for Fair Evaluation of rPPG

The paper under consideration discusses the development and introduction of a comprehensive open-source benchmark framework designed to evaluate and compare various remote photoplethysmography (rPPG) techniques. rPPG is a burgeoning field of paper in remote bio-sensing technology that analyzes Blood Volume Pulse (BVP) by utilizing the optical properties of hemoglobin through camera-captured images. It offers promising applications in areas like telemedicine, remote patient monitoring, and health assessments.

Core Challenges and Need for Benchmarking

Despite the potential of rPPG, it faces significant challenges, such as variations in skin color, different camera characteristics, ambient lighting conditions, and multiple noise sources that affect the accuracy of readings. Furthermore, there is a prominent variation in datasets, along with the lack of openly available code for various proposed models, which significantly impairs reproducibility and fair comparison of performance in existing studies.

The paper argues for an objective and comprehensive evaluation system, stressing the urgency of standardized benchmarking to drive forward both academic inquiry and commercial application. Addressing these issues, the authors present an open-source framework aiming to standardize the evaluation of both conventional non-deep neural network (non-DNN) and deep neural network (DNN) rPPG approaches.

Framework Components and Functionalities

The proposed benchmark framework is structured into several layers, encompassing data analysis, preprocessing, modeling, training, evaluation, and application utilities.

• Data Analysis Layer: This segment breaks down the plethora of available datasets, examining alignment discrepancies between labels and actual video data, estimating biases such as those introduced by skin color, and offering a data loader tool to simplify data handling. Additionally, this layer addresses dataset collection under predefined environmental conditions or standardized processes.
• Preprocessing Layer: Acknowledging the diverse formatting of datasets, this layer standardizes preprocessing into four main approaches: Difference Normalization (DiffNorm), Z-score normalization, Spatial-Temporal Mapping (STMap), and direct raw data processing. These methods streamline data preparation for model training and evaluation.
• Model Layer: Within this framework, a wide array of both DNN and traditional non-DNN rPPG methodologies are implemented and made available, leveraging PyTorch for DNN models. This layer supports the open exchange and comparison of model performance, effectively mitigating reproducibility issues endemic to previous research efforts.
• Training and Evaluation Layer: Besides configurable model settings through YAML files, users benefit from structured data splitting for rigorous testing, ensuring no overlap of subjects among training, validation, and test sets. Model performance is evaluated using several metrics such as MAE, RMSE, MAPE, and Pearson correlation, among others, over different time intervals, facilitating a robust comparative analysis.
• Application Layer: While not fully elaborated in the described paper, this layer is intended to provide practical interfaces and demonstration utilities for deploying rPPG models, highlighting their application potential.

Results and Future Directions

From the evaluation conducted using different datasets (e.g., UBFC and PURE), the framework enables cross-validation and reproducibility, critical for scientific rigour. It is observed that time intervals significantly impact the model prediction outcomes, offering insights into improving the temporal aspect of data used in model training. For future work, an emphasis is placed on continually updating the framework with emerging datasets and models, supporting a dynamic research environment conducive to collaboration and technological advancement.

Conclusion

This paper marks a significant contribution to the standardization and transparency of remote bio-sensing evaluations. The introduction of a fair and reproducible benchmarking framework is set to be a pivotal tool, ameliorating the fragmented approach historically prevalent in rPPG studies. By facilitating systematic evaluations and enhancing accessibility, this framework is poised to accelerate the progression and adoption of rPPG technologies within both academic and industrial settings.