Passive Heart Rate Monitoring During Smartphone Use in Everyday Life

Abstract: Resting heart rate (RHR) is an important biomarker of cardiovascular health and mortality, but tracking it longitudinally generally requires a wearable device, limiting its availability. We present PHRM, a deep learning system for passive heart rate (HR) and RHR measurements during everyday smartphone use, using facial video-based photoplethysmography. Our system was developed using 225,773 videos from 495 participants and validated on 185,970 videos from 205 participants in laboratory and free-living conditions, representing the largest validation study of its kind. Compared to reference electrocardiogram, PHRM achieved a mean absolute percentage error (MAPE) < 10% for HR measurements across three skin tone groups of light, medium and dark pigmentation; MAPE for each skin tone group was non-inferior versus the others. Daily RHR measured by PHRM had a mean absolute error < 5 bpm compared to a wearable HR tracker, and was associated with known risk factors. These results highlight the potential of smartphones to enable passive and equitable heart health monitoring.

• The paper presents a deep learning system that uses 8-second facial video clips to passively estimate heart rate and daily resting heart rate using TS-CNN and Kalman filtering.
• Methodology focuses on processing video frames through stabilization, cropping, and interpolation to achieve measurement accuracies with MAPE below 10% in controlled settings.
• Validation across diverse populations using over 400,000 videos demonstrates robust performance and equitable accuracy in real-world cardiovascular monitoring.

Passive Heart Rate Monitoring During Smartphone Use in Everyday Life

This paper presents a deep learning system for passive heart rate (HR) and resting heart rate (RHR) measurements using facial video-based photoplethysmography during everyday smartphone use. The proposed system, referred to as PHRM, enables HR monitoring without the need for wearable devices, thereby enhancing accessibility and simplification of cardiovascular health monitoring.

System Design and Development

The PHRM system is designed with two major components: an end-to-end HR estimation module and an algorithm for deriving daily RHR. The HR estimation module processes short (8-second) facial video clips to predict HR and associated confidence measures. This involves video stabilization, cropping, resizing, interpolation, and computing frame differences. An ensemble of computationally-efficient temporal shift convolutional neural networks (TS-CNN) is used for HR prediction.

Figure 1: System overview, development, and validation of the deep learning system for passive heart rate (HR) and daily resting HR (RHR) measurements (PHRM) during smartphone use.

To compute daily RHR, the PHRM aggregates valid HR measurements throughout the day and applies a Kalman filter for improving estimate accuracy. This passive operation, initiated upon screen unlock events, allows non-invasive HR monitoring seamlessly integrated into regular smartphone usage.

Study Populations and Data Collection

Data for developing and validating PHRM was collected from five independent, prospective laboratory studies and a free-living study. In total, 225,773 videos from 495 participants were used for system development, while 185,970 videos from 205 participants were used for validation. This marks the largest validation study of its kind.

The study population was diversified across age, sex, and skin tone groups, with ECG used as the reference for HR measurements. In the prospective free-living study, facial videos were recorded during normal phone usage over eight days, capturing a wide array of environmental conditions and participant interactions.

Figure 2: Representative examples of the diversity of free-living data used to validate the PHRM.

Performance and Validation Results

In laboratory settings, PHRM demonstrated strong performance with HR measurement success rates of 77.7% under controlled conditions. The mean absolute percentage error (MAPE) for HR predictions was well below the pre-specified target of 10% across different lighting and physiological states.

Figure 3: Accuracy of passive HR and RHR measurements by the PHRM in free-living conditions.

In free-living settings, the HR measurement success rate was lower at 43.1%, yet PHRM achieved a video-level MAPE of 4.83% and participant-level MAPE of 6.09%, indicating robust performance under real-world conditions. Measurement performance was equitable across skin tone groups, with overall MAPE differences across skin pigmentation groups within acceptable limits.

For daily RHR, PHRM-derived estimates showed strong correlation with traditional measurements while demonstrating superior reproducibility. Mean absolute errors (MAE) were below the pre-specified target of 5 bpm, and the daily RHR measurements were associated with known cardiovascular health metrics and risk factors.

Discussion and Implications

The PHRM system provides a novel approach to HR and RHR monitoring by leveraging ubiquitous smartphone technology. Its tested performance across diverse demographic and environmental conditions demonstrates its potential for equitable and widespread application without the need for wearable devices. This system broadens access to cardiovascular health monitoring, offering valuable data that can be used to monitor longitudinal changes in HR and assess cardiovascular risk factors.

Challenges such as maintaining accuracy in darker skin tones and high-motion scenarios were identified, which offer directions for future research, including optimizing camera exposure settings and improving algorithms for low-contrast conditions.

Privacy concerns are addressed by ensuring that the system can run locally on smartphones, reducing the need for cloud processing and preserving user data confidentiality. Implementing strict consent protocols and on-device security measures are paramount in real-world deployment.

The findings underscore the relevance of smartphones in passive cardiovascular health monitoring, presenting vast opportunities for detailed health assessments in everyday contexts. Future work will involve optimizing for battery usage and integrating additional sensor data, such as accelerometer readings, to further enhance measurement accuracy.

Conclusion

The research demonstrates the efficacy of PHRM, a smartphone-based system, in providing passive, accurate, and equitable HR and RHR monitoring across diverse real-world environments and participant populations. This advance in remote photoplethysmography methods points towards significant improvements in accessibility and effectiveness of cardiovascular health tracking, making it an essential tool in modern healthcare technology.