Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Orientation-conditioned Facial Texture Mapping for Video-based Facial Remote Photoplethysmography Estimation (2404.09378v3)

Published 14 Apr 2024 in cs.CV

Abstract: Camera-based remote photoplethysmography (rPPG) enables contactless measurement of important physiological signals such as pulse rate (PR). However, dynamic and unconstrained subject motion introduces significant variability into the facial appearance in video, confounding the ability of video-based methods to accurately extract the rPPG signal. In this study, we leverage the 3D facial surface to construct a novel orientation-conditioned facial texture video representation which improves the motion robustness of existing video-based facial rPPG estimation methods. Our proposed method achieves a significant 18.2% performance improvement in cross-dataset testing on MMPD over our baseline using the PhysNet model trained on PURE, highlighting the efficacy and generalization benefits of our designed video representation. We demonstrate significant performance improvements of up to 29.6% in all tested motion scenarios in cross-dataset testing on MMPD, even in the presence of dynamic and unconstrained subject motion, emphasizing the benefits of disentangling motion through modeling the 3D facial surface for motion robust facial rPPG estimation. We validate the efficacy of our design decisions and the impact of different video processing steps through an ablation study. Our findings illustrate the potential strengths of exploiting the 3D facial surface as a general strategy for addressing dynamic and unconstrained subject motion in videos. The code is available at https://samcantrill.github.io/orientation-uv-rppg/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Non-contact heart rate monitoring utilizing camera photoplethysmography in the neonatal intensive care unit—a pilot study. Early Human Development, 89(12):943–948, 2013.
  2. Face2PPG: An unsupervised pipeline for blood volume pulse extraction from faces. IEEE Journal of Biomedical and Health Informatics, 2023.
  3. DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. In IEEE/CVF European Conference on Computer Vision (ECCV), pages 349–365, 2018.
  4. Gerard de Haan and Vincent Jeanne. Robust Pulse Rate From Chrominance-Based rPPG. IEEE Transactions on Biomedical Engineering, 60(10):2878–2886, 2013.
  5. Gerard de Haan and Arno van Leest. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiological Measurement, 35(9):1913, 2014.
  6. Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation. Applied Sciences, 10(23):8630, 2020.
  7. DeepFakesON-Phys: DeepFakes Detection based on Heart Rate Estimation. arXiv preprint arXiv:2010.00400, 2020.
  8. ETA-rPPGNet: Effective Time-domain Attention Network for Remote Heart Rate Measurement. IEEE Transactions on Instrumentation and Measurement, 70:1–12, 2021a.
  9. Robust Heart Rate Estimation With Spatial-Temporal Attention Network From Facial Videos. IEEE Transactions on Cognitive and Developmental Systems, 14(2):639–647, 2021b.
  10. Non-contact PPG signal and heart rate estimation with multi-hierarchical convolutional network. Pattern Recognition, 139:109421, 2023a.
  11. Learning Motion-Robust Remote Photoplethysmography through Arbitrary Resolution Videos. In AAAI Conference on Artificial Intelligence, pages 1334–1342, 2023b.
  12. Face Liveness Detection by rPPG Features and Contextual Patch-Based CNN. In International Conference on Biometric Engineering and Applications, pages 61–68. Association for Computing Machinery, 2019.
  13. Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement. Advances in Neural Information Processing Systems, 33:19400–19411, 2020.
  14. EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Cardiac Measurement. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4997–5006. IEEE, 2023.
  15. rPPG-Toolbox: Deep Remote PPG Toolbox. Advances in Neural Information Processing Systems, 36, 2024a.
  16. rPPG-MAE: Self-supervised Pre-training with Masked Autoencoders for Remote Physiological Measurement. IEEE Transactions on Multimedia, 2024b.
  17. Dual-GAN: Joint BVP and Noise Modeling for Remote Physiological Measurement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12399–12408, 2021.
  18. MediaPipe: A Framework for Building Perception Pipelines. arXiv preprint arXiv:1906.08172, 2019.
  19. Remote Heart Rate Estimation Based on 3D Facial Landmarks. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 2634–2637, 2020.
  20. Daniel McDuff. Camera Measurement of Physiological Vital Signs. ACM Computing Surveys, 55(9):1–40, 2023.
  21. SynRhythm: Learning a Deep Heart Rate Estimator from General to Specific. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 3580–3585, 2018.
  22. VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-Constrained Face Video. In IEEE/CVF Asian Conference on Computer Vision (ACCV), pages 562–576. Springer International Publishing, 2019a.
  23. RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation. IEEE Transactions on Image Processing, 29:2409–2423, 2019b.
  24. Robust Remote Heart Rate Estimation from Face Utilizing Spatial-temporal Attention. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pages 1–8, 2019c.
  25. Video-based Remote Physiological Measurement via Cross-verified Feature Disentangling. In European ConECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 295–310. Springer, 2020.
  26. The Benefit of Distraction: Denoising Camera-Based Physiological Measurements Using Inverse Attention. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 4955–4964, 2021.
  27. Motion matters: Neural motion transfer for better camera physiological measurement. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5933–5942, 2024.
  28. HeartTrack: Convolutional Neural Network for Remote Video-Based Heart Rate Monitoring. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1163–1171, 2020.
  29. Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam. IEEE Transactions on Biomedical Engineering, 58(1):7–11, 2010a.
  30. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Optics Express, 18(10):10762–10774, 2010b.
  31. Dual-path TokenLearner for Remote Photoplethysmography-based Physiological Measurement with Facial Videos. arXiv preprint arXiv:2308.07771, 2023.
  32. Visual Heart Rate Estimation with Convolutional Neural Network. In British Machine Vision Conference (BMVC), pages 3–6, 2018.
  33. Non-Contact Video-Based Pulse Rate Measurement on a Mobile Service Robot. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 1056–1062, 2014.
  34. MMPD: Multi-Domain Mobile Video Physiology Dataset. arXiv preprint arXiv:2302.03840, 2023.
  35. An advanced detrending method with application to HRV analysis. IEEE Transactions on Biomedical Engineering, 49(2):172–175, 2002.
  36. A Novel Algorithm for Remote Photoplethysmography: Spatial Subspace Rotation. IEEE Transactions on Biomedical Engineering, 63(9):1974–1984, 2015.
  37. Algorithmic Principles of Remote PPG. IEEE Transactions on Biomedical Engineering, 64(7):1479–1491, 2016.
  38. Optimising rPPG Signal Extraction by Exploiting Facial Surface Orientation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2164–2170, 2022.
  39. Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks. In British Machine Visison Conference (BMVC), 2019.
  40. AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching. IEEE Signal Processing Letters, 27:1245–1249, 2020.
  41. TransRPPG: Remote Photoplethysmography Transformer for 3d Mask Face Presentation Attack Detection. IEEE Signal Processing Letters, 28:1290–1294, 2021.
  42. PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4186–4196, 2022.
  43. Multimodal Spontaneous Emotion Corpus for Human Behavior Analysis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3438–3446, 2016.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets