Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Advancing Generalizable Remote Physiological Measurement through the Integration of Explicit and Implicit Prior Knowledge (2403.06947v1)

Published 11 Mar 2024 in cs.CV

Abstract: Remote photoplethysmography (rPPG) is a promising technology that captures physiological signals from face videos, with potential applications in medical health, emotional computing, and biosecurity recognition. The demand for rPPG tasks has expanded from demonstrating good performance on intra-dataset testing to cross-dataset testing (i.e., domain generalization). However, most existing methods have overlooked the prior knowledge of rPPG, resulting in poor generalization ability. In this paper, we propose a novel framework that simultaneously utilizes explicit and implicit prior knowledge in the rPPG task. Specifically, we systematically analyze the causes of noise sources (e.g., different camera, lighting, skin types, and movement) across different domains and incorporate these prior knowledge into the network. Additionally, we leverage a two-branch network to disentangle the physiological feature distribution from noises through implicit label correlation. Our extensive experiments demonstrate that the proposed method not only outperforms state-of-the-art methods on RGB cross-dataset evaluation but also generalizes well from RGB datasets to NIR datasets. The code is available at https://github.com/keke-nice/Greip.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. W. Verkruysse, L. O. Svaasand, and J. S. Nelson, “Remote plethysmographic imaging using ambient light.” Optics express, vol. 16, no. 26, pp. 21 434–21 445, 2008.
  2. M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Non-contact, automated cardiac pulse measurements using video imaging and blind source separation.” Optics express, vol. 18, no. 10, pp. 10 762–10 774, 2010.
  3. W. Wang, A. C. Den Brinker, S. Stuijk, and G. De Haan, “Amplitude-selective filtering for remote-ppg,” Biomedical optics express, vol. 8, no. 3, pp. 1965–1980, 2017.
  4. G. De Haan and V. Jeanne, “Robust pulse rate from chrominance-based rppg,” IEEE Transactions on Biomedical Engineering, vol. 60, no. 10, pp. 2878–2886, 2013.
  5. H. Wang, E. Ahn, and J. Kim, “Self-supervised representation learning framework for remote physiological measurement using spatiotemporal augmentation loss,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, 2022, pp. 2431–2439.
  6. Y. Yang, X. Liu, J. Wu, S. Borac, D. Katabi, M.-Z. Poh, and D. McDuff, “Simper: Simple self-supervised learning of periodic targets,” arXiv preprint arXiv:2210.03115, 2022.
  7. X. Liu, Y. Zhang, Z. Yu, H. Lu, H. Yue, and J. Yang, “rppg-mae: Self-supervised pre-training with masked autoencoders for remote physiological measurement,” arXiv preprint arXiv:2306.02301, 2023.
  8. Z. Sun and X. Li, “Contrast-phys+: Unsupervised and weakly-supervised video-based remote physiological measurement via spatiotemporal contrast,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
  9. K. Kurihara, D. Sugimura, and T. Hamamoto, “Non-contact heart rate estimation via adaptive rgb/nir signal fusion,” IEEE Transactions on Image Processing, vol. 30, pp. 6528–6543, 2021.
  10. R. Yang, Z. Guan, Z. Yu, X. Feng, J. Peng, and G. Zhao, “Non-contact pain recognition from video sequences with remote physiological measurements prediction,” arXiv preprint arXiv:2105.08822, 2021.
  11. D. Huang, X. Feng, H. Zhang, Z. Yu, J. Peng, G. Zhao, and Z. Xia, “Spatio-temporal pain estimation network with measuring pseudo heart rate gain,” IEEE Transactions on Multimedia, vol. 24, pp. 3300–3313, 2021.
  12. D. McDuff, S. Gontarek, and R. Picard, “Remote measurement of cognitive stress via heart rate variability,” in 2014 36th annual international conference of the IEEE engineering in medicine and biology society.   IEEE, 2014, pp. 2957–2960.
  13. J. Speth, N. Vance, A. Czajka, K. W. Bowyer, D. Wright, and P. Flynn, “Deception detection and remote physiological monitoring: A dataset and baseline experimental results,” in 2021 IEEE International Joint Conference on Biometrics (IJCB).   IEEE, 2021, pp. 1–8.
  14. Z. Yu, X. Li, P. Wang, and G. Zhao, “Transrppg: Remote photoplethysmography transformer for 3d mask face presentation attack detection,” IEEE Signal Processing Letters, vol. 28, pp. 1290–1294, 2021.
  15. H. Qi, Q. Guo, F. Juefei-Xu, X. Xie, L. Ma, W. Feng, Y. Liu, and J. Zhao, “Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms,” in Proceedings of the 28th ACM international conference on multimedia, 2020, pp. 4318–4327.
  16. G. Balakrishnan, F. Durand, and J. Guttag, “Detecting pulse from head motions in video,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2013, pp. 3430–3437.
  17. A. Lam and Y. Kuno, “Robust heart rate measurement from video using select random patches,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 3640–3648.
  18. X. Li, J. Chen, G. Zhao, and M. Pietikainen, “Remote heart rate measurement from face videos under realistic situations,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 4264–4271.
  19. M.-Z. Poh, D. J. McDuff, and R. W. Picard, “Advancements in noncontact, multiparameter physiological measurements using a webcam,” IEEE transactions on biomedical engineering, vol. 58, no. 1, pp. 7–11, 2010.
  20. W. Wang, S. Stuijk, and G. De Haan, “A novel algorithm for remote photoplethysmography: Spatial subspace rotation,” IEEE transactions on biomedical engineering, vol. 63, no. 9, pp. 1974–1984, 2015.
  21. W. Wang, A. C. Den Brinker, S. Stuijk, and G. De Haan, “Algorithmic principles of remote ppg,” IEEE Transactions on Biomedical Engineering, vol. 64, no. 7, pp. 1479–1491, 2016.
  22. G. De Haan and A. Van Leest, “Improved motion robustness of remote-ppg by using the blood volume pulse signature,” Physiological measurement, vol. 35, no. 9, p. 1913, 2014.
  23. A. Das, H. Lu, H. Han, A. Dantcheva, S. Shan, and X. Chen, “Bvpnet: Video-to-bvp signal prediction for remote heart rate estimation,” in 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021).   IEEE, 2021, pp. 01–08.
  24. Z. Yu, X. Li, and G. Zhao, “Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks,” arXiv preprint arXiv:1905.02419, 2019.
  25. W. Chen and D. McDuff, “Deepphys: Video-based physiological measurement using convolutional attention networks,” in Proceedings of the european conference on computer vision (ECCV), 2018, pp. 349–365.
  26. Z. Yu, Y. Shen, J. Shi, H. Zhao, P. H. Torr, and G. Zhao, “Physformer: Facial video-based physiological measurement with temporal difference transformer,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 4186–4196.
  27. Z. Sun and X. Li, “Contrast-phys: Unsupervised video-based remote physiological measurement via spatiotemporal contrast,” in European Conference on Computer Vision.   Springer, 2022, pp. 492–510.
  28. H. Lu, H. Han, and S. K. Zhou, “Dual-gan: Joint bvp and noise modeling for remote physiological measurement,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 404–12 413.
  29. X. Liu, J. Fromm, S. Patel, and D. McDuff, “Multi-task temporal shift attention networks for on-device contactless vitals measurement,” Advances in Neural Information Processing Systems, vol. 33, pp. 19 400–19 411, 2020.
  30. S.-Q. Liu and P. C. Yuen, “Robust remote photoplethysmography estimation with environmental noise disentanglement,” IEEE Transactions on Image Processing, 2023.
  31. Z. Yu, Y. Shen, J. Shi, H. Zhao, Y. Cui, J. Zhang, P. Torr, and G. Zhao, “Physformer++: Facial video-based physiological measurement with slowfast temporal difference transformer,” International Journal of Computer Vision, vol. 131, no. 6, pp. 1307–1330, 2023.
  32. X. Liu, B. Hill, Z. Jiang, S. Patel, and D. McDuff, “Efficientphys: Enabling simple, fast and accurate camera-based cardiac measurement,” in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2023, pp. 5008–5017.
  33. W.-H. Chung, C.-J. Hsieh, S.-H. Liu, and C.-T. Hsu, “Domain generalized rppg network: Disentangled feature learning with domain permutation and domain augmentation,” in Proceedings of the Asian Conference on Computer Vision, 2022, pp. 807–823.
  34. H. Lu, Z. Yu, X. Niu, and Y.-C. Chen, “Neuron structure modeling for generalizable remote physiological measurement,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 18 589–18 599.
  35. W. Sun, X. Zhang, H. Lu, Y. Chen, Y. Ge, X. Huang, J. Yuan, and Y. Chen, “Resolve domain conflicts for generalizable remote physiological measurement,” in Proceedings of the 31st ACM International Conference on Multimedia, 2023, pp. 8214–8224.
  36. J. Du, S.-Q. Liu, B. Zhang, and P. C. Yuen, “Dual-bridging with adversarial noise generation for domain adaptive rppg estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10 355–10 364.
  37. X. Niu, S. Shan, H. Han, and X. Chen, “Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation,” IEEE Transactions on Image Processing, vol. 29, pp. 2409–2423, 2019.
  38. R. Špetlík, V. Franc, and J. Matas, “Visual heart rate estimation with convolutional neural network,” in Proceedings of the british machine vision conference, Newcastle, UK, 2018, pp. 3–6.
  39. R. Song, H. Chen, J. Cheng, C. Li, Y. Liu, and X. Chen, “Pulsegan: Learning to generate realistic pulse waveforms in remote photoplethysmography,” IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 5, pp. 1373–1384, 2021.
  40. J. Gideon and S. Stent, “The way to my heart is through contrastive learning: Remote photoplethysmography from unlabelled video,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 3995–4004.
  41. J. Speth, N. Vance, P. Flynn, and A. Czajka, “Non-contrastive unsupervised learning of physiological signals from video,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14 464–14 474.
  42. S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi, “Generalizing across domains via cross-gradient training,” arXiv preprint arXiv:1804.10745, 2018.
  43. X. Yue, Y. Zhang, S. Zhao, A. Sangiovanni-Vincentelli, K. Keutzer, and B. Gong, “Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 2100–2110.
  44. Y. Ganin and V. Lempitsky, “Unsupervised domain adaptation by backpropagation,” in International conference on machine learning.   PMLR, 2015, pp. 1180–1189.
  45. G. Parascandolo, A. Neitz, A. Orvieto, L. Gresele, and B. Schölkopf, “Learning explanations that are hard to vary,” arXiv preprint arXiv:2009.00329, 2020.
  46. M. Wang, Y. Liu, J. Yuan, S. Wang, Z. Wang, and W. Wang, “Inter-class and inter-domain semantic augmentation for domain generalization,” IEEE Transactions on Image Processing, 2024.
  47. F. Lv, J. Liang, S. Li, B. Zang, C. H. Liu, Z. Wang, and D. Liu, “Causality inspired representation learning for domain generalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8046–8056.
  48. S. Sankaranarayanan and Y. Balaji, “Meta learning for domain generalization,” in Meta-Learning with Medical Imaging and Health Informatics Applications.   Elsevier, 2023, pp. 75–86.
  49. K. Tang, M. Tao, J. Qi, Z. Liu, and H. Zhang, “Invariant feature learning for generalized long-tailed classification,” in European Conference on Computer Vision.   Springer, 2022, pp. 709–726.
  50. S. Bobbia, R. Macwan, Y. Benezeth, A. Mansouri, and J. Dubois, “Unsupervised skin tissue segmentation for remote photoplethysmography,” Pattern Recognition Letters, vol. 124, pp. 82–90, 2019.
  51. R. Stricker, S. Müller, and H.-M. Gross, “Non-contact video-based pulse rate measurement on a mobile service robot,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication.   IEEE, 2014, pp. 1056–1062.
  52. S. Chen, S. K. Ho, J. W. Chin, K. H. Luo, T. T. Chan, R. H. So, and K. L. Wong, “Deep learning-based image enhancement for robust remote photoplethysmography in various illumination scenarios,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 6076–6084.
  53. X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 1501–1510.
  54. A. Revanur, Z. Li, U. A. Ciftci, L. Yin, and L. A. Jeni, “The first vision for vitals (v4v) challenge for non-contact video-based physiological estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2760–2767.
  55. L. Xi, W. Chen, C. Zhao, X. Wu, and J. Wang, “Image enhancement for remote photoplethysmography in a low-light environment,” in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020).   IEEE, 2020, pp. 1–7.
  56. E. M. Nowara, T. K. Marks, H. Mansour, and A. Veeraraghavan, “Near-infrared imaging photoplethysmography during driving,” IEEE transactions on intelligent transportation systems, vol. 23, no. 4, pp. 3589–3600, 2020.
  57. B. Sun and K. Saenko, “Deep coral: Correlation alignment for deep domain adaptation,” in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14.   Springer, 2016, pp. 443–450.
  58. D. Krueger, E. Caballero, J.-H. Jacobsen, A. Zhang, J. Binas, D. Zhang, R. Le Priol, and A. Courville, “Out-of-distribution generalization via risk extrapolation (rex),” in International Conference on Machine Learning.   PMLR, 2021, pp. 5815–5826.
  59. C. X. Tian, H. Li, X. Xie, Y. Liu, and S. Wang, “Neuron coverage-guided domain generalization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 1, pp. 1302–1311, 2022.
  60. S. Tulyakov, X. Alameda-Pineda, E. Ricci, L. Yin, J. F. Cohn, and N. Sebe, “Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2396–2404.
  61. J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6299–6308.
  62. X. Niu, Z. Yu, H. Han, X. Li, S. Shan, and G. Zhao, “Video-based remote physiological measurement via cross-verified feature disentangling,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16.   Springer, 2020, pp. 295–310.
  63. K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
  64. X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 15 750–15 758.
  65. J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020.
  66. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning.   PMLR, 2020, pp. 1597–1607.
  67. S. Liu, P. C. Yuen, S. Zhang, and G. Zhao, “3d mask face anti-spoofing with remote photoplethysmography,” in Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14.   Springer, 2016, pp. 85–100.
  68. E. M. Nowara, A. Sabharwal, and A. Veeraraghavan, “Ppgsecure: Biometric presentation attack detection using photopletysmograms,” in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).   IEEE, 2017, pp. 56–62.
  69. G. Heusch, A. Anjos, and S. Marcel, “A reproducible study on remote heart rate measurement,” arXiv preprint arXiv:1709.00962, 2017.
  70. N. Erdogmus and S. Marcel, “Spoofing face recognition with 3d masks,” IEEE transactions on information forensics and security, vol. 9, no. 7, pp. 1084–1097, 2014.
  71. S.-Q. Liu, X. Lan, and P. C. Yuen, “Remote photoplethysmography correspondence feature for 3d mask face presentation attack detection,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 558–573.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now