Efficient and Robust Multidimensional Attention in Remote Physiological Sensing through Target Signal Constrained Factorization (2505.07013v1)

Abstract: Remote physiological sensing using camera-based technologies offers transformative potential for non-invasive vital sign monitoring across healthcare and human-computer interaction domains. Although deep learning approaches have advanced the extraction of physiological signals from video data, existing methods have not been sufficiently assessed for their robustness to domain shifts. These shifts in remote physiological sensing include variations in ambient conditions, camera specifications, head movements, facial poses, and physiological states which often impact real-world performance significantly. Cross-dataset evaluation provides an objective measure to assess generalization capabilities across these domain shifts. We introduce Target Signal Constrained Factorization module (TSFM), a novel multidimensional attention mechanism that explicitly incorporates physiological signal characteristics as factorization constraints, allowing more precise feature extraction. Building on this innovation, we present MMRPhys, an efficient dual-branch 3D-CNN architecture designed for simultaneous multitask estimation of photoplethysmography (rPPG) and respiratory (rRSP) signals from multimodal RGB and thermal video inputs. Through comprehensive cross-dataset evaluation on five benchmark datasets, we demonstrate that MMRPhys with TSFM significantly outperforms state-of-the-art methods in generalization across domain shifts for rPPG and rRSP estimation, while maintaining a minimal inference latency suitable for real-time applications. Our approach establishes new benchmarks for robust multitask and multimodal physiological sensing and offers a computationally efficient framework for practical deployment in unconstrained environments. The web browser-based application featuring on-device real-time inference of MMRPhys model is available at https://physiologicailab.github.io/mmrphys-live

Efficient and Robust Multidimensional Attention in Remote Physiological Sensing through Target Signal Constrained Factorization

The paper "Efficient and Robust Multidimensional Attention in Remote Physiological Sensing through Target Signal Constrained Factorization" presents a novel approach aimed at addressing the challenges faced in remote physiological sensing applications. Specifically, it explores advancements in the extraction of physiological signals, such as remote photoplethysmography (rPPG) and respiratory signs (rRSP), utilizing camera-based sensing technologies. The introduction and evaluation of the Target Signal Constrained Factorization Module (TSFM) in conjunction with the MMRPhys architecture demonstrate significant improvements in the robustness and efficiency of multimodal physiological sensing systems.

Overview of Methodology

The paper addresses a crucial gap in the current state of remote physiological sensing—the lack of robustness to domain shifts caused by variations in lighting, camera specifications, head movements, and physiological states. To address this, the Target Signal Constrained Factorization Module (TSFM) is introduced as a multidimensional attention mechanism that incorporates physiological signal characteristics directly as factorization constraints. This approach contrasts with traditional methods that do not integrate explicit constraints relating specifically to the signals being extracted, leveraging non-negative matrix factorization (NMF) principles for efficient approximation of low-rank features without latency overhead associated with deep network backpropagation.

The MMRPhys architecture capitalizes on TSFM by establishing an efficient dual-branch 3D-CNN framework that processes RGB and thermal video inputs to concurrently estimate rPPG and rRSP signals. The architecture allows adaptable deployment across multiple spatial resolutions and single or multimodal input configurations, optimized for resource-constrained applications.

Results and Contributions

The results, quantified through cross-dataset evaluations across multiple benchmark datasets, firmly establish the superiority of TSFM compared to existing attention mechanisms, such as FSAM, which do not incorporate specific constraints. Notably, the inclusion of accurate physiological constraint significantly enhances feature discrimination capability and ultimately signal extraction precision. Specifically, TSFM delivered marked improvements in estimating both heart rate (HR) and respiratory rate (RR), demonstrating significant reductions in mean absolute errors when compared to conventional approaches.

For MMRPhys, the robust model architecture supported by TSFM achieved state-of-the-art performance in multitask learning scenarios, particularly evident in the simultaneous extraction of rPPG and rRSP signals. This capability is pivotal for applications such as mobile health and remote monitoring, where real-time processing and adaptability to environmental variations are critical.

Implications and Future Work

The implications of these findings are considerable, particularly in enhancing device capabilities for non-invasive monitoring and interaction within healthcare environments. The reduction in computational complexity promise to stimulate developments in wearable technology, expanding physiological sensing functionalities without compromising response time—a crucial consideration in clinical applications.

Future research can explore broader applications of TSFM in diverse domains of computer vision and AI, potentially extending further into other physiological signals beyond rPPG and rRSP. Moreover, examining the integration of biophysical constraints within neural architectures might yield enhanced prediction accuracies and further mitigate interference from noise or domain shifts in signal estimation tasks.

Overall, this paper makes a substantive contribution to the field of remote physiological sensing, paving the way for robust and efficient frameworks. The proposed methodologies stand out for their practical relevance and technical rigor, offering comprehensive solutions to overcome longstanding challenges in AI-powered physiological monitoring systems.