- The paper presents a novel multi-spectral state space model that fuses RGB and NIR streams to significantly improve rPPG robustness and accuracy in driver monitoring.
- It introduces the CSLM mechanism and bidirectional channel SSM scanning to address challenges from motion and varying illumination, achieving state-of-the-art performance.
- Experimental evaluations on MR-NIRP Car and MS-Drive datasets demonstrate reduced error rates and enhanced generalization across diverse, natural driving conditions.
Multi-Spectral State Space Modeling for rPPG in Driver Monitoring
Introduction
This work addresses the limitations of camera-based remote photoplethysmography (rPPG) for health monitoring, focusing on heart rate estimation in unconstrained driver monitoring scenarios. Conventional rPPG methods, often relying exclusively on RGB cameras, are highly sensitive to environmental variations in illumination and subject motion—challenges prevalent in real-world driving. The authors propose MS-rPPG, a multi-spectral framework that fuses RGB and near-infrared (NIR) facial video streams and employs a novel Multi-Spectral Mamba (MS-Mamba) module, both designed to improve rPPG robustness and accuracy under realistic conditions (2606.21115).
Technical Approach
Multi-Spectral Feature Fusion with CSLM
The authors introduce a Cross-Spectral Linear Modulation (CSLM) mechanism for adaptive fusion of RGB and NIR features. CSLM computes feature-wise modulation parameters in the frequency domain, leveraging physiological priors to retain cardiac-related signal components within the relevant heart rate band (40–200 bpm). By modulating each modality with complementary information from the other, CSLM dynamically balances the high-SNR but illumination-sensitive nature of RGB and the illumination-robust but low-SNR nature of NIR.
Multi-Spectral State Space Modeling
Building on the state space model (SSM) family and the recent Mamba paradigm, the MS-Mamba module employs time-axis and channel-axis selective state space modeling optimized for long-range sequence dependencies with linear computational complexity. Compared to CNN and ViT, SSMs enable efficient global temporal context aggregation. The MS-Mamba module processes RGB and NIR streams through shared and independent SSM layers, then fuses them and applies bidirectional SSM scanning along the channel axis. This design explicitly models both temporal and inter-channel relationships, capturing richer spectral-temporal interactions than prior SSM-based (e.g., PhysMamba, RhythmMamba) or ViT-based approaches.
Supervisory Signal
The model is trained with a composite loss comprising a negative Pearson correlation loss (to enforce time-domain signal alignment) and a frequency-domain cross-entropy loss (to constrain the spectral concentration on physiologically plausible heart rates). This dual-domain supervision ensures both plausible waveform reconstruction and accurate HR prediction.
Experimental Evaluation
Real-World Datasets
Evaluation is performed on the public MR-NIRP Car dataset and a newly collected MS-Drive dataset, the latter containing synchronized RGB/NIR video and ECG ground-truth from 50 participants of diverse skin types. The MS-Drive dataset captures challenging, naturalistic driving conditions with substantial illumination variation and subject motion.
Intra- and Cross-Dataset Results
MS-rPPG establishes new state-of-the-art performance on both datasets. On MR-NIRP Car (975nm NIR, small motion), MS-rPPG achieves MAE = 4.32 and Pearson r = 0.728, outperforming prior CNN, ViT, and SSM-based rPPG models by significant margins. Under large motion, the method demonstrates enhanced robustness (MAE reduction by up to 30.5 points compared to SSM baselines), indicating effective motion and illumination desensitization.
Cross-dataset experiments (trained on MR-NIRP, tested on MS-Drive) highlight the generalization capabilities of the approach. MS-rPPG achieves MAE = 11.797 (small motion) and 9.601 (large motion), consistently outperforming all compared methods. Ablation studies confirm the crucial role of both CSLM and bidirectional channel SSM scanning, with performance drops observed when replacing CSLM with naive convolutional fusion or omitting channel-wise SSM operations.
Modality Contribution Analysis
The inclusion of both RGB and NIR was validated to be essential: removal of NIR leads to minor performance degradation under nominal conditions, but its contribution grows in challenging settings. The multi-spectral approach consistently surpasses single-modality baselines.
Implications and Future Directions
The multi-spectral SSM paradigm substantially improves the reliability of contactless HR monitoring in noisy, real-world vehicular environments. This enables practical deployment of camera-based driver state and health monitoring systems without the traditional constraints of PPG sensors or the instability of unimodal RGB approaches. The bidirectional channel SSM offers a template for future multi-modal temporal encoding strategies, especially in other physiological sensing or multi-sensor fusion applications.
Future directions include extending the framework for additional vital signs (e.g., respiration), integration with privacy-preserving on-device implementations, and adaptation to larger-scale, population-diverse datasets. The efficiency of SSM modules also paves the way for edge deployment in embedded driver assistance systems.
Conclusion
MS-rPPG leverages frequency-domain, cross-spectral feature modulation and multi-spectral state space modeling to set a new benchmark in rPPG-based driver monitoring. The fusion of RGB and NIR with bidirectional temporal and channel-wise SSM yields superior robustness to motion, illumination, and subject diversity, as evidenced by strong empirical performance on both controlled and naturalistic datasets. The methodology constitutes a significant advance in making remote vital-sign monitoring viable in unconstrained, safety-critical applications.
