SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals (2405.17766v1)

Abstract: Sleep is a complex physiological process evaluated through various modalities recording electrical brain, cardiac, and respiratory activities. We curate a large polysomnography dataset from over 14,000 participants comprising over 100,000 hours of multi-modal sleep recordings. Leveraging this extensive dataset, we developed SleepFM, the first multi-modal foundation model for sleep analysis. We show that a novel leave-one-out approach for contrastive learning significantly improves downstream task performance compared to representations from standard pairwise contrastive learning. A logistic regression model trained on SleepFM's learned embeddings outperforms an end-to-end trained convolutional neural network (CNN) on sleep stage classification (macro AUROC 0.88 vs 0.72 and macro AUPRC 0.72 vs 0.48) and sleep disordered breathing detection (AUROC 0.85 vs 0.69 and AUPRC 0.77 vs 0.61). Notably, the learned embeddings achieve 48% top-1 average accuracy in retrieving the corresponding recording clips of other modalities from 90,000 candidates. This work demonstrates the value of holistic multi-modal sleep modeling to fully capture the richness of sleep recordings. SleepFM is open source and available at https://github.com/rthapa84/sleepfm-codebase.

• The paper introduces SleepFM, integrating brain, ECG, and respiratory signals using pairwise and leave-one-out contrastive learning to enhance sleep analysis.
• The model achieves robust results with a macro AUROC of 0.88 for sleep stage classification and 0.85 for SDB detection, outperforming traditional CNNs.
• The study highlights SleepFM's versatility in demographic prediction and few-shot learning, paving the way for more efficient, automated clinical workflows.

Multi-modal Representation Learning for Sleep: An Expert Overview

"SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals" presents a comprehensive paper leveraging multi-modal data obtained from polysomnography (PSG). The paper introduces SleepFM, a foundation model trained on a substantial dataset consisting of over 100,000 hours of multi-modal sleep recordings from over 14,000 participants. This paper focuses on enhancing sleep analysis using advanced machine learning techniques, particularly multi-modal contrastive learning (CL).

Background and Related Work

Sleep monitoring remains vital for diagnosing sleep disorders and assessing overall health across various physiological systems. Traditionally, sleep data analysis required manual inspection, which is labor-intensive and error-prone. Recent developments in supervised deep learning have automated many of these tasks, yet they largely focus on single-modality data and limited task-specific labels.

Contrastive learning has gained attention for its effectiveness in representation learning by maximizing alignment between data modalities. Previous efforts predominantly applied CL to uni-modal data, such as images or time-series ECG signals. Multi-modal CL, especially for diverse physiological data in sleep studies, has not been widely explored until the introduction of SleepFM.

Methodology

The dataset used in this paper was obtained from the Stanford Sleep Clinic, encompassing a wide age range of participants (2-91 years) and spanning various sleep measurement channels, including brain activity signals (EEG, EOG, EMG), ECG, and respiratory channels. The preprocessing approach involved segmenting sleep recordings into 30-second clips standardized at 256 Hz, with expert-labeled annotations for sleep stages and sleep-disordered breathing (SDB).

The SleepFM model utilizes three 1D convolutional neural networks (CNNs) to encode BAS, ECG, and respiratory data independently. Two contrastive learning frameworks, pairwise CL and leave-one-out CL, were employed to integrate these multi-modal embeddings.

Pairwise CL constructs contrastive tasks between each pair of modalities, pushing embeddings of matching clips closer while penalizing mismatched pairs.

Leave-one-out CL contrasts embeddings of each modality against the average embedding of the remaining modalities, promoting comprehensive multi-modal alignment.

Experimental Results

The paper presented multiple evaluations:

1. Demographic Attributes Classification:
   • Using logistic regression on SleepFM embeddings yielded superior performance in predicting age and gender compared to an end-to-end CNN. Notably, leave-one-out CL achieved the highest accuracy, reflecting the effective capture of demographic information from short PSG clips.
2. Retrieval Analysis:
   • SleepFM showed high efficacy in retrieving corresponding modality clips using another modality's embeddings. Specifically, pairwise CL excelled in retrieval tasks, further validating the rich and distinct representations learned.
3. Downstream Classification Tasks:
   • In sleep stage and SDB event classification, SleepFM outperformed traditional CNNs, especially models pre-trained with leave-one-out CL. For sleep stages, it achieved a macro AUROC of 0.88 versus 0.72 for CNN, and for SDB detection, an AUROC of 0.85 versus 0.69.
4. Few-Shot Learning:
   • SleepFM demonstrated strong performance even with limited training data, significantly outperforming CNNs across varying sample sizes. The evaluation confirmed the robustness and utility of the pre-trained embeddings for practical, low-data settings.
5. Multi-Modal Pretraining Benefits:
   • Ablation studies highlighted the superior performance of models trained on three modalities compared to those trained on single or paired modalities. The integrative approach significantly enhanced downstream task efficacy, underscoring the importance of comprehensive multi-modal training.
6. External Validation:
   • The model generalized well to an external dataset, exceeding the performance of a locally trained supervised CNN. This external validation underscores SleepFM's adaptability and potential applicability across diverse clinical settings.

Discussion and Implications

The success of SleepFM in leveraging multi-modal PSG data suggests profound implications for automating and enhancing sleep analysis. The paper highlights the importance of holistic data integration, demonstrating that joint modeling of BAS, ECG, and respiratory signals captures the complexity of physiological dynamics during sleep.

Practically, these findings can streamline clinical workflows, reducing manual annotation time and potential errors. Theoretically, the model’s ability to extract meaningful representations from large-scale unlabeled data opens avenues for future developments in foundation models for other physiological domains.

Future research should focus on extending multi-site and multi-modal pretraining, handling missing data, and exploring other sleep-related tasks. Potential investigations into other self-supervised learning methods might further optimize performance for specific applications.

The paper acknowledges its current limitations, particularly the need for broader validation across various clinical settings and data acquisition protocols. Addressing these limitations will be crucial for transitioning research findings into robust, real-world solutions.

In conclusion, the introduction of SleepFM marks a significant advancement in sleep paper methodologies, emphasizing the strength of multi-modal representation learning in capturing the intricacies of sleep physiology. This work not only demonstrates practical improvements over existing methods but also sets the stage for future innovations in the field.