Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning

Abstract: Accurate prediction of cardiovascular diseases remains imperative for early diagnosis and intervention, necessitating robust and precise predictive models. Recently, there has been a growing interest in multi-modal learning for uncovering novel insights not available through uni-modal datasets alone. By combining cardiac magnetic resonance images, electrocardiogram signals, and available medical information, our approach enables the capture of holistic status about individuals' cardiovascular health by leveraging shared information across modalities. Integrating information from multiple modalities and benefiting from self-supervised learning techniques, our model provides a comprehensive framework for enhancing cardiovascular disease prediction with limited annotated datasets. We employ a masked autoencoder to pre-train the electrocardiogram ECG encoder, enabling it to extract relevant features from raw electrocardiogram data, and an image encoder to extract relevant features from cardiac magnetic resonance images. Subsequently, we utilize a multi-modal contrastive learning objective to transfer knowledge from expensive and complex modality, cardiac magnetic resonance image, to cheap and simple modalities such as electrocardiograms and medical information. Finally, we fine-tuned the pre-trained encoders on specific predictive tasks, such as myocardial infarction. Our proposed method enhanced the image information by leveraging different available modalities and outperformed the supervised approach by 7.6% in balanced accuracy.

• The paper introduces a four-step multi-modal self-supervised learning framework that integrates ECG, CMRI, and tabular data to improve CVD prediction accuracy by 7.6%.
• The methodology employs Masked Autoencoder pre-training for ECG and SimCLR for CMRI images to robustly extract modality-specific features.
• The integration of diverse data sources reduces reliance on extensive labeled datasets, enhancing model scalability and clinical applicability.

Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning

Introduction

The study titled "Enhancing Cardiovascular Disease Prediction through Multi-Modal Self-Supervised Learning" aims to address the inaccuracies typically inherent in predictive models focusing on cardiovascular diseases (CVDs). Using a multi-modal approach incorporating cardiac magnetic resonance images (CMRI), electrocardiogram (ECG) signals, and supplementary medical information, it seeks to enhance the predictive accuracy and comprehensiveness of CVD models. By capitalizing on shared information across these varied modalities, the proposed system facilitates improved insights into individual cardiovascular health, achieving integration not possible with uni-modal datasets.

Methodology Overview

The methodology employs a four-step computational framework that integrates self-supervised contrastive learning with multi-modal data. Key steps include:

1. Pre-training ECG Encoder:

The ECG signal encoder is pre-trained using a Masked Autoencoder (MAE) approach. This technique fortifies the model's ability to extract relevant features from raw ECG data through a masking and reconstruction process.

Figure 1: Visualization of the pre-training pipeline for ECG signals. The signal is divided into patches and a fraction of them is masked.

2. Image Encoder Pre-training:

Utilizing a ResNet50 backbone, the image encoder is optimized with SimCLR loss. This step is crucial for creating a robust representation of CMRI images, maximizing latent space agreement between augmented image views.

Figure 2: Visualization of the pre-training pipeline for CMRI.

3. Multi-modal Contrastive Learning:

The learned representations from ECG and tabular data are merged and aligned within a shared latent space. Through Cross-Modal Contrastive Learning, information from CMRIs is efficiently transferred to cheaper, more accessible modalities like ECG and tabular data.

Figure 3: Visualization of the multi-modal pre-training pipeline.

4. Fine-tuning:

Pre-trained encoders are subjected to supervised fine-tuning, improving predictive performance for specific cardiovascular pathologies such as myocardial infarction (MI).

Figure 4: Pre-trained signal and tabular encoders are fine-tuned in a supervised manner.

Results and Comparisons

The proposed method demonstrates a significant 7.6% improvement in balanced accuracy over standard supervised techniques when applied to a comprehensive UK Biobank clinical dataset. In the comparative analysis across various modalities (CMRI, ECG, and tabular data), the results showcase the heightened predictive capabilities stemming from the integration of multiple data sources. These findings imply that MAE pre-training confers enhanced standing to the ECG-based models, further reinforced by the assimilation of data from CMRI.

In an evaluation against supervised learning paradigms using Neural Networks (NN), the self-supervised approach presented in this work exhibited superior performance, especially given the scarcity of labeled data available for CV predictions. This underscores the utility and efficiency of transfer learning from substantive modalities like CMRI to others that are more readily accessible.

Implications and Future Directions

The implications of this research extend into several critical areas within healthcare and AI:

• Healthcare Integration: By employing self-supervised learning techniques, the reliance on large annotated datasets diminishes. This advancement widens applicability in real-world clinical settings, especially where labeled data is sparse.
• Model Scalability: The presented approach leverages the heterogeneity of modalities to glean a fuller picture of the physiological state, making model applications versatile and diagnostically potent.
• Further Explorations: Future research may encompass applying SSL methods to other easily accessible patient data such as textual information from electronic health records (EHRs), thus elevating diagnostic models' comprehensiveness and specificity.
• Dataset Diversity: While predominantly validated on the UK Biobank dataset, future endeavors should factor in diverse demographic data for improved generalizability and clinical applicability.

Conclusion

The paper introduces a multi-modal self-supervised learning framework that significantly improves cardiovascular disease prediction accuracy. By transcending uni-modal limitations and harnessing the synergistic potential of diverse data types, this approach paves the way for enhanced early diagnosis and strategic interventions in clinical contexts. Limitations pertaining to dataset demographics stand acknowledged, asserting the importance of broadening dataset diversity for future implementations.