MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

Abstract: Existing vision-text contrastive learning like CLIP aims to match the paired image and caption embeddings while pushing others apart, which improves representation transferability and supports zero-shot prediction. However, medical image-text datasets are orders of magnitude below the general images and captions from the internet. Moreover, previous methods encounter many false negatives, i.e., images and reports from separate patients probably carry the same semantics but are wrongly treated as negatives. In this paper, we decouple images and texts for multimodal contrastive learning thus scaling the usable training data in a combinatorial magnitude with low cost. We also propose to replace the InfoNCE loss with semantic matching loss based on medical knowledge to eliminate false negatives in contrastive learning. We prove that MedCLIP is a simple yet effective framework: it outperforms state-of-the-art methods on zero-shot prediction, supervised classification, and image-text retrieval. Surprisingly, we observe that with only 20K pre-training data, MedCLIP wins over the state-of-the-art method (using around 200K data). Our code is available at https://github.com/RyanWangZf/MedCLIP.

• The paper introduces a decoupled contrastive learning framework that replaces InfoNCE with a semantic matching loss to tackle unpaired medical datasets.
• It leverages state-of-the-art vision and text encoders, including Swin Transformer and BioClinicalBERT, for effective multimodal semantic alignment.
• Experiments on datasets like CheXpert and MIMIC-CXR show over a 10% accuracy improvement in zero-shot classification using only 10% of the data.

MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

Introduction

The paper "MedCLIP: Contrastive Learning from Unpaired Medical Images and Text" addresses the challenges of using contrastive learning in the medical domain, where the availability of paired image-text data is significantly limited compared to general-domain datasets. Traditional methods like CLIP are limited by their dependence on large amounts of such paired data. In contrast, MedCLIP introduces a novel approach that decouples images and texts, leveraging combinatorial data expansion and medical semantics to improve data efficiency and model performance.

Challenges and Novel Approaches

Medical image-text datasets often suffer from insufficient data and the presence of false negatives. This paper identifies key challenges:

1. Data Insufficiency: The scarcity of paired medical image-text data poses a significant hurdle. MedCLIP effectively addresses this by decoupling images and texts to utilize vast unpaired datasets, scaling the training data to a combinatorial magnitude.
2. Semantic Mismatch and False Negatives: Traditional methods like InfoNCE introduce false negatives by misclassifying semantically related but unpaired data as negatives. MedCLIP overcomes this by replacing InfoNCE with a semantic matching loss, grounded in medical domain knowledge, to align image and text semantics accurately.

   Figure 1: Illustration of challenges in medical image-text contrastive learning such as ignored datasets and false negatives.

Methodology

MedCLIP's architecture consists of a vision and text encoder system that utilizes domain-specific knowledge for semantic alignment between data sources.

• Vision and Text Encoders: It employs structures like Swin Transformer as its vision encoder and BioClinicalBERT for text encoding, ensuring embeddings are applicable across medical data modalities.
• Semantic Matching Loss: This novel loss function uses medical knowledge to develop a similarity matrix between separately encodable images and texts, reducing falsely classified negatives.

  Figure 2: Workflow of MedCLIP highlighting knowledge extraction, semantic similarity matrix construction, and embedding alignment.

Experiments and Results

Extensive experimentation across multiple datasets (CheXpert, MIMIC-CXR) demonstrates MedCLIP's superiority in several key metrics:

• Zero-shot Classification: MedCLIP achieved an average accuracy improvement of over 10% compared to baseline models using only 10% of the data used by competitors, demonstrating exceptional data efficiency and semantic transferability.
• Fine-tuning and Transferability: In downstream tasks, MedCLIP maintained strong performance, comparable to state-of-the-art methods even in zero-shot settings.

  Figure 3: Comparison of MedCLIP's zero-shot performance against baseline models using different volumes of pre-training data.

Implications and Future Directions

MedCLIP’s approach facilitates broader application of multimodal learning in the medical domain, showcasing notable improvements in data efficiency and semantic understanding. By effectively harnessing unpaired datasets and medical knowledge, MedCLIP sets a foundation for future innovations in medical AI, suggesting potential for further exploration in automated diagnosis and cross-domain knowledge applications.

Conclusion

MedCLIP successfully introduces a framework for decoupled contrastive learning, overcoming traditional limitations in medical image-text pre-training. Its innovative use of domain semantics and efficient data utilization positions MedCLIP as a strategic advancement in medical AI, facilitating more semantic-rich and resource-efficient model training. Future works could optimize semantic processing and further expand on this foundational work.