QwenCLIP: Medical Vision-Language Pretraining

• QwenCLIP is a medical vision-language pretraining framework that replaces traditional CLIP text encoders with an 8-billion parameter LLM to process long radiology reports.
• By integrating a hybrid static and soft prompt tuning mechanism, it aligns comprehensive medical semantics with image embeddings effectively.
• Empirical results on ROCOv2 benchmarks show that QwenCLIP delivers state-of-the-art zero-shot retrieval performance while operating with a frozen text encoder.

QwenCLIP is a medical vision-language pretraining framework designed to address the limitations of standard Contrastive Language-Image Pretraining (CLIP) architectures in representing long, information-rich radiology reports. By replacing CLIP’s transformer-based text encoder with a LLM embedding module and introducing a hybrid prompt tuning scheme, QwenCLIP captures comprehensive medical semantics and achieves state-of-the-art results on zero-shot medical image–text retrieval benchmarks (Wei et al., 17 Nov 2025).

1. Framework Overview and Architectural Design

QwenCLIP introduces two core architectural innovations. First, the original CLIP text encoder—a GPT-2‐117M transformer limited to 77 tokens—is replaced with Qwen3-Embedding-8B, an 8-billion parameter LLM embedding model that natively supports context windows up to 1 million tokens. This modification eliminates both the 77-token bottleneck of CLIP and the 512-token restriction of BERT derivatives (e.g., PubMedBERT, ClinicalBERT). Qwen3-Embedding outputs a dense, high-dimensional text embedding in $\mathbb{R}^H$ with $H \gg 512$.

To align the text and image embedding spaces, QwenCLIP employs a two-layer MLP projection head $$f_{\mathrm{text}}:\mathbb{R}^H \rightarrow \mathbb{R}^{512}$$ to map LLM embeddings into the standard 512-dimensional space output by the CLIP ViT-B/16 vision encoder $H_{\mathrm{visual}}$. The image branch is left unchanged, using ViT-B/16 initialized from open CLIP weights and projecting images $I$ into $v = H_{\mathrm{visual}}(I) \in \mathbb{R}^{512}$. During pretraining, Qwen3 is frozen, and only the projection head, vision encoder, prompts, and temperature $\tau$ are updated.

2. Prompt Tuning Mechanism

QwenCLIP adopts a hybrid static and soft prompt tuning approach to bridge the LLM’s general-purpose language space with the medical vision–language alignment task. In the formed textual input, every caption $T$ is reformulated as

$$T' = \text{[“Instruct:”]} \oplus P_s \oplus \text{[“query:”]} \oplus T$$

where explicit tokens “Instruct:” and “query:” provide a fixed structural cue, and $P_s \in \mathbb{R}^{K \times E}$ is a learnable soft prompt with $H \gg 512$0 tokens, each matching Qwen3’s token embedding size $H \gg 512$1. $H \gg 512$2 is initialized from the phrase “Create a dense embedding that represents the medical meaning of this text for image retrieval.” and is updated by backpropagation during training.

The final text embedding is computed as

$H \gg 512$3

This hybrid prompt tuning demonstrates that explicit structure plus learnable tokens effectively adapts frozen LLM semantics for cross-modal alignment.

3. Training Objective and Optimization

QwenCLIP leverages the standard symmetric InfoNCE contrastive loss across a batch of $H \gg 512$4 image–text pairs $H \gg 512$5. Let $H \gg 512$6 and $H \gg 512$7, with $H \gg 512$8. The loss functions are: $H \gg 512$9

$$f_{\mathrm{text}}:\mathbb{R}^H \rightarrow \mathbb{R}^{512}$$0

$$f_{\mathrm{text}}:\mathbb{R}^H \rightarrow \mathbb{R}^{512}$$1

The temperature $$f_{\mathrm{text}}:\mathbb{R}^H \rightarrow \mathbb{R}^{512}$$2 is learnable. No auxiliary regularization beyond weight decay on prompts/projection head is used. Only the vision encoder, projection head, prompts, and $$f_{\mathrm{text}}:\mathbb{R}^H \rightarrow \mathbb{R}^{512}$$3 are updated during pretraining; the LLM text encoder is frozen.

4. Datasets and Evaluation Protocols

QwenCLIP is pretrained exclusively on ROCOv2 (Radiology Objects in COntext v2), comprising 60,163 train, 9,945 validation, and 9,972 test image–caption pairs. Captions average 32 tokens, with 4.4% exceeding CLIP's 77-token limit. No external radiology report datasets are incorporated.

Downstream performance is assessed in a zero-shot image-to-image retrieval setting:

• ROCOv2 CUI@K: NDCG over test set, using shared UMLS Concept Unique Identifiers (CUIs) for rank quality.
• Precision@K on custom ROCOv2 subsets: Task-based retrieval using categorical CUI metadata (Modality, Organ, Modality+Organ).
• IRMA Precision@K: Retrieval on the IRMA dataset (14,000 X-ray images), evaluated via IRMA code agreement.

5. Empirical Results and Comparisons

QwenCLIP establishes new state-of-the-art results across multiple retrieval benchmarks:

Metric	QwenCLIP	CLIP	ClinicalCLIP	PMC-CLIP	LLM2CLIP
ROCOv2 CUI@5	45.96	44.64	45.18	45.58	45.43
ROCOv2 CUI@10	47.23	45.90	46.62	46.93	46.72
ROCOv2 CUI@50	52.47	50.53	50.97	51.71	51.78
Modality P@5	96.24	94.41	-	-	-
Organ P@5	85.99	83.70	-	-	-
Modality+Organ P@5	91.35	89.11	-	-	-
IRMA Organ retrieval	98.49	97.08	-	-	-

Ablation studies reveal that upgrading to LLM (with prompts frozen) already yields a major improvement (CUI@5: 45.63, +0.99 vs. CLIP), while adding learnable prompt tuning provides a further 0.3–0.5 point increment.

6. Analysis, Strengths, and Limitations

The principal findings are:

• LLM embeddings fully resolve the input truncation issue and produce richer semantic features, outperforming both vanilla CLIP and BERT-scale medical adapters.
• Hybrid prompt tuning effectively repurposes frozen LLMs for vision–language representation.
• The frozen-encoder strategy optimizes memory and computation, allowing pretraining on a single A100 80 GB GPU (batch 32, 10 epochs).

QwenCLIP is scalable to arbitrarily long radiology reports and supports end-to-end contrastive training without a two-stage LLM adaptation procedure (in contrast to LLM2CLIP).

However, reliance on a large frozen LLM results in higher inference latency and resource requirements than BERT-scale models. Current evaluation is restricted to retrieval tasks, and extension to classification, grounding, or report generation is not explored.

Future directions include:

• Fine-tuning lower layers of the LLM embedding module for further adaptation.
• Incorporating regional alignment (e.g., lesion bounding boxes).
• Model distillation into lighter embedding architectures for deployment.
• End-to-end training on multi-section, real clinical reports drawn from PACS systems (Wei et al., 17 Nov 2025).