MedCT-VLM: CT Vision-Language Models

• MedCT-VLM is a framework for adapting contrastive vision-language models to 3D CT analysis, integrating radiology reports and fine-grained anatomical segmentation.
• It utilizes specialized 3D vision transformers and text encoders to align volumetric CT data with clinical reports using InfoNCE loss and region-level contrast.
• Advanced models incorporate lightweight design, robust losses, and multimodal fusion to achieve state-of-the-art zero-shot and few-shot diagnostic performance.

MedCT-VLM refers to the domain of contrastive vision-LLMs and foundation models specifically developed for medical computed tomography (CT) analysis. These systems adapt, extend, or specialize CLIP-style architectures—originally designed for natural images and general descriptive text—to handle the idiosyncratic data, modalities, and annotation schemes in clinical CT interpretation, including 2D and 3D inputs, radiology reports, and related information modalities such as speech.

1. Foundations and Key Architectures for Vision-Language Pretraining in CT

The cornerstone of MedCT-VLM is the pretraining of joint vision-LLMs using paired 3D CT volumes and radiology reports. Early efforts, such as CT-CLIP, substitute CLIP’s 2D ViT with a 3D vision transformer (CT-ViT) backbone, typically resampled and cropped to fixed volumetric dimensions (e.g., 480×480×240 voxels at 0.75×0.75×1.5 mm) and processed with non-overlapping 3D patch embeddings (Hamamci et al., 26 Mar 2024). Text encoders are specialized versions of transformers (e.g., CXR-BERT) pretrained on radiology reports. The training objective is a symmetric InfoNCE loss aligning CT volume and report embeddings, with a learnable temperature.

Recent models push these boundaries by decomposing both CT images and textual reports into localized (anatomy- or organ-level) representations. fVLM and CT-GLIP construct fine-grained image-text pairs (e.g., segmenting a volume into 104 anatomic regions via TotalSegmentator, pairing each with an organ-specific report sentence) and apply per-anatomy contrastive objectives, often with auxiliary segmentation losses to regularize anatomical structure (Shui et al., 24 Jan 2025, Lin et al., 23 Apr 2024). These models move beyond global volume-report pairing, capturing local context and allowing region-level zero-shot queries.

Further, OpenVocabCT introduces multi-granular contrastive learning by leveraging LLMs to parse diagnostic text into both report-level and organ-level prompts, strengthening fine-grained localization and semantic generalization (Li et al., 8 Mar 2025).

2. Robust Learning under Data Limitations: Lightweight Design and Risk-Sensitive Losses

Data scarcity and limited annotations are persistent challenges in MedCT-VLM. CT-CLIP addresses these via a paradigm that combines a frozen generalist CLIP encoder (ViT-L/14 trained on natural image–text pairs) with a compact, trainable multilayer perceptron (MLP) classifier (Lin et al., 13 Mar 2024). The use of Conditional Value at Risk (CVaR) as a robust loss focuses optimization on high-loss (difficult) samples, while Sharpness-Aware Minimization (SAM) is employed to flatten the loss landscape and improve generalization. Empirical results on COV19-CT-DB demonstrate that this strategy can yield substantial improvements in macro F1 score (e.g., 0.886 vs. 0.780 for baseline CNN+RNN in COVID-19 detection; +10.6 percentage points).

Semi-supervised extensions exploit teacher–student frameworks, where a teacher trained on labeled samples is used to pseudo-label large-scale unlabeled slices, which are then incorporated into student training, further boosting data efficiency.

3. Fine-Grained and Region-Level Alignment: Anatomical Decomposition and Negative Mining

Recent advances underscore the importance of region-level vision-language alignment. fVLM constructs explicit anatomy-aware contrastive loss functions for each anatomical structure/reported finding (Shui et al., 24 Jan 2025). This approach combats false negatives arising from frequently normal regions (not mentioned explicitly in reports) and diseases with shared features. Calibration via co-teaching or self-distillation is used to correct negative sampling biases: for anatomy–text pairs where both are normal, positives are defined accordingly; similar diseases across patients are soft-matched using model confidence.

CT-GLIP and OpenVocabCT similarly utilize organ masking, diagnostic phrase parsing, and hard negative dictionaries (abnormality-specific prompts) to optimize discriminative region-text associations (Lin et al., 23 Apr 2024, Li et al., 8 Mar 2025). Collectively, these strategies yield robust zero-shot and few-shot performance gains for both detection and segmentation.

4. Extensions: Beyond Report Text—Speech, Contrastive Label Embeddings, and Multimodal Fusion

MedCT-VLM research also addresses modalities beyond classic text reports. SpeechCT-CLIP introduces direct speech–image alignment, leveraging synthesized spoken radiology reports and a Whisper-based speech encoder (Buess et al., 24 Sep 2025). A knowledge distillation term from a pretrained text-image CT-CLIP model facilitates the transfer of alignment properties from text to speech, resulting in near-parity zero-shot performance for audio-driven diagnostic tasks (F1 improved from 0.623 to 0.705 internally; >88% of performance difference to text-based models recovered).

Other advances include:

• CT-CLIP-Driven Universal Models using CLIP label embeddings rather than one-hot vectors for segmentation, capturing anatomical/semantic relationships between structure labels and enhancing multi-organ and tumor segmentation (Liu et al., 2023).
• Integration with practical segmentation frameworks such as SAM2CLIP2SAM, where CLIP-driven mask selection guides zero-shot segmentation for downstream classification (e.g., COVID-19 detection) (Kollias et al., 22 Jul 2024).
• Augmentation schemes (language, image, and hierarchy) to adapt natural-image CLIP models to CT, providing substantial gains in organ/station-level classification accuracy (Kakkar et al., 31 May 2024).
• Multi-modal frameworks (X2CT-CLIP) bridging CXR, CT, and reports via tri-modal alignment, enabling transfer of multi-abnormality detection capabilities from CT-label supervision to X-ray encoders (You et al., 4 Mar 2025).

5. Quantitative Performance and Generalization

MedCT-VLMs deliver state-of-the-art results across a range of tasks:

• Zero-shot abnormality detection: CT-CLIP achieves AUROC gains of 0.099 (internal) and 0.082 (external) over prior fully supervised baselines (Hamamci et al., 26 Mar 2024); fVLM attains an average AUC of 81.3% on 54 diagnosis tasks (+12.9 pts vs. vanilla CLIP) (Shui et al., 24 Jan 2025).
• Organ/tumor segmentation: OpenVocabCT yields mean DSC 90.7% (TotalSegmentator, 104 organs), outperforming prior text-driven and vision-only models (Li et al., 8 Mar 2025).
• Region-level and fine-grained: fVLM and CT-GLIP consistently outpace global models and previous local-alignment models (+15–20% F1, +16–20% AUC in common-structure detection) (Lin et al., 23 Apr 2024, Shui et al., 24 Jan 2025).
• Few-shot adaptation/cross-modal transfer: X2CT-CLIP achieves cross-modal AUROC of 0.716 zero-shot and 0.843 for 50% few-shot fine-tuning (CT-RATE), surpassing baselines in cross-domain adaptation (You et al., 4 Mar 2025).
• Clinical outcome prediction: Cardiac-CLIP enables zero-shot and fine-tuned prediction of complex endpoints (e.g., acute coronary syndrome, functional stenosis) from cardiac CT, with prospective fine-tuned AUROC of 0.802 and substantial outperformance over 3D ViT baselines (Hu et al., 29 Jul 2025).

6. Limitations and Future Research Directions

Current MedCT-VLM systems face several technical and practical constraints:

• Most implementations focus on either slice-level (ignoring 3D context, e.g., CT-CLIP for COVID-19 detection (Lin et al., 13 Mar 2024)) or whole-volume approaches that may miss fine regional cues.
• Synthetic (TTS) speech datasets may not adequately capture radiologist dictation variability, and speech encoders are computationally demanding (Buess et al., 24 Sep 2025).
• Generalization to unseen terminology, rare pathologies, or non-chest domains remains incomplete, with text-side filtering (e.g., RadLex) sometimes discarding clinically relevant prompts (Li et al., 8 Mar 2025).
• Pretraining is typically task- or region-specific (e.g., cardiac CT only in Cardiac-CLIP), and further research is needed to create broad, domain-agnostic CT foundation models (Hu et al., 29 Jul 2025).
• Accurate handling of partial or weak supervision—common in large-scale clinical datasets—demands further development of robust calibration, uncertainty estimation, and semi-supervised learning strategies.

Looking forward, open directions include: joint optimization of text and image encoders; combining segmentation, detection, and report-generation within a unified MedCT-VLM; speech- and multi-modal integration into radiology PACS; federated training across institutions; and leveraging large-scale generative or question-answering fine-tuning to expand clinical utility.

7. Principal Models and Empirical Benchmarks

Model	Core Mechanism/Architecture	Highlighted Metric	Reference
CT-CLIP	3D ViT + CXR-BERT + InfoNCE	AUROC +0.099 (internal ZS, 18 abn.)	(Hamamci et al., 26 Mar 2024)
fVLM	Fine-grained anatomy-level contrastive	Avg AUC 81.3% (54 dx tasks, +12.9pts)	(Shui et al., 24 Jan 2025)
OpenVocabCT	LLM-generated prompts, multi-granular loss	Mean DSC 90.7% (104 organs, segmentation)	(Li et al., 8 Mar 2025)
CT-GLIP	Organ-masked alignment, abnormality dictionary	+15% F1, +16% AUC over vanilla CLIP	(Lin et al., 23 Apr 2024)
SpeechCT-CLIP	Whisper speech encoder, distill from text-CLIP	Audio F1 0.705 (ZS, ≈88% of text model)	(Buess et al., 24 Sep 2025)
Cardiac-CLIP	3D MAE pretrain + soft-label contrastive	ACS AUROC 0.802 (fine-tuned)	(Hu et al., 29 Jul 2025)
X2CT-CLIP	Tri-modal contrast (CXR, CT, report)	FS@50% AUC 0.843 (multi-abn, CT-RATE)	(You et al., 4 Mar 2025)

These benchmarks chart both the advances and the continuing areas for research in robust, generalizable, and clinically-integrated MedCT-VLMs.