Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Abstract: Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN-ViT hybrid encoder that replaces ViT's patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.

• The paper presents a novel disease-centric vision–language pretraining framework using a hybrid CNN–ViT encoder that effectively processes 3D CT scans.
• It introduces fine-grained disease-level semantic alignment via learnable query tokens and aggregated report-derived prototypes, enabling robust disentanglement of co-occurring pathologies.
• The paper demonstrates significant improvements with AUC gains over benchmarks and robust zero-shot performance, highlighting its clinical diagnostic potential.

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

Introduction

This paper presents a disease-centric vision–language pretraining (VLP) framework for 3D CT, termed CT-DiagVLM, that systematically advances the precision, interpretability, and scalability of medical vision–LLMs. The framework directly targets major limitations of prior work: inadequacy of visual backbones for 3D volumetric data and the semantic conflation of disparate pathologies within the same anatomical region. The authors’ technical contributions span a hybrid CNN–ViT visual encoder, disease-level contrastive alignment supervised by learnable condition queries, and a diagnosis-aware prompting method leveraging aggregated, report-derived prototypes for robust inference.

Disease-Centric Vision-Language Pretraining Architecture

The cornerstone of the approach is the combination of a 3D ResNet-18 backbone for local multi-scale feature extraction with a Vision Transformer (ViT) initialized from large-scale cross-modal models. This hybrid structure is proposed to inherit the global attention and transferability established by ViT pretraining, while efficiently encoding volumetric anatomical textures that are often missed by pure patchified ViT schemes when extended to 3D medical imaging. The patch embedding module of ViT is supplanted with a multi-scale fusion patch embedding (MSF-PE) block that integrates hierarchical CNN output at aligned spatial resolutions, producing token initializations enriched with local texture and structural detail. End-to-end anatomical segmentation supervision via a U-Net style decoder maintains spatial coherence and aids the alignments of anatomical regions.

Figure 1: The proposed disease-centric VLP framework, highlighting the CNN–ViT hybrid encoder, disease query tokens extracting semantic detail, and condition-specific contrastive alignment for fine-grained cross-modal supervision.

Fine-Grained Disease-Level Semantic Alignment

Unlike previous frameworks, which perform global or organ-level image-report contrastive learning, this work introduces learnable disease query tokens per organ. These queries dynamically extract condition-specific semantics from unstructured clinical reports via cross-attention, circumventing the need for manual report decomposition. The core learning signal is a disease-level contrastive loss: organ-wise visual embeddings are explicitly aligned with disease-specific textual prototypes and separated from other condition pairs. Presence of pathology is determined from either ground-truth annotation or automatically extracted labels using LLMs, providing flexibility for both curated and noisy real-world settings.

The result is the disentangling of co-occurring pathologies (e.g., “emphysema” vs “atelectasis” vs “nodules” within the lung), a granularity beyond that offered by organ-level alignment prevalent in methods such as fVLM and ViSD-Boost.

Robust Zero-Shot Inference via Diagnosis-Aware Prompting

To close the distributional gap between report-derived pretraining signals and downstream zero-shot diagnostic inference, the model constructs class prototypes not from artificial templates, but by retrieving and aggregating real clinical phrases associated with each disease-organ pair. For each protoype, ≥500 reports are averaged, ensuring robust and semantically consistent disease representations. During inference, similarity between visual and textual prototypes yields organ- and disease-specific diagnostic scores, delivering both diagnostic fidelity and robustness to prompt phrasing variations.

Figure 2: Model performance as a function of number of retrieved reports ($M$) for prototype aggregation, demonstrating plateauing of performance for large and diverse retrievals.

Experimental Results

CT-DiagVLM achieves a reported AUC of 84.4% on CT-RATE and 75.4% on Rad-ChestCT—gains of 5.1% and 5.4% over the strongest benchmarks, with improvements recapitulated across ACC, F1, and precision. Most notably, under a challenging 60-disease setting with LLM-generated pseudo-labels, the model attains an AUC of 85.6%, a 9.8% increase over the base model, highlighting strong transfer and robustness under label noise. The introduction of the diagnosis-aware prompt strategy further calibrates sensitivity and specificity, providing more clinically relevant decision boundaries, especially in low-prevalence and noisy-label regimes.

Ablation studies reveal that each architectural intervention is orthogonal and additive: the CNN–ViT backbone, diagnosis-aware prompting, and disease-level loss each yield measurable, independent improvements in model performance. Furthermore, visualization of activation maps supports that the learned representations are localized and semantically consistent with ground-truth pathological regions despite the absence of explicit disease localization supervision.

Figure 3: Activation maps from the model demonstrate emergent localization of pathological regions, achieved solely through cross-modal semantic alignment.

Practical and Theoretical Implications

The framework’s capacity for disease-level semantic disentanglement in 3D volumes directly addresses clinical needs for fine-grained, interpretable computer-aided diagnosis. The use of learnable queries, automatic report parsing with LLMs, and prototype-based inference matches the operational realities of medical AI, where comprehensive annotation is rare and label noise is intrinsic. The robustness to LLM-generated pseudo-labels and scalability to rare and low-prevalence pathologies further support practical deployment scenarios.

From a methodological standpoint, the study demonstrates that cross-modal priors learned in natural image-language pretraining (e.g., SigLIP2) can be efficiently and effectively transferred to the medical domain when architectural compatibility is preserved, supporting continued cross-fertilization between general-purpose foundation models and specialized clinical tasks. Moreover, the hybrid CNN–ViT design suggests a successful template for future multi-modal, multi-scale representation learning, especially where spatial detail and transferability are jointly required.

Limitations and Future Outlook

The approach is contingent on external tools (e.g., LLMs for label/report parsing and pseudo-label generation, third-party organ segmentation for pseudo masks), an increasingly standard but still externalized aspect of medical AI pipelines. Pretraining is framed as task-agnostic, so as downstream benchmarks for 3D medical VLP mature, further evaluation of domain robustness, rare disease sensitivity, and fairness may be warranted.

Further innovation may focus on endogenizing parsing and segmentation within the framework, closing the loop on extrinsic supervision, and leveraging broader cross-modal pretraining data. The diagnosis-aware prompt paradigm also motivates future work in learned prompt optimization, calibration, and unsupervised disease taxonomy discovery.

Conclusion

This work establishes a rigorous and scalable standard for vision–language pretraining in volumetric medical imaging. Through the co-design of a multi-scale CNN–ViT encoder, fine-grained disease-centric contrastive adaptation, and robust, report-derived prompting, CT-DiagVLM achieves state-of-the-art diagnostic performance, strong generalizability, and interpretable activation mapping. The innovations set forth offer a concrete path towards clinically reliable, zero-shot, and broadly applicable medical AI platforms for 3D CT analysis.

Reference: "Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography" (2606.25546)