MedVista3D: Multi-Scale CT Vision-Language Model

• MedVista3D is a multi-scale, semantic-enriched vision-language foundation model for chest CT analysis that unifies global and local representations to address under-reading, inattentional blindness, and communication failures.
• It employs a dual-pathway design with a 3D vision encoder and specialized text encoder, using transformer blocks and LLM-based report rewriting to enhance diagnostic precision.
• Empirical evaluations show that MedVista3D improves zero-shot disease classification, report retrieval, and transfer performance compared to models focusing solely on global or local cues.

Searching arXiv for MedVista3D and closely related 3D medical visualization / 3D VLM papers. MedVista3D is a multi-scale semantic-enriched vision-language pretraining framework for 3D CT analysis, introduced for chest CT to address radiologic diagnostic errors associated with under-reading, inattentional blindness, and communication failures. Its central design couples local and global image-text alignment over full 3D volumes with LLM-based report rewriting and a Radiology Semantic Matching Bank, yielding a single framework that supports zero-shot disease classification, report retrieval, medical visual question answering, and transfer to organ segmentation and prognosis prediction (Li et al., 4 Sep 2025).

1. Clinical motivation and conceptual scope

MedVista3D is explicitly organized around three classes of radiologic error. Under-reading errors correspond to missed local findings such as small tumors or focal opacities. Inattentional blindness refers to failures of global integration across the full scan. Communication failures arise when findings are described with ambiguous language, inconsistent terminology, or incomplete statements. The framework treats these not as separate downstream problems, but as coupled failures of representation learning in 3D CT and radiology text (Li et al., 4 Sep 2025).

The model is positioned against two pre-existing tendencies in 3D CT vision-language modeling. One class of systems is global only, exemplified in the paper by CT-CLIP, which aligns whole volumes with entire reports and is therefore effective for volume-level retrieval and classification but weaker for local lesion localization. Another class is local only, exemplified by fVLM, which aligns organ-wise regions with organ-wise descriptions and is therefore effective for local detection but weaker for holistic interpretation. MedVista3D is designed as a multi-scale, semantic-enriched vision-language pretraining framework that learns local and global alignment jointly rather than privileging one scale over the other (Li et al., 4 Sep 2025).

This architecture gives MedVista3D a dual identity. In the narrow sense, it is a chest-CT foundation model. In the broader sense, it belongs to a class of systems that aim to connect volumetric representation, semantic supervision, and clinically meaningful interaction. That broader class includes browser-based brain MRI pipelines, multi-user VR planning environments, AR volumetric viewers, and other 3D medical VLMs; those systems do not implement MedVista3D itself, but they illuminate the larger technical ecosystem in which it sits.

2. Dual-pathway architecture and multi-scale representation learning

MedVista3D consists of a 3D vision encoder $f_I$ and a text encoder $f_T$. The vision encoder is either a 3D ViT-B with 768-dimensional embeddings and 12 transformer blocks, or a 3D UniMISS-Small backbone. The text encoder is BiomedVLP CXR-BERT-specialized. The framework separates representation learning into a global pathway and a local pathway, both derived from the same CT volume $x_i \in \mathbb{R}^{1 \times D \times H \times W}$ (Li et al., 4 Sep 2025).

In the global pathway, a 3D convolution produces patch embeddings

$p_i \in \mathbb{R}^{c \times d \times h \times w},$

which are processed by transformer blocks into a latent representation $v_i^L$, and then into a global image embedding $v_i^G$. The paired report $y_i$ is encoded as a global text embedding $t_i^G = f_T(y_i)$. This pathway provides full-volume context, which is the mechanism by which MedVista3D attempts to mitigate inattentional blindness (Li et al., 4 Sep 2025).

In the local pathway, each anatomical region $r$ is associated with a segmentation mask $M_i^r \in \{0,1\}^{D \times H \times W}$. The mask is downsampled by mask pooling to

$f_T$0

and the active tokens from $f_T$1 are passed through the last transformer block to produce a local image embedding $f_T$2. The corresponding region-level text phrase $f_T$3 is encoded as

$f_T$4

Because the local tokens are selected from the globally contextualized token grid, the local representation is not independent of whole-volume context; this is a defining feature of the architecture rather than a post hoc fusion stage (Li et al., 4 Sep 2025).

The pretraining objective is framed in terms of mutual information. The model defines global mutual information $f_T$5 and local mutual information $f_T$6, then introduces a unified quantity

$f_T$7

The multi-scale contrastive loss is

$f_T$8

where the global and local terms are symmetric CLIP-like InfoNCE losses over cosine similarity. The stated motivation is that optimizing only one scale is suboptimal, whereas the unified objective is intended to preserve both local anomaly sensitivity and global contextual reasoning (Li et al., 4 Sep 2025).

3. Semantic enrichment and the Radiology Semantic Matching Bank

A distinctive aspect of MedVista3D is that it does not treat raw radiology reports as sufficiently clean supervision. The paper characterizes CT-RATE reports as long, unstructured, and linguistically variable, with synonyms, vague phrasing, and inconsistent mention of disease presence or absence. To reduce communication failures at the supervision level, the framework uses LLMs, specifically GPT-4o and Qwen2.5, to extract abnormalities and rewrite findings into discrete, explicit presence/absence statements. The same rewriting procedure is applied at the organ level using few-shot prompts (Li et al., 4 Sep 2025).

This process yields a global enriched report $f_T$9 with embedding $x_i \in \mathbb{R}^{1 \times D \times H \times W}$0, and local enriched region text $x_i \in \mathbb{R}^{1 \times D \times H \times W}$1 with embedding $x_i \in \mathbb{R}^{1 \times D \times H \times W}$2. The enriched text is not described as an ontology mapping or a RadLex-style normalization pipeline; instead, it is a semantics-preserving rewrite procedure designed to reduce stylistic and lexical variability while keeping medically precise content (Li et al., 4 Sep 2025).

The second semantic component is the Radiology Semantic Matching Bank (RSMB), defined as a 64K-sized first-in-first-out queue of text embeddings. It stores global and local enriched embeddings from prior batches. For each new enriched embedding, the system performs nearest-neighbor retrieval in the bank using cosine similarity, obtaining a semantically similar description with potentially different wording. These retrieved neighbors become contrastive targets in additional semantic alignment losses. The global semantic term aligns $x_i \in \mathbb{R}^{1 \times D \times H \times W}$3 with $x_i \in \mathbb{R}^{1 \times D \times H \times W}$4, and the local semantic term aligns $x_i \in \mathbb{R}^{1 \times D \times H \times W}$5 with $x_i \in \mathbb{R}^{1 \times D \times H \times W}$6 (Li et al., 4 Sep 2025).

The full MedVista3D objective is written as

$x_i \in \mathbb{R}^{1 \times D \times H \times W}$7

with

$x_i \in \mathbb{R}^{1 \times D \times H \times W}$8

The first term aligns raw reports and phrases at both scales; the second aligns enriched, standardized semantics. The intended effect is robustness to synonymy, reporting style variation, and noisy free text. In the language of the paper, semantic alignment is the principal mechanism by which MedVista3D addresses communication failures in 3D radiology workflows (Li et al., 4 Sep 2025).

4. Data regime, preprocessing, and downstream capabilities

Primary pretraining uses CT-RATE, with 24,128 chest CT volumes for training and 1,564 volumes as an internal test split, together with radiology reports and CT-based VQA annotations. RadGenome-Chest CT provides organ segmentation masks and region-level phrases for local alignment. Volumes are resampled to $x_i \in \mathbb{R}^{1 \times D \times H \times W}$9 mm, intensities are normalized following CT-CLIP, and inputs are padded or center-cropped to a fixed size of $p_i \in \mathbb{R}^{c \times d \times h \times w},$0. The model encodes full volumes rather than sub-volumes, which is presented as essential for preserving global context (Li et al., 4 Sep 2025).

Training uses AdamW with weight decay $p_i \in \mathbb{R}^{c \times d \times h \times w},$1, learning rate $p_i \in \mathbb{R}^{c \times d \times h \times w},$2 with linear warmup for 200 steps and cosine decay, batch size 32 for MedVista3D-ViT and 20 for MedVista3D-UniMISS, and 16 epochs of pretraining on NVIDIA A100 80GB GPUs. The architecture is therefore computationally substantial but not described as multi-stage in the style of some earlier 3D VLM pipelines; instead, its complexity lies in simultaneous global, local, and semantic alignment (Li et al., 4 Sep 2025).

The framework supports several task families. In zero-shot disease classification, the global embedding $p_i \in \mathbb{R}^{c \times d \times h \times w},$3 is compared with disease prompts such as “There is pulmonary embolism” and “No pulmonary embolism,” and the local embedding $p_i \in \mathbb{R}^{c \times d \times h \times w},$4 supports organ-specific zero-shot inference when masks are available. In report retrieval, CT embeddings are matched against report embeddings with recall@5 and recall@10 evaluation. In medical VQA and report generation, the authors build MedVista3D-LLaVA by pairing pretrained MedVista3D-ViT with LLaMA-3.1-7B and a 2-layer MLP-GELU projector, followed by projector alignment and supervised finetuning with LoRA of rank 128 and $p_i \in \mathbb{R}^{c \times d \times h \times w},$5 (Li et al., 4 Sep 2025).

The representation also transfers beyond vision-language tasks. For organ segmentation on TotalSegmentator with 1204 CTs and 104 organs, MedVista3D-UniMISS is used as encoder with an STU-Net-B decoder in the nnUNet framework, trained on $p_i \in \mathbb{R}^{c \times d \times h \times w},$6 voxels and evaluated with Dice similarity coefficient. For prognosis prediction on STOIC 2021, a linear classifier is trained on top of MedVista3D-UniMISS for binary severe versus non-severe COVID prediction using AUROC (Li et al., 4 Sep 2025).

5. Empirical performance, ablations, and qualitative behavior

On the CT-RATE internal test set, MedVista3D improves global zero-shot disease classification over CT-CLIP and fVLM. For global disease zero-shot, CT-CLIP reports AUC 0.704, ACC 0.651, F1 0.691; fVLM reports AUC 0.591 and F1 0.684; MedVista3D-ViT reports AUC 0.778, ACC 0.737, F1 0.760; and MedVista3D-UniMISS reports AUC 0.782, ACC 0.745, F1 0.770. The paper explicitly summarizes the improvement of MedVista3D-UniMISS over CT-CLIP as AUC: +7.8 points and F1: +7.9 points (Li et al., 4 Sep 2025).

For report retrieval, CT-CLIP yields recall@5: 2.34% and recall@10: 3.95%, whereas MedVista3D-ViT reaches recall@5: 6.64% and recall@10: 10.68%, and MedVista3D-UniMISS reaches recall@5: 5.01% and recall@10: 8.65%. The paper states that MedVista3D therefore roughly doubles to triples retrieval recall compared with CT-CLIP (Li et al., 4 Sep 2025).

On local zero-shot tasks in CT-RATE, fVLM remains strong, but MedVista3D preserves competitive or slightly better performance while simultaneously retaining global capacity. fVLM reports AUC 0.778, F1 0.751, ACC 0.718; MedVista3D-ViT reports AUC 0.780, F1 0.765, ACC 0.742; and MedVista3D-UniMISS reports AUC 0.753, F1 0.754, ACC 0.726. The paper interprets this as empirical support for the claim that the multi-scale design mitigates both under-reading and inattentional blindness rather than trading one for the other (Li et al., 4 Sep 2025).

External generalization is evaluated on Rad-ChestCT with 3,626 CTs. For global zero-shot tasks, MedVista3D-UniMISS achieves AUC 0.713, ahead of CT-CLIP at 0.632, Merlin at 0.596, and CLIP at 0.609. For local zero-shot tasks, MedVista3D-ViT achieves AUC 0.710, ahead of the fVLM value of 0.680, while MedVista3D-UniMISS reaches 0.697, stated as ~0.017 above fVLM (Li et al., 4 Sep 2025).

The language-conditioned model MedVista3D-LLaVA also improves VQA and report generation. On CT-RATE VQA, long answer BLEU-1 rises from 0.480 for CT-CHAT to 0.516; short answer BLEU-1 rises from 0.280 to 0.299; report generation BLEU-1 rises from 0.381 to 0.474, which the paper summarizes as +9.3 points; and multiple choice BLEU-1 rises from 0.838 to 0.936, with 91.5% multiple-choice accuracy for MedVista3D-LLaVA (Li et al., 4 Sep 2025).

Ablation analysis isolates the contribution of each component. Starting from Global alignment only, the model reports AUC 0.675, F1 0.689, and extremely weak region grounding. Adding Local alignment improves region grounding but introduces a small trade-off in global metrics. Adding Mask pooling further improves grounding. Adding Global semantic alignment raises global zero-shot performance to AUC 0.807, F1 0.789, ACC 0.758. The final model, with Local semantic alignment added, yields the best overall balance: AUC 0.778, F1 0.760, ACC 0.737, region grounding top-10: 0.83%, top-50: 3.46%, and report retrieval recall@5: 6.64%, recall@10: 10.68% (Li et al., 4 Sep 2025).

Attention visualizations provide qualitative support for the architecture. With organ masks, fVLM and MedVista3D both focus correctly, while CLIP is diffuse. Without masks, fVLM focuses on irrelevant background, CLIP remains diffuse, and MedVista3D focuses on relevant anatomy even without explicit masks. The paper treats this as evidence that MedVista3D combines precise local grounding with global context (Li et al., 4 Sep 2025).

Transfer performance reinforces the representation-learning claim. On TotalSegmentator, MedVista3D-UniMISS reaches Dice 0.872, ahead of nnUNet 0.852, Merlin 0.860, CT-CLIP 0.805, and M3D 0.597. On STOIC prognosis, it reaches AUC 0.807, ahead of CT-CLIP 0.631, Merlin 0.782, RadFM 0.649, and M3D 0.627 (Li et al., 4 Sep 2025).

6. Relation to adjacent systems and the broader MedVista3D design space

Several adjacent research threads provide context for what MedVista3D is not, and what it could plausibly be combined with. A browser-based brain MRI system described in “Interactive Manipulation and Visualization of 3D Brain MRI for Surgical Training” builds an end-to-end pipeline from raw NIfTI MRI through SynthSeg, 3D Slicer Flying Edges, and a Three.js frontend, with Django + SQLite orchestration and CPU-based segmentation for compatibility. Its primary use case is human brain haptic feedback simulation for surgeon training, with secondary use cases in clinical data analysis, teaching, and surgical planning (Jha et al., 2024). MedVista3D does not provide such a visualization stack, but this demonstrates an operational route from segmentation and reconstruction to interactive 2D/3D access.

“Immersive Anatomical Scenes that Enable Multiple Users to Occupy the Same Virtual Space” describes a different axis of the design space: Mimics-based segmentation, Unity3D and SteamVR, HTC Vive, and Photon Bolt networking for shared immersive anatomical scenes across local or distributed networks. That system targets pre-surgical planning, anatomical education, and device visualization, and emphasizes transform-only networking, avatar awareness, and laser-pointer collaboration rather than vision-language learning (Deakyne et al., 2020). This suggests that MedVista3D’s learned 3D CT representations could, in principle, be coupled to collaborative XR environments, although such an integration is not present in the MedVista3D paper.

At the rendering and interaction level, HoloView contributes a thin-client AR architecture with remote stereo rendering, eye-tracked foveated rendering, and hybrid surface-volume rendering on HoloLens 2. It uses cryosection data from the Visible Korean Human dataset, surface meshes, gesture-only interaction, and reports approximately 60 fps with foveated rendering compared with about 7 fps at native full-resolution stereo streaming (Kaushik et al., 15 Jan 2025). A separate XR system for coordinated 2D–3D visualization of volumetric medical data couples Multi-layered Multi-planar Reconstruction, 3D meshes, and LLM-enabled voice commands on Meta Quest 3, reporting 63–83 FPS and average memory usage of ~1.27 GB, together with lower NASA-TLX workload in the full multimodal condition (Liu et al., 28 Jun 2025). These systems do not perform MedVista3D-style representation learning, but they define mature front-end paradigms for immersive display and multimodal control.

Within 3D vision-language modeling itself, Med3DVLM is the most direct neighboring system. It targets whole-volume 3D medical image-text retrieval, report generation, and VQA using DCFormer, SigLIP, a dual-stream MLP-Mixer projector, ClinicalBERT, and Qwen2.5-7B-Instruct, trained on the M3D dataset of 120,084 3D medical images (Xin et al., 25 Mar 2025). Compared with MedVista3D, Med3DVLM emphasizes efficient whole-volume 3D encoding and multi-task generative capability across CT volumes, whereas MedVista3D emphasizes local-global alignment in chest CT and semantic enrichment of radiology reports. This suggests complementary strategies rather than interchangeable ones.

Taken together, these neighboring works show a broader convergence: web-based 2D/3D viewing, immersive XR interaction, collaborative anatomy exploration, and 3D medical vision-language modeling are increasingly overlapping research directions. MedVista3D occupies the representation-learning core of that convergence rather than its visualization or telepresence endpoints.

7. Limitations, assumptions, and prospective extensions

The explicit limitations of MedVista3D are narrow but significant. Pretraining is limited to chest CT, and only CT is used; no MRI, PET, or other modalities are incorporated. Evaluation is retrospective on public datasets, with no prospective clinical study or deployment, so real-world reduction of diagnostic errors remains unproven. The semantic rewriting pipeline introduces another risk: although the LLM rewrites are used only for training supervision, they may still introduce hallucinations if the original report is misinterpreted (Li et al., 4 Sep 2025).

The framework also assumes access to organ segmentation masks, or at least reliable automatic organ segmentation such as TotalSegmentator, for local tasks. That assumption is not unusual in chest CT pipelines, but it means MedVista3D’s local pathway is not purely self-supervised. The paper’s strongest local behavior is achieved when masking is applied before the last transformer block, where the ablation reports local AUC 0.780, compared with 0.703 when masking is applied before the first block and 0.732 before the second block (Li et al., 4 Sep 2025).

Future directions stated in the paper include extending the model to other anatomical sites such as brain, head-and-neck, and abdomen/pelvis; extending it to other modalities such as MRI and PET; integrating it into clinical workflows such as PACS and RIS; and exploring broader multi-modal integration, including EHR data plus imaging (Li et al., 4 Sep 2025). A plausible implication is that the visualization and XR systems cited above could serve as downstream interfaces for MedVista3D-like encoders, while MedVista3D itself could provide the semantic and representational substrate for decision support, report drafting, and CT-aware conversational systems.

In its current form, MedVista3D is best understood as a 3D CT foundation model whose novelty lies in unifying local-global contrastive learning with semantic normalization of radiology language. Its significance derives less from any single benchmark than from the claim, supported by the reported ablations and transfer results, that clinically motivated error categories can be translated into architectural constraints for 3D vision-language pretraining (Li et al., 4 Sep 2025).