VELVET-Med: 3D Medical VLP Framework

• VELVET-Med is a vision-language pretraining framework for volumetric medical imaging, combining 3D CT scans with radiology reports through hierarchical contrastive learning.
• It employs uni-modal self-supervision and a specialized TriBERT encoder to capture report-, sentence-, and word-level semantics for improved multi-scale alignment.
• The framework enhances downstream tasks such as segmentation, retrieval, visual question answering, and report generation, even with limited paired data.

Searching arXiv for VELVET-Med and closely related medical VLM benchmarking papers. VELVET-Med is a vision-language pretraining framework for volumetric medical imaging, introduced for 3D CT scans paired with radiology reports under conditions where paired data are limited rather than web-scale. It is formulated around three technical choices: uni-modal self-supervision for both vision and language, a report encoder called TriBERT that models report-, sentence-, and word-level semantics, and hierarchical contrastive learning that aligns visual and textual representations at multiple semantic granularities. Using 38,875 CT-report pairs derived from a filtered version of M3D-CAP, the framework targets transfer across 3D segmentation, cross-modal retrieval, visual question answering, and report generation (Zhang et al., 16 Aug 2025).

1. Problem setting and design rationale

VELVET-Med addresses vision-language pretraining for volumetric medical data, specifically 3D CT scans and free-text radiology reports. The motivating difficulty is that paired 3D scan-report datasets are scarce, while CT volumes are information-dense and radiology reports are long and structured. The framework is therefore positioned against the assumption that improvements in medical VLP for volumetric data must come primarily from larger datasets or larger models. Instead, the central claim is that limited 3D paired data can be better exploited through more effective pretraining objectives and model design (Zhang et al., 16 Aug 2025).

The architecture reflects two hierarchical observations. First, a 3D Swin Transformer is hierarchical by design and yields feature maps at multiple stages and resolutions. Second, radiology reports are naturally organized at report, sentence, and word levels, whereas standard BERT-style encoding treats the report largely as a flat token sequence. VELVET-Med therefore aligns coarse visual structure with global report semantics, intermediate visual context with sentence-level semantics, and lower-level visual context with word-level semantics. This design is explicitly intended to move beyond single global CLIP-style alignment, which the authors argue may miss localized findings and intermediate clinical concepts in CT-report pairs (Zhang et al., 16 Aug 2025).

The framework adopts a two-stage coupling of modalities. It first aligns image and text through cross-modal contrastive objectives at multiple levels, and then fuses them through a multimodal encoder for image-text matching, multimodal masked language modeling, and downstream generative use. The paper characterizes this as a “fuse after alignment” design adapted to volumetric medical data (Zhang et al., 16 Aug 2025).

2. Architecture and representational hierarchy

VELVET-Med contains three encoders: a vision encoder $\theta_v$, a language encoder $\theta_t$, and a multimodal encoder $\theta_{mm}$. The vision encoder is a 3D Swin Transformer, instantiated as SwinVIT-T, SwinVIT-S, or SwinVIT-B; the language encoder is the proposed TriBERT; and the multimodal encoder is a stack of transformer decoder layers placed on top of the language encoder output (Zhang et al., 16 Aug 2025).

For visual input handling, the model uses both a global/downsampled volume $E_v^{ds}$ and a sub-volume crop $E_v^{sub}$, each shaped as $\mathbb{R}^{1 \times 96 \times 96 \times 96}$. The global volume is used for cross-modal and multimodal objectives, while the sub-volume is used for vision self-supervision. The vision encoder yields the last three feature maps,

$$f_v^k \in \mathbb{R}^{N \times c_v^k \times h^k \times w^k \times d^k}, \quad k \in \{top, mid, bot\},$$

and the top-level map is spatially average pooled to obtain a global feature $f_v^{top} \in \mathbb{R}^{N \times c_v^{top}}$ (Zhang et al., 16 Aug 2025).

TriBERT is the principal language-side innovation. It is designed to encode multi-level textual semantics through three representational levels: the full-report meaning via the $[\mathrm{CLS}]$ token, sentence-level meaning via learned $[\mathrm{SENT}_i]$ tokens, and word-level meaning via token embeddings aggregated back to words when WordPiece splits occur. Its input construction splits reports by punctuation and line breaks, removes empty lines, discards sentences with fewer than 2 words, tokenizes with WordPiece, prepends $\theta_t$0 to each sentence, prepends $\theta_t$1 to the entire report, and adds token embeddings, sentence-type embeddings, and positional embeddings (Zhang et al., 16 Aug 2025).

A second modification is TriBERT’s tri-level self-attention mask. Report-level and word tokens can attend to the full context, whereas each sentence token $\theta_t$2 can attend only to words in the same sentence and to the global $\theta_t$3 token. This is intended to preserve sentence-level independence while retaining connection to a global report summary. Appendix configurations report TriBERT-S with 70M parameters, feature dimension 768, and 6 layers, and TriBERT-B with 109M parameters, feature dimension 768, and 12 layers; the best-performing setup uses TriBERT-B (Zhang et al., 16 Aug 2025).

All visual and textual representations are linearly projected into a shared 512-dimensional embedding space. This shared space supports the framework’s hierarchical contrastive objectives across report, sentence, and word levels (Zhang et al., 16 Aug 2025).

3. Pretraining objectives and mathematical formulation

VELVET-Med trains with four groups of objectives:

1. uni-modal vision self-supervision,
2. uni-modal language self-supervision,
3. cross-modal hierarchical contrastive learning,
4. multimodal fusion objectives (Zhang et al., 16 Aug 2025).

The vision-side self-supervision is defined as

$\theta_t$4

These terms correspond respectively to masked volume inpainting, 3D rotation prediction, and contrastive coding. The inpainting component uses a 30% sub-volume drop rate and a shallow upsampling stack to recover dropped content. Rotation prediction is a 4-class task over $\theta_t$5, $\theta_t$6, $\theta_t$7, and $\theta_t$8. Contrastive coding uses a 512-dimensional embedding and InfoNCE loss. The stated rationale is that cross-modal learning alone may overemphasize low-frequency global semantics, whereas these visual SSL tasks preserve fine-grained spatial and anatomical detail (Zhang et al., 16 Aug 2025).

The language-side self-supervision is

$\theta_t$9

namely masked language modeling on text alone with a 15% masking ratio. This is used to strengthen the text encoder’s intra-text semantic modeling during multimodal pretraining (Zhang et al., 16 Aug 2025).

The cross-modal objective is the most distinctive component:

$\theta_{mm}$0

At the top level, a pooled global visual representation is aligned with the report embedding in a CLIP-style contrastive objective. At the middle level, sentence embeddings are contextualized by cross-attention with mid-level visual features and then matched to the original sentence embeddings through a bidirectional contrastive loss. At the bottom level, word embeddings are contextualized by cross-attention with lower-level visual features and contrasted against their original word embeddings. The paper frames this as alignment of global anatomy/report meaning, sentence-level findings/statements, and word-level medical concepts (Zhang et al., 16 Aug 2025).

The multimodal fusion objective is

$\theta_{mm}$1

The matching term performs binary classification of whether a scan and report match, using hard negatives mined within the minibatch and the fused $\theta_{mm}$2 representation. The multimodal MLM term masks word tokens and predicts them from fused visual-text context. Both uni-modal and multimodal MLM use a 15% masking ratio (Zhang et al., 16 Aug 2025).

The total pretraining loss is

$\theta_{mm}$3

This additive design is central to the paper’s claim that data efficiency in volumetric medical VLP can be improved by combining strong uni-modal inductive support with coarse-to-fine cross-modal alignment and post-alignment fusion (Zhang et al., 16 Aug 2025).

4. Dataset curation, preprocessing, and implementation

Pretraining uses M3D-CAP, whose original release contains 120,092 scan-report pairs. The authors report two data-quality problems in this source: disordered slice stacks along the $\theta_{mm}$4-axis and scans with too few slices to resample reliably. They therefore construct two cleaned versions. M3D-CAP-clean removes disordered scans and retains 69,686 pairs. M3D-CAP-filtered further removes scans with fewer than 48 slices, producing the final pretraining set of 38,875 pairs used by VELVET-Med (Zhang et al., 16 Aug 2025).

The preprocessing logic is directly tied to 3D anatomical consistency. If the number of slices is greater than 96, the method uniformly samples 96 slices along $\theta_{mm}$5 to form the full volume $\theta_{mm}$6. If the number of slices is between 48 and 96, the scan is upsampled along $\theta_{mm}$7 by a factor of 2 to obtain $\theta_{mm}$8. The $\theta_{mm}$9 and $E_v^{ds}$0 dimensions are resized with 2D interpolation, and the resulting global/downsampled volume and sub-volume both have shape $E_v^{ds}$1 (Zhang et al., 16 Aug 2025).

The report side is also characterized quantitatively. On M3D-CAP-filtered, the reported statistics are: max words per report 687, min words 4, mean words 109.9, mean sentences 10.5, and max sentences 61. Training limits are maximum 50 sentences, maximum 200 words per sentence, and maximum 512 tokens per report. These values support the paper’s motivation for explicit sentence-level modeling in TriBERT (Zhang et al., 16 Aug 2025).

The best configuration from the ablations is TriBERT-B with SwinVIT-S. SwinVIT-S has 22M parameters, embed dimension 48, depth $E_v^{ds}$2, and number of heads $E_v^{ds}$3. Pretraining uses 50 epochs, AdamW, cosine annealing, learning rate $E_v^{ds}$4, $E_v^{ds}$5, $E_v^{ds}$6, weight decay $E_v^{ds}$7, batch size 10 per GPU, and 4 NVIDIA H100 GPUs. The reported training time is about 50 hours per run, with the best checkpoint selected by validation loss. The implementation uses Accelerate with mixed precision and data parallelism (Zhang et al., 16 Aug 2025).

5. Downstream transfer and empirical performance

VELVET-Med is evaluated on 3D semantic segmentation, cross-modal retrieval, visual question answering, and report generation. Across these tasks, the paper’s stated conclusion is that hierarchical contrastive learning improves over top-level-only CLIP-style alignment, uni-modal self-supervision gives additional gains, and multimodal fusion helps some tasks while slightly hurting pure retrieval (Zhang et al., 16 Aug 2025).

For cross-modal retrieval, evaluation is performed on 2,000 high-quality test samples using Recall@$E_v^{ds}$8, Recall@$E_v^{ds}$9, and Recall@$E_v^{sub}$0 for both scan-to-report retrieval (SRR) and report-to-scan retrieval (RSR). The reported baselines are PMC-CLIP, M3D retrieval, and CLIP-3D. CLIP-3D on the filtered data reaches SRR $E_v^{sub}$1 and RSR $E_v^{sub}$2. Top-level contrast alone, $E_v^{sub}$3, improves this to SRR $E_v^{sub}$4 and RSR $E_v^{sub}$5. Full hierarchical contrastive learning, $E_v^{sub}$6, reaches SRR $E_v^{sub}$7 and RSR $E_v^{sub}$8. The full VELVET-Med entry is reported as SRR $E_v^{sub}$9 and RSR $\mathbb{R}^{1 \times 96 \times 96 \times 96}$0; the table is noted to contain a typo repeating $\mathbb{R}^{1 \times 96 \times 96 \times 96}$1, but the intended row is the full model (Zhang et al., 16 Aug 2025).

For 3D segmentation, the vision encoder is transferred into SwinUNETR and evaluated on AbdomenCT-1K and CT-ORG using Dice, NSD, and HD95. On AbdomenCT-1K, scratch SwinUNETR-S gives Dice 93.91, NSD 95.93, and HD95 3.07, whereas full VELVET-Med gives Dice 94.03, NSD 96.20, and HD95 2.39. On CT-ORG, scratch yields Dice 83.80, NSD 72.29, and HD95 61.31, while full VELVET-Med reaches Dice 88.30, NSD 82.09, and HD95 37.97. This is the strongest result in the CT-ORG table (Zhang et al., 16 Aug 2025).

For report generation, the baseline M3D-LaMed reimplementation obtains BLEU 17.32, ROUGE 11.73, and BERTScore 84.09. VELVET-Med reaches BLEU 23.54, ROUGE 13.28, and BERTScore 85.16. For generative VQA on M3D-VQA, the baseline gives BLEU 20.63, ROUGE 23.47, and BERTScore 87.40, whereas VELVET-Med gives BLEU 22.93, ROUGE 34.75, and BERTScore 88.96. For classification-based VQA on the yes/no subset, VELVET-Med reports AUC 84.27 and accuracy 76.17 (Zhang et al., 16 Aug 2025).

The ablations are central to the paper’s argument. Adding mid-level and bottom-level alignment to top-level contrast improves SRR Recall@$\mathbb{R}^{1 \times 96 \times 96 \times 96}$2 from 32.77 to 35.96. Adding uni-modal language or vision objectives to $\mathbb{R}^{1 \times 96 \times 96 \times 96}$3 yields moderate additional gains for retrieval. The best model-size tradeoff is TriBERT-B plus SwinVIT-S, which outperforms larger or more classical pairings such as BERT-B plus ViT-B or TriBERT-B plus ViT-B. The paper explicitly interprets this as evidence that larger backbones are not always better in data-scarce 3D medical VLP (Zhang et al., 16 Aug 2025).

6. Significance, limitations, and adjacent evaluation questions

The main significance attributed to VELVET-Med is methodological rather than purely scale-driven. The paper argues that better objectives and architecture can compensate for limited 3D paired data; that TriBERT improves language modeling for complex medical reports; that hierarchical contrastive learning improves vision-language correspondence; that uni-modal self-supervision should not be neglected in medical VLP; and that high-quality filtered CT-report data can outperform larger noisy datasets (Zhang et al., 16 Aug 2025).

Several caveats are explicit. The training scale remains limited at about 38k paired scan-report samples; the framework is trained only on CT; the appendix notes a remaining gap to expert-level performance, especially for fine-grained clinical tasks and rare pathologies; and potential domain bias, fairness risks, privacy issues from volumetric anatomy, over-reliance in clinical settings, unclear liability, and compute or energy cost are all mentioned as broader concerns (Zhang et al., 16 Aug 2025). There is also a task-specific caveat in the results: multimodal fusion objectives can hurt retrieval even when they help VQA, report generation, and CT-ORG segmentation, so not every additional loss is uniformly beneficial (Zhang et al., 16 Aug 2025).

An adjacent but distinct line of work concerns trustworthy medical VLM evaluation under broken evidence rather than representation learning alone. MedVIGIL evaluates whether a medical vision-LLM recognizes when the evidential basis for answering has failed, using false-premise traps, wording perturbations, knowledge-only rewrites, ROI-masked images, ROI-only images, and risk-weighted silent-failure metrics (Jiang et al., 8 May 2026). A plausible implication is that VELVET-Med’s gains in generative VQA and report generation do not by themselves establish evidence-conditional safe failure, refusal behavior, or ROI-conditioned grounding, because those properties are outside the reported evaluation scope of VELVET-Med (Zhang et al., 16 Aug 2025, Jiang et al., 8 May 2026).

Within volumetric medical VLP, the distinctive contribution of VELVET-Med is therefore the combination of 3D Swin-based volumetric encoding, a structured report encoder with explicit sentence tokens and sentence-type embeddings, hierarchical contrastive alignment across report, sentence, and word levels, and uni-modal self-supervision integrated into a single end-to-end pretraining framework. The paper’s broad empirical lesson is that in low-resource 3D medical multimodal learning, careful hierarchical alignment, structured text modeling, data curation, and selective fusion can matter at least as much as raw model scale or raw dataset size (Zhang et al., 16 Aug 2025).