VoxTell: 3D Medical Image Segmentation

• VoxTell is a vision–language model that maps free-text prompts to volumetric masks on CT, MRI, and PET images, enabling open-set segmentation.
• Its architecture employs a multi-scale encoder–decoder with cross-attention between text and image features, achieving Dice scores over 85% for organs.
• Prompt alignment and specificity analysis ensure precise anatomical grounding and robust instance-level generalization without task-specific retraining.

VoxTell is a universal, free-text-promptable vision–LLM (VLM) for three-dimensional (3D) medical image segmentation. It directly maps arbitrary free-text descriptions—ranging from single anatomical names to full clinical sentences—into volumetric masks on CT, MRI, and PET images, enabling open-set, language-driven medical segmentation without task-specific retraining. Trained on over 62,000 volumetric studies spanning more than 1,000 anatomical and pathological concepts, VoxTell achieves robust zero-shot performance, exhibits precise spatial conditioning to linguistic cues, and demonstrates cross-modality and instance-level generalization (Rokuss et al., 14 Nov 2025, Pai et al., 2 May 2026).

1. Model Architecture and Vision–Language Fusion

VoxTell employs an encoder–fusion–decoder paradigm optimized for 3D medical data. The vision branch utilizes a ResEncL-based 3D U-Net encoder with six resolution stages, extracting multi-scale feature maps from the input volume:

$$\mathcal{Z} = \{z_1, \ldots, z_S\}, \quad z_s \in \mathbb{R}^{C_s \times H_s \times W_s \times D_s}, \quad S = 6$$

The text branch features a frozen pretrained model (Qwen3-Embedding-4B) producing a $d$-dimensional prompt embedding. Prompts are presented as clinical sentences, prefixed with a lightweight instruction to focus contextual grounding (“Instruct: Given an anatomical term query, retrieve the precise anatomical entity and location it represents. Query: [text].”).

At the bottleneck, a six-layer transformer executes cross-attention between the text embedding $q$ and the coarsest image feature $z_S$:

$$\mathrm{Attn}(q, z_S) = \mathrm{Softmax}(qW_Q (z_S W_K)^\top) (z_S W_V)$$

where $W_Q$, $W_K$, $W_V$ are learned projections. Projected outputs ($T_s$) provide stage-specific guidance, aligned by channel, for each decoder stage.

The decoder propagates features through sequential upsampling and skip connections, modulating each scale by prompt-derived tensors. Text–spatial fusion is implemented via channelwise dot-products and concatenation:

$$\tilde{y}_s = [T_s \odot z'_s], \quad y_s = \mathrm{ConvBlock}([z'_s;\,\tilde{y}_s])$$

Deep supervision heads at each scale ensure prompt signals permeate all levels of the hierarchy. The segmentation loss at each stage combines Dice and binary cross-entropy (BCE), with scale-wise loss weights following nnU-Net convention, except with left–right mirroring disabled for laterality preservation.

2. Prompt Encoding, Vocabulary Design, and Robustness

VoxTell’s prompt vocabulary encompasses 1,087 base concepts and 9,682 textual variants, harmonized across 158 public datasets (totaling 4 TB). A multi-stage expansion pipeline extends each base label (e.g., “kidney”) to multiple clinically valid synonyms and phrasings; conflicts are resolved using LLMs and expert curation. During training, two positive (present structures) and one negative (absent structures) prompts per volume are sampled. Negative prompts train VoxTell to suppress unrelated segmentations.

Five candidate text encoders from the MTEB medical subset were benchmarked on segmentation accuracy; Qwen3-Embedding-4B achieved an optimal trade-off between performance (peak Dice ≈ 62.6%) and memory (<24 GB), so it was selected for all main experiments. All embeddings are precomputed and frozen at segmentation training time.

Performance is highly robust to prompt synonyms, rephrasings, and typographical variations: stable Dice > 85% for organs across diverse textual inputs, outperforming other promptable models (Rokuss et al., 14 Nov 2025).

3. Prompt Alignment, Specificity, and Conditioning

Comprehensive experiments in the NSCLC context demonstrate that VoxTell’s segmentation behavior is predominantly steered by anatomical tokens in the prompt. Prompt decomposition studies separate each input into six categories: diagnosis, demographic, staging, anatomical, generic, and irrelevant. Attributewise perturbations—systematically swapping one attribute per prompt—reveal that 63.4% of location swaps induce catastrophic segmentation failures (mean $d$0DSC = –0.560, catastrophic defined as $d$1DSC$d$2), whereas histology and staging substitutions have minimal effects (Pai et al., 2 May 2026).

A specificity ladder quantifies prompt informativeness, ranging from L0 (“tumor”) to L6 (over-specified full clinical note). Mean Dice improves monotonically from generic (L0: 0.537) to full (L5: 0.861); diagnosis-only prompts underperform those with explicit anatomical information. Over-specification (L6) saturates the effect—no further accuracy gain observed.

Cross-case prompt swaps (evaluated via $d$3 matched/mismatched matrixes) confirm strong patient-specific conditioning: matched prompt–image pairs achieve DSC = 0.906, mismatched pairs drop to DSC = 0.406 and produce zero segmentation masks in 44% of cases.

4. Zero-Shot and Cross-Modality Segmentation Performance

VoxTell sets a new standard in promptable, zero-shot segmentation by generalizing across organ systems, pathologies, and imaging modalities. Averaged across eleven out-of-distribution datasets (anatomical and pathological; CT, MRI, PET), mean Dice is 70.85%. For comparison, SAT achieves 51.23% and TotalSegmentator approximately 67% (organs only) (Rokuss et al., 14 Nov 2025).

On cross-modality and unknown-concept benchmarks, VoxTell exceeds all baselines: e.g., segmentation of breast (PET), pancreas (MRI), and sarcoma (MRI) yielded 72.3%, 35.7%, and 40.3% Dice, respectively. On unseen concepts such as bladder (MRI) and esophagus (CT), Dice were 25.8% and 69.1%, respectively, surpassing competing models.

On real-world, instance-specific tasks (e.g., “spiculated carcinoma in the left upper lobe with pleural contact,” from SBRT reports), VoxTell attains 50.2% Dice, with competing models at or near zero. On ReXGroundingCT, VoxTell scores 28.2% Dice and 67.8% HIT$d$4, outperforming SAT (13.1% Dice, 49.8% HIT$d$5).

5. Ablation Studies and Fusion Mechanisms

Ablation analysis isolates the contribution of architectural design. Fusion at a single decoder stage yields 51.7–55.1% Dice (Mask2Former/MaskFormer style), whereas three-stage and five-stage fusions increase Dice to 60.2% and 61.5%, respectively. Integrating deep supervision with five-scale fusion increases Dice to 62.6%. Scaling training to batch size 128 on 128 GPUs further raises Dice to 69.4% (Rokuss et al., 14 Nov 2025).

Multi-scale, prompt-modulated fusion at every decoder stage—rather than late-only fusion—is critical for robust open-set and fine-grained instance-level grounding, as confirmed by both quantitative results and observed failure modes in prior architectures.

6. Clinical and Methodological Implications

Prompt-alignment analyses establish that segmentation accuracy alone does not ensure linguistic controllability: prompt dimensions must be explicitly interrogated. Anatomy-centric prompting (e.g., tokens specifying laterality or lobar detail) is essential for guiding spatial attention; abstract descriptors (diagnosis, stage) do not localize masks effectively. This spatial prioritization has direct implications for radiotherapy contouring and clinical deployment, where “where to look” outweighs “what to look for.”

Irrelevant prompts reliably produce empty masks, functioning as a fail-safe against hallucination of non-existent findings. This property—learned via negative prompts—is essential for robust deployment in open-set, language-driven clinical workflows.

Prompt perturbation, specificity ladder, and swap matrix analyses are advocated as critical evaluation tools for promptable segmentation models, revealing practical controllability and interpretable failure modes that are obscured by averaging metrics such as overall Dice.

7. Implementation and Reproducibility

VoxTell is implemented for large-scale distributed training (64 × NVIDIA A100 40 GB GPUs, batch size 128), with public code, processed vocabularies, and training scripts available at https://github.com/MIC-DKFZ/VoxTell (Rokuss et al., 14 Nov 2025). The repository provides out-of-the-box recipes for dataset preparation, training, and evaluation—including extension to new modalities and concepts. Standard nnU-Net augmentations are employed, except for left–right flips. The optimization regime employs SGD with polynomial decay for 2,000 epochs. All computations, ablations, and prompt studies are directly reproducible from the released materials.

In summary, VoxTell constitutes a state-of-the-art vision–language framework for free-text-prompted 3D medical image segmentation, achieving robust zero-shot and cross-modality performance, precise anatomical grounding, and reliable language-driven control across diverse clinical contexts (Rokuss et al., 14 Nov 2025, Pai et al., 2 May 2026).