Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation

Abstract: Zero-shot vision-LLMs (VLMs) offer a promptable alternative to task-specific training for gross tumor volume (GTV) delineation in non-small-cell lung cancer (NSCLC), but the prompt dimensions that govern their spatial behavior remain poorly understood. We study this question by probing alignment directions in VoxTell on a held-out internal NSCLC tumor dataset through sub-prompt decomposition into diagnosis, demographic, staging, anatomical, generic, and irrelevant controls; attribute-wise perturbation robustness; specificity ladders; and cross-case prompt swaps, while benchmarking against fine-tuned and zero-shot baselines using the Dice Similarity Coefficient (DSC) with Wilcoxon signed-rank tests and Benjamini-Hochberg correction. Alignment analyses revealed that anatomical location is the dominant driver of VoxTell's spatial attention: 63.4 percent of location perturbations caused catastrophic drops, prompt specificity improved from generic to full descriptions except for diagnosis-only prompts, irrelevant prompts correctly yielded zero segmentation, and cross-case prompt swaps confirmed patient-specific conditioning (matched DSC 0.906 vs. mismatched 0.406). Histology and stage substitutions had minimal effect, indicating that the model prioritizes "where to look" over "what to look for." In this context, VoxTell, operating fully zero-shot, achieved a mean DSC of 0.613, statistically indistinguishable from nnUNet (0.690, adjusted p = 0.156) and Ahmed et al. (0.675, adjusted p = 0.679), while significantly outperforming all other zero-shot models. Together, these findings argue that segmentation VLMs should be evaluated not only by Dice, but also by the prompt dimensions to which they align.

• The paper demonstrates that anatomical prompt specificity is key for robust NSCLC tumor segmentation with zero-shot VLMs like VoxTell, achieving performance comparable to fine-tuned models.
• The study employs sub-prompt decomposition, perturbation analysis, and cross-case prompt swaps to assess segmentation accuracy using the Dice Similarity Coefficient.
• The analysis reveals that non-spatial details minimally affect segmentation, emphasizing the importance of patient-specific anatomical cues in clinical prompt design.

Alignment of Clinical Prompts in Zero-Shot Segmentation VLMs for NSCLC Tumor Delineation

Introduction

The study "Exploring Prompt Alignment with Clinical Factors in Zero-Shot Segmentation VLMs for NSCLC Tumor Segmentation" (2605.01266) systematically analyzes the role of clinical prompt attributes in conditioning volumetric segmentations produced by zero-shot vision-LLMs (VLMs) for non-small-cell lung cancer (NSCLC) gross tumor volume (GTV) delineation. Previous work has shown VLMs as a promising alternative to supervised deep learning for medical segmentation; however, the specific clinical prompt attributes that modulate their spatial output have remained uncharacterized. This work interrogates the alignment mechanisms of VoxTell—a state-of-the-art zero-shot VLM—using fine-grained prompt decomposition, perturbation analysis, prompt specificity ramps, and cross-patient prompt swaps, all benchmarked against a suite of supervised and zero-shot baselines on an independently curated clinical dataset.

Methodology and Experimental Design

A held-out dataset of 93 internal NSCLC cases (HarvardRT) with expert-segmented planning CTs and granular free-text clinical notes provided a robust testbed. Prompts were decomposed to isolate diagnosis, demographic, TNM/stage, anatomical, generic, and irrelevant control phrases. The alignment of VoxTell to these factors was probed across four primary experimental axes:

• Sub-prompt decomposition: Segmentation masks produced to individual or composite prompt fragments.
• Perturbation robustness: Controlled swaps of prompt attributes (histology, stage, demographics, anatomy, irrelevant/negative controls), with performance quantified by $\Delta$DSC.
• Prompt specificity ladder: Progressive additions of spatial and clinical detail to prompts (from generic to fully elaborate).
• Cross-case prompt swaps: Prompt-image pairs deliberately mismatched to assess if VLM segmentations reflect patient-specific or generic tumor detection.

All results were situated in a broader model benchmark: three fine-tuned models (e.g., nnUNet), two zero-shot vision-only models, and four zero-shot VLMs (including VoxTell). Dice Similarity Coefficient (DSC) and significance testing (Wilcoxon signed-rank/BH correction) were used throughout.

Anatomical Dominance in Model Alignment

Fragment and perturbation analyses revealed that anatomical location vocabularies are the overwhelming driver of VoxTell’s spatial output. Swapping anatomical location in prompts was catastrophically destabilizing, with 63.4% of such perturbations producing $|\Delta\text{DSC}| > 0.5$ and a mean decrement of $-0.560$ in DSC (see below).

Figure 1: $\Delta$DSC distribution by perturbation showing catastrophic degradation for location swaps, while tumor type and stage swaps are largely benign.

In contrast, swaps of histology, stage, or demographic details had statistically negligible effect. Diagnosis-only prompts generated inconsistent results (success for common labels, failures for rare), confirming that anatomical grounding, rather than histological specificity, is essential for robust mask generation. Irrelevant prompts (e.g., "liver cyst") were correctly suppressed, supporting prompt-conditional alignment as opposed to indiscriminate segmentation.

Prompt Specificity and Granularity Effects

VoxTell segmentation improved nearly monotonically as prompts progressed from generic descriptors to fully-specified clinical language, with the largest gains realized at the organ-to-lobe specificity transition.

Figure 2: Prompt specificity ladder shows DSC improvement from generic ("tumor") to precise clinical prompts, except for diagnosis-only prompts at L4.

Over-specification—adding fabricated detail—provided no further benefit and sometimes slightly confused the output, saturating the model’s alignment mechanism. This staircase pattern was not observed in diagnosis-and-staging only prompts devoid of explicit anatomical anchors (L4), which underperformed compared to naturalistic, spatially grounded prompts.

Patient-Specific Conditioning and Mismatched Prompts

Cross-case prompt swaps further established that VoxTell performs genuinely patient-conditional segmentation, rather than simply acting as a general tumor detector. Matched prompt-image pairs displayed a mean DSC of 0.906, in stark contrast to mismatched pairs at 0.406, with 44% of mismatched prompts suppressed to zero output.

Figure 3: Cross-case prompt swap $5\times5$ matrix—the diagonal (matched) entries outperform all off-diagonals, confirming robust patient specificity.

Generic lung tumor prompts yielded intermediate performance, lagging behind full clinical prompt matches. This specificity substantiates the claim that prompt content, especially anatomical detail, acts as an active spatial filter in VLM inference.

Benchmark Performance: VoxTell versus SOTA

In benchmarking across nine models, VoxTell achieved a mean DSC of 0.613, statistically indistinguishable from the fine-tuned nnUNet ($p_\text{adj}=0.156$) and Ahmed et al. ($p_\text{adj}=0.679$) models (respective means: 0.690, 0.675). VoxTell robustly outperformed all other zero-shot VLMs (BiomedParse, SAT, CAT: $<$0.26) and zero-shot vision-only models except TotalSeg-Nodules.

Figure 4: Per-case DSC distributions; VoxTell (orange) demonstrates comparable central tendency to leading fine-tuned models (blue), but with higher variance.

Figure 5: Pairwise DSC differences with respect to VoxTell, positive indicates VoxTell outperformance; significance assessed after BH correction.

These findings argue that strong prompt-aligned conditioning, as opposed to prompt invariance, is compatible with state-of-the-art segmentation efficacy in a fully zero-shot clinical scenario.

Theoretical and Practical Implications

The results show that in clinically needful settings (e.g., radiotherapy planning), VLMs such as VoxTell tend to parse and bind spatial prompt cues to volumetric image features, with little integration of non-spatial, text-derived diagnostic factors. This selective alignment is aligned with architectures employing cross-attention to localize spatially relevant regions, while deprioritizing non-anatomical descriptors. The practical upshot is that clinical prompt design should emphasize anatomical precision over histological or abstract detail for optimal VLM segmentation.

Methodologically, this work demonstrates the necessity of prompt-alignment and robustness analyses in the evaluation of text-conditioned segmentation models and establishes a template for similar studies in other clinical and imaging contexts. The study also underscores the importance of comprehensive prompt engineering and validation for safe model deployment.

Future research should extend alignment probing to multi-institutional datasets, incorporate dose-volume and treatment intent endpoints, and explore automated prompt optimization strategies. Investigating how cross-modal text-image attention mechanisms can be enhanced to integrate diagnostic or treatment context—beyond spatial cues—remains an open research frontier.

Conclusion

Anatomical location is the dominant conditioning factor for text-prompted, zero-shot VLM segmentation of NSCLC, with segmentation masks acutely sensitive to spatial detail in clinical prompts and largely indifferent to histology or demographic context. VoxTell matches fine-tuned models in segmentation accuracy when supplied with elaborated spatial prompts, confirming that high-quality, patient-specific mask generation is achievable without task-specific retraining. Thorough alignment and robustness assessments are critical complements to standard accuracy benchmarks, informing both theoretical understanding and the safe clinical translation of promptable VLMs. The synergy of clinical language and volumetric image analysis in segmentation VLMs represents a scalable avenue for automating radiotherapy workflows, contingent upon thorough prospective validation across diverse populations and targets.