TextSAM-EUS: Text-Driven Pancreatic Tumor Segmentation

• TextSAM-EUS is a lightweight, text-guided variant of SAM that automates pancreatic tumor segmentation in ultrasound by combining a BiomedCLIP text encoder with LoRA-based adaptation.
• It employs text prompt learning through context optimization and an iterative refinement process to replace manual geometric cues with automatic spatial guidance.
• Evaluated on a public EUS pancreatic database, the method achieves 82-83% Dice and 85-86% NSD, outperforming traditional supervised models and prior SAM-based approaches.

TextSAM-EUS is a lightweight, text-driven adaptation of the Segment Anything Model (SAM) for automatic pancreatic tumor segmentation in endoscopic ultrasound (EUS). It combines a BiomedCLIP text encoder, text prompt learning through context optimization, and LoRA-based adaptation of SAM so that segmentation can be performed without manual geometric prompts at inference, while tuning only 0.86% of the total parameters. On the public Endoscopic Ultrasound Database of the Pancreas, it reports 82.69% Dice and 85.28% normalized surface distance (NSD) with automatic prompts, and 83.10% Dice and 85.70% NSD with manual geometric prompts, exceeding both prior supervised deep-learning baselines and several SAM-based or foundation-model comparators (Spiegler et al., 24 Jul 2025).

1. Clinical setting and problem formulation

TextSAM-EUS is motivated by pancreatic cancer, which the paper describes as the sixth leading cause of cancer-related death with about 10% survival. In EUS-guided workflows, pancreatic tumor segmentation is relevant to fine-needle aspiration or biopsy, targeted radiotherapy or seed placement, and procedural planning. The method is designed specifically for EUS, where segmentation is difficult because images exhibit strong speckle noise, low contrast, blurry or unintuitive tumor boundaries, and high variability in appearance (Spiegler et al., 24 Jul 2025).

The paper positions this difficulty against two limitations in earlier approaches. First, fully supervised architectures such as nnU-Net, SwinUNet, and older U-Net variants require large pixel-wise annotated datasets, whereas expert annotations in pancreatic EUS are limited. Second, vanilla SAM is pretrained on natural images rather than ultrasound and ordinarily depends on manual geometric prompts such as points, boxes, or masks. TextSAM-EUS addresses both issues by using biomedical text as a semantic prior and by restricting trainable components to a small parameter budget (Spiegler et al., 24 Jul 2025).

A central premise is the distinction between where to segment and what to segment. The paper argues that geometric prompts specify location, whereas text prompts encode the target concept. For pancreatic tumor EUS, that semantic signal is intended to disambiguate structures whose visual boundaries are weak or irregular. This suggests that TextSAM-EUS is not only a prompt-substitution mechanism, but also a task-specific semantic conditioning strategy for a domain-shifted foundation model.

2. Architectural composition

The framework has three principal components and an iterative refinement stage. The text branch uses the BiomedCLIP text encoder with learnable context tokens. The image branch uses SAM’s ViT image encoder, adapted with LoRA modules. The mask decoder combines image features with prompt embeddings, where prompt embeddings originate from the text branch and may later be augmented with geometric prompts during refinement (Spiegler et al., 24 Jul 2025).

The paper restates the standard SAM formulation. For an input image $\bm{X} \in \mathbb{R}^{3 \times H \times W}$, the image encoder produces

$$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$

The prompt encoder maps prompts to

$$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$

and the mask decoder outputs

$$\hat{\bm{Y}} = \mathcal{D}_{\text{mask}}(\bm{F}, \bm{P}) \in \mathbb{R}^{H \times W}.$$

TextSAM-EUS modifies this pipeline by replacing purely geometric prompting with a learned biomedical text representation and by adapting the image encoder and mask decoder with low-rank updates. The resulting architecture can operate in two modes: a fully automatic mode that begins from text alone, and a manual-prompt mode that accepts one point or a perturbed bounding box for comparison. The automatic pathway is the principal contribution because it removes the need for manual geometric prompting at inference (Spiegler et al., 24 Jul 2025).

3. Text prompt learning with BiomedCLIP

The text component uses the BiomedCLIP text encoder rather than a generic language encoder because BiomedCLIP is pretrained on biomedical image-text pairs and is described as better aligned with clinical terminology. In the reported task, the effective class name is “tumor,” and the text encoder maps tokenized prompts to

$$\mathbf{T} = \bm{E}_t(\bm{X}_t) \in \mathbb{R}^{C \times D},$$

where $C$ is the number of text prompts or classes and $D$ is the text embedding dimension (Spiegler et al., 24 Jul 2025).

The distinctive mechanism is text prompt learning through context optimization. Instead of fixing a phrase such as “tumor” or “pancreatic tumor in ultrasound scan,” the method learns continuous prompt tokens

$\{P^i \in \mathbb{R}^{d_t}\}_{i=1}^{b},$

which are concatenated with fixed tokens:

$$[P_0, W_0] = [P^1, P^2, \cdots, P^b, w^1, w^2, \cdots, w^S].$$

Here $d_t$ is the BiomedCLIP text hidden dimension, $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$0 is the number of fixed tokens, $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$1 is the number of learnable context tokens, and $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$2 matches the fixed context length of the text encoder. The prompt sequence is propagated through the BiomedCLIP transformer layers, with prompt depth later examined in ablation; for BiomedCLIP, the paper states $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$3 layers (Spiegler et al., 24 Jul 2025).

The final token, usually the [EOS] token, is projected to obtain the text embedding,

$$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$4

That embedding is then mapped into SAM’s prompt-token space by a small adapter:

$$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$5

with $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$6, $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$7, $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$8, $$\bm{F} = \bm{E}_{\text{img}}(\bm{X}) \in \mathbb{R}^{h \times w \times d}.$$9, and $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$0. The text-derived prompt tokens are concatenated with dense mask embeddings,

$$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$1

and after refinement geometric prompts are included as

$$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$2

This design makes the text branch more than a label encoder. The learned context tokens alter how the biomedical text encoder interprets the prompt, and the projection module explicitly aligns the resulting semantics with SAM’s prompt space. A plausible implication is that the method attempts to bridge not only domain shift in image appearance, but also representational mismatch between biomedical language embeddings and SAM’s segmentation decoder.

4. LoRA adaptation and iterative refinement

To adapt SAM to the EUS domain, TextSAM-EUS applies LoRA to the SAM image encoder and the SAM mask decoder. The low-rank update is expressed as

$$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$3

where $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$4, $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$5, $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$6, and $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$7. The original SAM weights remain frozen, and the main model uses rank $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$8 (Spiegler et al., 24 Jul 2025).

The trainable components are the learnable text context tokens, the text projection or adapter, and the LoRA modules on the image encoder and mask decoder. Everything else is frozen. The paper reports 1.69M trainable parameters, corresponding to 0.86% of SAM’s total parameters. This is the basis for the method’s characterization as parameter-efficient (Spiegler et al., 24 Jul 2025).

Inference proceeds in two stages in the automatic setting. First, the EUS image and the learned text prompt “tumor” are used to obtain an initial mask. Second, a bounding box and centroid are computed from that mask and fed back as geometric cues for one additional segmentation pass. The paper states that automatically derived geometric prompts from the first prediction help slightly, with a language-only initial mask at 82.00% DSC and the best performance obtained by adding centroid and box prompts; among tested auto prompts, centroid-based refinement is reported as best (Spiegler et al., 24 Jul 2025).

This two-stage procedure clarifies a potential misconception. TextSAM-EUS does not eliminate geometry altogether; rather, it removes the need for manual geometric prompting. Geometry is reintroduced automatically after semantic initialization. The method therefore combines semantic guidance and self-generated spatial refinement instead of treating them as mutually exclusive alternatives.

5. Evaluation protocol and reported results

The evaluation uses the public Endoscopic Ultrasound Database of the Pancreas, consisting of 18 tumor patients, grayscale EUS frames with resolution 711 × 457, and expert pixel-wise tumor annotations. The patient-wise split is 11,363 images from 12 patients for training, 986 images from 2 patients for validation, and 4,185 images from 4 held-out patients for testing. The paper emphasizes the patient-wise split as a safeguard against leakage (Spiegler et al., 24 Jul 2025).

Training is reported for 5 epochs with learning rate 0.001, batch size 1, AdamW, weight decay 0.01, cosine annealing learning-rate schedule, and model selection by lowest validation loss, using a single NVIDIA A100 40GB. The text prompt setup uses the class name “tumor,” 4 context tokens, prompt depth 12, and LoRA rank 16. Evaluation uses Dice Similarity Coefficient and Normalized Surface Distance at a 3-pixel tolerance, with mean and standard deviation scaled to 0–100%, and significance testing via two-tailed paired-sample t-tests with threshold $$\bm{P} = \bm{E}_{\text{prompt}}(\text{prompt}) \in \mathbb{R}^{k \times d},$$9 (Spiegler et al., 24 Jul 2025).

Setting	DSC	NSD
Automatic prompts	82.69%	85.28%
Manual geometric prompts	83.10%	85.70%

The automatic-prompt result is the paper’s main practical claim, because it exceeds prior methods that do not require manual prompts. The manual-prompt setting serves as an upper-bound style comparison against prompt-based SAM baselines. Specific comparisons reported in the paper include MedSAM at 82.66 DSC / 85.75 NSD, SAMUS fine-tuned at 82.81 DSC / 85.50 NSD, SAM (Box) at 78.15 DSC / 83.70 NSD, and SAM (Point) at 39.80 DSC / 41.54 NSD (Spiegler et al., 24 Jul 2025).

The ablation studies are used to localize the contribution of individual design choices. For LoRA rank, $$\hat{\bm{Y}} = \mathcal{D}_{\text{mask}}(\bm{F}, \bm{P}) \in \mathbb{R}^{H \times W}.$$0 yields 44.00 DSC, $$\hat{\bm{Y}} = \mathcal{D}_{\text{mask}}(\bm{F}, \bm{P}) \in \mathbb{R}^{H \times W}.$$1 yields 81.33 DSC, and $$\hat{\bm{Y}} = \mathcal{D}_{\text{mask}}(\bm{F}, \bm{P}) \in \mathbb{R}^{H \times W}.$$2 yields the best 82.69 DSC. For context-token count, 4 tokens are best at 82.69 DSC, while 8 and 12 tokens are worse or slightly worse. For prompt injection depth, deeper injection performs better, with the best result at all 12 BiomedCLIP layers. These results support the paper’s interpretation that moderate low-rank adaptation is sufficient for EUS domain shift and that a compact learned prompt is preferable to a longer one (Spiegler et al., 24 Jul 2025).

6. Position within text-guided SAM research

TextSAM-EUS is presented as the first attempt to incorporate prompt learning in SAM-based medical image segmentation (Spiegler et al., 24 Jul 2025). Within the broader space of text-guided SAM variants, its defining feature is that biomedical text embeddings are learned through context optimization and then mapped into SAM’s prompt-token space, rather than being used only as external metadata or as an indirect route to point or box generation.

This distinguishes it from CC-SAM, a SAM-based ultrasound segmentation framework that uses GPT-4-generated text descriptions, Med-BERT, and Grounding DINO so that text is converted into a bounding-box prompt for SAM’s prompt encoder. CC-SAM reports that text prompts outperform point-based prompting in ultrasound segmentation, but its text prompting is explicitly described as text-to-box prompting rather than direct native text conditioning inside SAM (Gowda et al., 2024). It also distinguishes TextSAM-EUS from SAM-PTx, which injects frozen CLIP-derived class-name embeddings into SAM’s image encoder through a Parallel-Text adapter, with text conditioning applied only to the MLP-parallel branch while preserving the attention pathway for spatial reasoning (Jalilian et al., 31 Jul 2025).

These comparisons clarify the method’s niche. Relative to standard SAM, TextSAM-EUS replaces manual geometric interaction with biomedical semantic prompting and low-rank domain adaptation. Relative to other medical SAM variants such as MedSAM, AutoSAM, AutoSAMUS, SAMed, and SPFS-SAM, it adds an explicit learned text prior. Relative to CC-SAM and SAM-PTx, it occupies a distinct design point: text prompt learning with BiomedCLIP, prompt-space alignment for SAM, and automatic geometric refinement specialized for pancreatic tumor EUS (Spiegler et al., 24 Jul 2025).

The paper also states several limitations: evaluation on a single dataset, only 18 patients, and binary tumor segmentation rather than multi-class segmentation. Future work is described in terms of multi-class segmentation, broader biomedical applications, and further prompt-design improvements (Spiegler et al., 24 Jul 2025). This suggests that the current contribution is best understood as a domain-specific proof of principle for semantically conditioned, parameter-efficient SAM adaptation in a difficult ultrasound setting, rather than a general solution for all medical segmentation tasks.