SAM-PTx: Text-Guided SAM Adaptation

• SAM-PTx is a parameter-efficient method that adapts the Segment Anything Model by incorporating class-level semantic guidance through CLIP text embeddings.
• It introduces Parallel-Text adapters in the image encoder to merge spatial point prompts with precomputed text cues for improved segmentation under low-data regimes.
• Empirical results show modest yet consistent gains across datasets like COCO, ADE20K, and COD10K by enhancing segmentation performance with semantic conditioning.

Searching arXiv for the named paper and closely related SAM-based adaptations to ground the article. Searching for "SAM-PTx" on arXiv and adjacent SAM adaptation papers. SAM-PTx most specifically denotes a parameter-efficient method for adapting the Segment Anything Model (SAM) to text-guided segmentation by injecting frozen CLIP-derived text embeddings into SAM’s image encoder through Parallel-Text adapters, while keeping most of the original architecture frozen (Jalilian et al., 31 Jul 2025). The method addresses a central limitation of baseline SAM: strong prompt-based segmentation from spatial cues such as points and boxes, but no intrinsic mechanism for class-level semantic guidance. In SAM-PTx, point prompts remain the spatial signal, whereas text embeddings provide class-aware conditioning, so segmentation is driven jointly by geometry and semantics. In adjacent discussions, the term has also been used more loosely for other SAM-derived prompt-tuned or prompt-propagation systems; however, the proper titled usage refers to the text-guided adapter method introduced in 2025 (Jalilian et al., 31 Jul 2025).

1. Problem setting and conceptual scope

SAM-PTx is designed for text-guided, parameter-efficient fine-tuning of SAM. Its motivating premise is that original SAM is highly effective at segmenting “something” given spatial hints, but it does not systematically exploit semantic category information such as “person,” “apple,” or other class labels (Jalilian et al., 31 Jul 2025). This matters most in scenes with multiple visually similar objects, in settings where a point prompt alone is insufficient to disambiguate class identity, and in low-data adaptation regimes where full fine-tuning of SAM is computationally costly.

The method therefore combines two forms of conditioning. The first is spatial guidance, retained through point prompts. The second is class-level semantic guidance, supplied through CLIP text embeddings generated from prompts of the form “a photo of a {class}” (Jalilian et al., 31 Jul 2025). The text encoder is not trained jointly with SAM-PTx; instead, its embeddings are precomputed, cached, and reused. This design preserves the semantic prior learned by CLIP while restricting optimization to small adapter modules and the SAM mask decoder.

A central claim of the work is that semantic conditioning can be introduced without modifying the full SAM backbone. This places SAM-PTx within the broader parameter-efficient fine-tuning literature for foundation segmentation models, but with a distinct emphasis on text-conditioned segmentation rather than purely spatial prompting or full-model specialization (Jalilian et al., 31 Jul 2025).

2. Architectural design and Parallel-Text adapters

SAM-PTx uses SAM (ViT-B) as the base segmentor and CLIP ViT-B/32 as the text encoder. The macro-architecture of SAM is retained: an image encoder, a prompt encoder, and a mask decoder. The key intervention is inside the image encoder, where trainable adapters are inserted into each transformer block (Jalilian et al., 31 Jul 2025).

The design follows the parallel-adapter pattern used in prior SAM adaptation work. A standard parallel adapter is expressed as

$$\text{Adapter}(x) = x + W_{\text{up}} \cdot \sigma\bigl(W_{\text{down}} x\bigr),$$

where $x \in \mathbb{R}^{N \times d_v}$ are visual tokens, $$W_{\text{down}} \in \mathbb{R}^{d_{\text{bottleneck}} \times d_v}$$, $$W_{\text{up}} \in \mathbb{R}^{d_v \times d_{\text{bottleneck}}}$$, and $\sigma$ is GELU (Jalilian et al., 31 Jul 2025). SAM-PTx preserves this adapter structure for the MHSA-parallel branch, but extends only the MLP-parallel branch with text conditioning. This is a deliberate architectural choice: the attention pathway is preserved for spatial reasoning, whereas semantic text is injected into the per-token MLP pathway.

For a class label $\ell$, the frozen CLIP text encoder produces a 512-dimensional embedding

$t = f_{\text{text}}(\ell) \in \mathbb{R}^{512}.$

This embedding is projected into SAM’s visual token space by

$\tilde{t} = \mathrm{Act}(W_t t),$

then broadcast-added to each visual token,

$x' = x + \tilde{t},$

and processed by the text-conditioned adapter

$$\text{TextAdapter}(x,t) = W_2 \cdot \mathrm{Act}(W_1 x').$$

Here, $x \in \mathbb{R}^{N \times d_v}$0, $x \in \mathbb{R}^{N \times d_v}$1, and $x \in \mathbb{R}^{N \times d_v}$2 are trainable (Jalilian et al., 31 Jul 2025). The effect is to shift image-token representations by a class-dependent semantic vector before bottleneck processing.

The prompt encoder remains unchanged, and SAM-PTx still uses point prompts as spatial input. During training, the method uses five foreground points per object. At inference, the model takes an image, point prompt(s), and a class label, and returns a segmentation mask guided by both spatial and semantic cues (Jalilian et al., 31 Jul 2025).

3. Frozen and trainable components, training protocol, and efficiency

SAM-PTx is explicitly a parameter-efficient fine-tuning method. The original SAM image encoder weights are frozen, the CLIP text encoder is frozen, and the SAM prompt encoder is frozen. The trainable components are the MHSA-parallel adapters, the MLP-parallel adapters with text conditioning, and the SAM mask decoder (Jalilian et al., 31 Jul 2025).

The training objective is binary segmentation per object instance. Given a predicted mask $x \in \mathbb{R}^{N \times d_v}$3 and ground-truth mask $x \in \mathbb{R}^{N \times d_v}$4, the loss is

$x \in \mathbb{R}^{N \times d_v}$5

The optimization setup is fixed and lightweight: Adam, learning rate $x \in \mathbb{R}^{N \times d_v}$6, batch size 1, and 30 epochs. Images are resized to $x \in \mathbb{R}^{N \times d_v}$7. SAM’s low-resolution output masks are bilinearly upsampled to $x \in \mathbb{R}^{N \times d_v}$8 before loss computation (Jalilian et al., 31 Jul 2025).

A notable efficiency mechanism is that the text embeddings are precomputed and cached per class. This removes runtime CLIP inference and confines the added computation to small per-block adapter operations (Jalilian et al., 31 Jul 2025). The paper does not give absolute adapter parameter counts, but it characterizes the method as having minimal computational complexity and as suitable for low-resource, low-data regimes.

The semantic prompts are simple and fixed in format: for each dataset class, the text is “a photo of a {class}.” This simplicity is methodologically important. It shows that the performance gains do not depend on complex natural-language descriptions or end-to-end multimodal training. Instead, they arise from injecting frozen class embeddings directly into the image encoder representation (Jalilian et al., 31 Jul 2025).

4. Empirical performance, datasets, and ablations

SAM-PTx is evaluated on COD10K, COCO 1_512, and ADE20K 1_64. COCO 1_512 is a low-data subset with 232 labeled images and 631 binary instance samples across 20 object categories. ADE20K 1_64 contains 316 labeled images and 2,535 binary instance samples. COD10K is used for camouflaged object segmentation and evaluated with MAE (Jalilian et al., 31 Jul 2025).

On COCO and ADE20K, the reported metric is mIoU. The comparison against frozen SAM, decoder-only fine-tuning, and purely spatial parallel adapters is as follows:

Method	COCO 1_512	ADE20K 1_64
No fine-tuning	62.09	65.14
Decoder-only	67.29	70.32
Parallel	67.35	71.29
Parallel-Text	67.77	71.38

These gains are numerically modest but consistent. Relative to the purely spatial Parallel baseline, SAM-PTx improves COCO by +0.42 mIoU and ADE20K by +0.09 mIoU (Jalilian et al., 31 Jul 2025). This suggests that fixed text embeddings contribute useful class-level information even when the model already receives point prompts.

On COD10K, the reported metric is MAE, where lower is better:

Method	MAE ↓
SAM Adapter	0.025
SU-SAM (Mixed Adapter)	0.025
SU-SAM (Parallel Adapter, reported)	0.054
SAM-PTx (Parallel-Text Adapter)	0.021

The paper also notes a retrained SU-SAM Parallel adapter result of 0.0213 MAE, versus 0.0206 for SAM-PTx, rounded to 0.021 in the table (Jalilian et al., 31 Jul 2025). The authors state, with explicit caution, that this is the first work to use text prompts for segmentation on the COD10K dataset (Jalilian et al., 31 Jul 2025).

The ablation studies identify where semantic conditioning is most effective. On ADE20K 1_64, text injection into the image encoder yields 71.38 mIoU, compared with 71.11 for prompt-encoder-only injection and 70.82 for mask-decoder injection (Jalilian et al., 31 Jul 2025). A second ablation compares placing text-conditioned adapters in MLP-only versus MLP + MHSA. The results are 71.38 and 71.25 mIoU, respectively, supporting the design decision to confine semantics to the MLP branch.

Qualitative analysis further indicates that SAM-PTx improves edge-case behavior, reduces spill into adjacent regions, and is more likely to segment additional unprompted instances that are semantically consistent with the class label (Jalilian et al., 31 Jul 2025). This suggests that text guidance is not only a local disambiguation signal but also a mechanism for broader class-aware grouping.

5. Position within the broader SAM adaptation literature

SAM-PTx belongs to a wider research program on adapting SAM to domains, modalities, and interaction regimes for which baseline spatial prompting is insufficient. Within that landscape, its distinctive feature is semantic text conditioning inside the image encoder, rather than auto-prompting, full fine-tuning, or temporal memory augmentation.

Several adjacent works illustrate alternative adaptation strategies. PTSAM uses prompt tuning to turn SAM into a task-specific specialist with only 2,048 additional parameters in the decoder-only setting, and 75,776 trainable parameters when both image encoder and decoder are prompt-tuned; it is aimed at fully automatic specialization rather than text-guided class conditioning (Piater et al., 23 Apr 2025). PathoSAM converts SAM into a histopathology foundation model through multi-dataset supervised fine-tuning and an added AIS decoder for automatic nucleus instance segmentation (Griebel et al., 1 Feb 2025). SIS-PT-SAM combines MobileSAM with CoTracker for real-time surgical instrument segmentation in video, using sparse tracked points as prompts and achieving 84.8 IoU and 91.0 Dice on EndoVis 2015, with over 25/90 FPS on RTX 4060/4090 hardware (Wu et al., 2024). EMA-SAM modifies SAM-2 for PTMC segmentation in ultrasound video by adding a confidence-weighted exponential moving average pointer to the memory bank, improving maxDice from 0.82 to 0.86 and maxIoU from 0.72 to 0.76 while adding <0.1\% FLOPs (Dialameh et al., 21 Oct 2025).

This comparison clarifies SAM-PTx’s niche. It is not primarily a video model, an auto-prompting system, or a pathology-specialist foundation model. It is a multimodal PEFT method for making SAM class-aware while preserving its prompt-based interaction model (Jalilian et al., 31 Jul 2025). A broader implication is that SAM adaptation now spans at least five technical axes: semantic conditioning, prompt tuning, domain-specialist supervised training, temporal propagation, and video-memory stabilization.

A related baseline study on medical images reinforces the importance of such adaptation. In zero-shot 2D medical image segmentation, SAM 2 does not generally surpass SAM, performing worse in lower-contrast CT and ultrasound while being on par with or better on MRI; negative prompts substantially improve both models, especially SAM 2 (Sengupta et al., 2024). This suggests that raw backbone replacement is not sufficient, and that prompt design or domain-aware adaptation often determines performance more strongly than nominal model generation.

6. Limitations, interpretations, and prospective extensions

The empirical gains of SAM-PTx are consistent but generally incremental on COCO and ADE20K, and the method still relies on simple class labels rather than richer phrases or compositional referring expressions (Jalilian et al., 31 Jul 2025). The paper does not present open-vocabulary evaluation, although the use of frozen CLIP text embeddings suggests that such extensions are plausible. This suggests a natural research direction, but it remains an implication rather than a demonstrated capability.

Another limitation is that SAM-PTx preserves the standard SAM interaction model: it still requires spatial prompts, specifically point prompts, and therefore does not solve the fully automatic segmentation problem addressed by methods such as PTSAM or by task-specific specialist models (Piater et al., 23 Apr 2025). Nor does it directly address temporal consistency, occlusion, or online inference in video, which are the core concerns of SIS-PT-SAM and EMA-SAM (Wu et al., 2024, Dialameh et al., 21 Oct 2025).

The ablations also imply a fairly precise design lesson. Semantics are most effective when they reshape early visual features in the image encoder, but they should not heavily perturb the attention pathway used for spatial reasoning (Jalilian et al., 31 Jul 2025). A plausible implication is that future multimodal SAM variants may benefit from keeping spatial and semantic pathways partially disentangled, rather than merging them indiscriminately across all sublayers.

More broadly, SAM-PTx shows that fixed text embeddings can improve SAM when they are integrated inside the encoder rather than used only as external prompts or post hoc filters (Jalilian et al., 31 Jul 2025). In that sense, it marks a shift from prompt engineering around SAM toward representation-level semantic conditioning. The method’s broader significance lies less in large absolute metric jumps than in establishing a concrete and efficient blueprint for class-aware, multimodal SAM adaptation.