Hybrid Prompt-Driven SAM (HyPSAM)

Updated 12 July 2026

HyPSAM is a framework that integrates multiple prompt modalities—including spatial, semantic, and dense prompts—to enhance segmentation precision and adaptability.
It leverages a frozen SAM backbone combined with lightweight prompt refiners, enabling efficient adaptation to domain-specific challenges without full retraining.
Empirical studies show significant improvements in metrics like mIoU and interaction speed across applications such as salient object detection and cryo-EM.

Searching arXiv for the cited HyPSAM-related papers and closely related prompt-driven SAM work. Hybrid Prompt-Driven Segment Anything Model (HyPSAM) denotes a family of Segment Anything Model (SAM)-based segmentation frameworks that combine multiple prompt types, prompt-learning mechanisms, or fusion regimes in order to improve segmentation quality, domain adaptation, or interaction efficiency. In the literature, the term is used both as a general descriptor for hybrid prompt-driven SAM adaptation and as the formal name of a two-stage RGB-thermal salient object detection method. Across these usages, the recurring idea is to preserve SAM’s promptable segmentation backbone while augmenting it with hybrid prompt inputs such as spatial and semantic embeddings, sparse point and dense mask prompts, text-mask-box prompt triplets, or lightweight refinement modules (Huang et al., 2024, Ahmad et al., 14 Jun 2026, Hou et al., 23 Sep 2025).

1. Terminological scope and research context

SAMs such as SAM and SEEM are built around promptable segmentation: a handcrafted prompt is provided as input, and the model returns a segmentation mask. The prompt space includes spatial prompts such as points and semantic prompts such as texts, and these prompt types can be combined to segment downstream data (Huang et al., 2024). HyPSAM emerges from attempts to make that prompt interface more adaptive, more efficient, or more domain-aware without retraining the full foundation model.

The term does not denote a single universally fixed architecture across the literature. In one line of work, HyPSAM refers to jointly learned spatial-semantic prompt embeddings for downstream segmentation (Huang et al., 2024). In another, it denotes a fully zero-shot seismic workflow that combines sparse user-defined point prompts with dense mask prompts derived from SAM’s internal feature activations (Ahmad et al., 14 Jun 2026). In cryo-EM, the label is used collectively for head prompt, prefix prompt, and encoder prompt strategies that adapt SAM while keeping the original ViT-encoder, prompt-encoder, and mask-decoder frozen (He et al., 2023). A later RGB-thermal salient object detection paper formalizes “HyPSAM” as the name of a coarse-to-fine model that uses text, mask, and box prompts to guide SAM refinement (Hou et al., 23 Sep 2025). This suggests that HyPSAM is best understood as a research pattern centered on hybrid prompt design rather than as one canonical model.

A common motivation across these variants is that standard SAM prompting can be under-specified for domain-specialized or detail-sensitive segmentation. The cited works target this issue from different directions: learning prompts directly in embedding space, injecting image-prompt synergy into late-fusion interaction, reusing SAM logits as dense prompts, or selecting prompts conditioned on modality quality and domain-specific representations (Yu et al., 2024, Ahmad et al., 14 Jun 2026, Hou et al., 23 Sep 2025).

2. Shared architectural principles

A representative HyPSAM formulation is given by Spatial-Semantic Prompt learning (SSPrompt), which keeps the base SAM components frozen during prompt learning: an image encoder $E^{I}$ mapping $x^{I}\in\mathbb{R}^{H\times W\times 3}$ to an image embedding $z^{I}$ , a text prompt encoder $E^{T}$ mapping class names to text embeddings, a spatial prompt encoder $E^{S}$ mapping a grid of 2D points to spatial embeddings, and a mask decoder $D$ that takes $(z^{I}; Z^{S}, Z^{T})$ and returns mask(s) and confidence (Huang et al., 2024). Hybrid prompting then appears as a modification of the prompt pathway rather than of the core segmentation backbone.

In SSPrompt, the hybridization is explicit in the fusion of frozen default embeddings with learnable “virtual” embeddings through scalar interpolation weights in $[0,1]$ :

$z^{S\,(n)}_{\mathrm{SpaPrompt}} = \hat w^{S}_{n}\,\hat z^{S}_{n} + (1-\hat w^{S}_{n})\,z^{S}_{n},$

$z^{T\,(c)}_{\mathrm{SemPrompt}} = \hat w^{T}_{c}\,\hat z^{T}_{c} + (1-\hat w^{T}_{c})\,z^{T}_{c}.$

The resulting spatial and semantic prompt sets replace the default prompt embeddings at inference time and drive the same frozen mask decoder (Huang et al., 2024).

Other HyPSAM variants preserve the same division of labor but alter the interaction schedule. SAM-REF performs a late-fusion coarse pass once, caches the heavy ViT image embedding, and then re-introduces image-prompt synergy through lightweight early-fusion-style refinement subnetworks called GlobalDiff and PatchDiff (Yu et al., 2024). The seismic HyPSAM similarly runs an initial SAM decoding pass to obtain a mask and pre-sigmoid logits, then feeds the logits back as a dense prompt together with sparse point prompts in a second pass:

$x^{I}\in\mathbb{R}^{H\times W\times 3}$ 0

This two-pass formulation turns SAM’s own output into an additional prompt modality (Ahmad et al., 14 Jun 2026).

These designs share three structural commitments. First, the heavy SAM backbone is often frozen or reused, reducing optimization cost and preserving pre-trained generality. Second, the hybrid component is small relative to the backbone: prompt embeddings, adapters, lightweight refiners, or prompt-construction modules. Third, the hybrid prompt is not merely a concatenation of inputs; it is usually a selective mechanism that decides how much to trust a default prompt, a learned prompt, a coarse mask, or a modality-specific view (Huang et al., 2024, Hou et al., 23 Sep 2025).

3. Embedding-space prompt learning: SSPrompt as a HyPSAM formulation

SSPrompt is one of the clearest formulations of HyPSAM as prompt learning in a frozen SAM. It introduces two parallel learnable modules: SpaPrompt for spatial prompt learning and SemPrompt for semantic prompt learning. SpaPrompt learns a new set of spatial embeddings by fusing default point embeddings with $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 1 virtual embeddings, while SemPrompt learns a new set of class embeddings by fusing default text embeddings with $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 2 virtual embeddings. Both text and spatial prompt encoders remain strictly frozen, and only the small sets $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 3 are updated (Huang et al., 2024).

The training objective is defined over a few-shot set $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 4:

$x^{I}\in\mathbb{R}^{H\times W\times 3}$ 5

The framework can also be interpreted through separate spatial and semantic loss terms combined by weighted summation (Huang et al., 2024).

Implementation details are notably lightweight. The reported vision backbones are Focal-Tiny or DaViT-Large with embedding dimension $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 6; the number of semantic prompts equals the number of dataset classes, and the number of spatial prompts is $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 7 grid points. Training uses SGD with learning rate $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 8, poly schedule $x^{I}\in\mathbb{R}^{H\times W\times 3}$ 9, and weight decay $z^{I}$ 0. The few-shot regime uses 4–16 images per class, with random horizontal flip and resizing of the shorter side to 512 px (Huang et al., 2024).

The quantitative results show consistent gains in 16-shot semantic segmentation with the Focal-Tiny backbone. SSPrompt reaches 55.2 mIoU on Cityscapes, 47.1 on BDD100K, 49.5 on Mapillary, 23.2 on ADE20K, and 51.2 on PASCALCtx, exceeding SEEM-T, CoOp, and LOCN on each listed dataset (Huang et al., 2024). On ACDC adverse-condition segmentation, SSPrompt reports 47.5 for fog, 32.1 for night, 39.9 for rain, 43.1 for snow, and 40.6 mean, again outperforming the reported baselines (Huang et al., 2024).

The method also generalizes across segmentation tasks without retraining the core SAM. On Cityscapes with 16 shots and Focal-Tiny, semantic segmentation improves from 39.2 to 55.2 mIoU, instance segmentation from 32.7 to 37.7 AP $z^{I}$ 1, and panoptic segmentation from 32.4 to 38.0 PQ. Efficiency measurements on ADE20K show training time per image decreasing from 87.5 ms for SEEM-T to 56.0 ms for SSPrompt and GPU memory dropping from 8.22 GB to 3.82 GB (Huang et al., 2024).

The paper attributes these gains to three specific properties: moving point-prompt search from 2D coordinates to a 512-D embedding space, mitigating foreground-background bias in the text encoder via learnable interpolation weights, and keeping the learned component very lightweight. Its stated limitations are equally precise: it requires 4–16 shots per class, hyperparameters such as the number of prompts and weight initialization may need tuning per dataset, and the frozen SAM backbone cannot correct model-level pre-training errors such as missing rare categories (Huang et al., 2024).

SAM-REF instantiates HyPSAM as a compromise between early-fusion specialist interactive segmentation and late-fusion SAM efficiency. Stage A performs a late-fusion coarse pass: the image is encoded once by a heavy ViT, the prompts are encoded, and the SAM decoder outputs coarse mask logits $z^{I}$ 2. Stage B introduces two refinement subnetworks. GlobalDiff is a full-image lightweight convolutional refiner that progressively injects the saved SAM image embeddings, while PatchDiff is an on-demand local-window refiner applied only to regions with the highest predicted error (Yu et al., 2024).

The central refinement equation for the global stage is

$z^{I}$ 3

where $z^{I}$ 4 is an error map and $z^{I}$ 5 is a detail prediction. The local patch stage uses the same form on cropped regions (Yu et al., 2024). This design preserves the late-fusion advantage that the ViT encoder is run only once, while recovering detail through additional image-prompt interaction.

The computational trade-off is explicit. On CPU with ViT-H, SimpleClick reports about 8.24 s SPC, SAM reports 0.413 s, and SAM-REF reports 0.511 s. In the FLOP accounting provided, if $z^{I}$ 6 GFlop, $z^{I}$ 7 GFlop, $z^{I}$ 8 GFlop, and a typical PatchDiff $z^{I}$ 9 GFlop, then SAM-REF costs about 26 GFlop per click compared with about 220 GFlop for SimpleClick (Yu et al., 2024).

The segmentation results are mixed in a manner typical of interactive evaluation. On the zero-shot NoC90/NoC95 benchmarks with ViT-H, SAM-REF reports $E^{T}$ 0 on GrabCut, $E^{T}$ 1 on Berkeley, $E^{T}$ 2 on DAVIS, and $E^{T}$ 3 on SBD. On the HQ-Seg benchmark, it reports 5-click mIoU = 89.6%, NoC90 = 5.44, NoC95 = 9.16, and latency $E^{T}$ 4 s, compared with FocSAM’s 88.6%, 5.74, 9.44, and $E^{T}$ 5 s, and SAM (ViT-B)’s 86.16%, 7.46, 12.42, and $E^{T}$ 6 s (Yu et al., 2024). The paper’s own conclusion is that GlobalDiff recovers most coarse errors, PatchDiff handles localized hard regions, and the heavy encoder reuse preserves sub-second interaction latency.

The seismic interpretation variant uses a different refinement loop but a comparable principle. It is a fully zero-shot workflow combining domain-guided input transformations with hybrid prompting. The input may be a raw amplitude image, a seismic attribute, or a color-mapped rendering. The user provides sparse foreground/background clicks, SAM decodes an initial mask $E^{T}$ 7 and pre-sigmoid logits $E^{T}$ 8, and the logits are then reused as a dense mask prompt to generate a refined mask $E^{T}$ 9 (Ahmad et al., 14 Jun 2026).

The reported results emphasize target dependence. For salt segmentation, F1 rises from 0.69 with point prompts to 0.90 with point+mask prompting, to 0.91 with attribute-guided HyPSAM, and to 0.927 with the best colormap. For mud-filled channels, the progression is 0.66, 0.78, 0.84, and 0.747. For Lower North Sea facies, the sequence is 0.90, 0.91, 0.89, and 0.874 (Ahmad et al., 14 Jun 2026). Prompt-density ablations show that salt and facies F1 plateaus around 0.85–0.89 by about 8–12 prompts, channels saturate around 0.70 without attributes and around 0.84 with attribute-guided HyPSAM, and additional clicks beyond about 12 yield diminishing returns (Ahmad et al., 14 Jun 2026). The associated best-practice guidance is explicit: attribute alignment should match the geological target, high-contrast colormaps are useful for salt-like structural bodies but often degrade subtle stratigraphic targets, and HyPSAM outputs should be treated as a first pass for interactive review (Ahmad et al., 14 Jun 2026).

5. Domain-specific instantiations

In cryo-EM micrographs, HyPSAM is framed as parameter-efficient SAM adaptation for protein particle picking. Three prompt-based strategies are reported. Head prompt prepends a small U-Net to enhance the input image before feeding it into frozen SAM. Prefix prompt prepends learnable tokens to each transformer layer. Encoder prompt inserts bottleneck adapter MLPs into each transformer block with scaling factor $E^{S}$ 0. In all three cases, the original ViT-encoder, prompt-encoder, and mask-decoder remain frozen, and training uses only Dice loss with Adam at learning rate $E^{S}$ 1, early stopping on validation Dice, and 100 epochs (He et al., 2023).

The trade-off between accuracy and compute is measured on CryoPPP using EMPIAR 10028 with 10 training micrographs. Head prompt reports Dice 0.726 with $E^{S}$ 2 trainable parameters and about 12 GB GPU RAM. Prefix prompt reports 0.752 with $E^{S}$ 3 parameters and less than 16 GB. Encoder prompt reports 0.712 with $E^{S}$ 4 parameters and more than 40 GB. Full fine-tuning reports 0.702 with $E^{S}$ 5 updated parameters and about 30 GB. The same table lists pretrained crYOLO at 0.723 and Topaz at 0.636 (He et al., 2023). Across 27 protein types, prompt-based methods outperform fine-tuning on 25 types, while encoder prompts achieve the best peak accuracy given sufficient data but require at least 20–30 images to avoid divergence (He et al., 2023).

The RGB-thermal salient object detection method titled “HyPSAM” formalizes the name as a two-stage coarse-to-fine architecture. The first stage, Dynamic Fusion Network (DFNet), processes aligned RGB and thermal images with a dual-stream SwinV2-B backbone, a Dynamic Interaction Module with $E^{S}$ 6 kernels per layer, and a Multi-branch Decoding Module that produces modality-specific saliency maps, a fused map, and a boundary map. Decision Fusion aggregates these into an initial saliency map $E^{S}$ 7 (Hou et al., 23 Sep 2025).

The second stage, Plug-and-Play Refinement Network (P2RNet), uses CLIP-IQA with the antonym text prompts “bright vs dark” and “colorful vs dull” to select the more reliable modality. It then constructs three prompts from $E^{S}$ 8: a text prompt, a thresholded mask prompt, and a minimal-enclosing-rectangle box prompt. Frozen SAM is invoked as

$E^{S}$ 9

and the refined saliency map is

$D$ 0

DFNet is trained end-to-end with hybrid saliency loss $D$ 1 and boundary Dice loss, while P2RNet itself is not additionally trained (Hou et al., 23 Sep 2025).

On VT5000, VT1000, and VT821, the method reports $D$ 2/MAE values of 0.911/0.019, 0.944/0.011, and 0.903/0.020, respectively, improving over SwinNet, ADNet, and DFNet alone on the listed metrics. DFNet is reported at 133.3 M parameters, 74.1 G FLOPs, and 41 FPS, while the overall HyPSAM pipeline has high computational overhead due to the large SAM (ViT-H), listed as 817.6 M parameters and 3033 G FLOPs (Hou et al., 23 Sep 2025). Ablations show that removing the Dynamic Interaction Module drops $D$ 3 by about 3.5% on VT5000, removing Multi-branch Decoding causes a further about 2% drop, omitting P2RNet prompts reduces $D$ 4 to 0.905, and the full HyPSAM gains about 1.3% over DFNet alone (Hou et al., 23 Sep 2025).

Several adjacent works expand the HyPSAM design space even when they do not use the term as the formal model name. SAM-CP constructs two composable prompt types: Type-I semantic prompts scoring SAM patches against text labels, and Type-II instance prompts scoring whether patches belong to the same object instance. These are unified through query-patch affinity propagation with Dynamic Cross-Attention, Affinity Refinement, and Query Enhancement, yielding open-vocabulary and closed-set semantic, instance, and panoptic segmentation. Reported results include open-vocabulary COCO $D$ 5Cityscapes performance of PQ = 41.0, AP = 29.3, and mIoU = 47.9, as well as closed-set COCO-Panoptic performance of PQ = 52.7, AP = 45.2, and mIoU = 61.8 with Swin-L (Chen et al., 2024). The paper explicitly proposes extending this framework into a HyPSAM by adding point or box prompts as extra queries.

VTPSeg offers a training-free remote sensing pipeline that combines Grounding DINO $D$ 6, CLIP Filter $D$ 7, and FastSAM. Grounding DINO $D$ 8 generates multi-scale, synonym-enriched bounding boxes; CLIP Filter $D$ 9 prunes them using visual and textual prompt fusion; FastSAM takes the centroid of each surviving box as a point prompt and returns object masks. The pipeline reports MIoU 53.22 on LoveDA, 46.62 on Inria Aerial, 50.19 on xBD, and 61.0% on iSAID, with ablations showing that multi-scale inference plus NMS raises recall by about 8 points and CLIP Filter $(z^{I}; Z^{S}, Z^{T})$ 0 adds about 1.7 MIoU on LoveDA (Zi et al., 10 Mar 2025). This is not presented as HyPSAM by name, but it is a direct example of hybrid prompt composition across detector, vision-language filter, and promptable segmenter.

SAMPOT contributes a different mechanism: prompt evolution. With SAM and an oracle regressor both frozen, a point prompt $(z^{I}; Z^{S}, Z^{T})$ 1 is optimized by gradient ascent,

$(z^{I}; Z^{S}, Z^{T})$ 2

to maximize a proxy quality score (Sathish et al., 2023). On 53 chest X-ray test images, the method improves Dice in 38 cases, approximately 72%, raising mean Dice from $(z^{I}; Z^{S}, Z^{T})$ 3 to $(z^{I}; Z^{S}, Z^{T})$ 4. The same work outlines several possible HyPSAM integrations, including multi-prompt injection, weighted fusion in prompt-embedding space, and gated refinement loops (Sathish et al., 2023).

Across the literature, the limitations are consistent. SSPrompt still requires 4–16 annotated shots per class and remains constrained by the frozen backbone’s pre-training coverage (Huang et al., 2024). SAM-REF adds quality but not for free, incurring about 25% overhead over the pure SAM decoder in SPC terms (Yu et al., 2024). Seismic HyPSAM remains interactive and domain-dependent; colormap rendering can degrade subtle facies and channel targets, and outputs should be reviewed as an assistant-generated first pass (Ahmad et al., 14 Jun 2026). Cryo-EM encoder prompts can demand more than 40 GB GPU RAM and may diverge at very low sample sizes (He et al., 2023). The RGB-thermal HyPSAM remains computationally heavy because the large SAM dominates total parameters and FLOPs, and performance degrades when thermal contrast is very low or initial mask noise is high (Hou et al., 23 Sep 2025).

Taken together, these works suggest several stable research directions. One is to keep the SAM backbone frozen and move learning into prompt-space or lightweight refiners. Another is to treat SAM’s own intermediate or output signals—logits, patches, mask proposals, confidence cues—as new prompt modalities rather than as terminal predictions. A third is to unify human prompts, learned prompts, and model-generated prompts inside a common decoder or affinity framework. In that sense, HyPSAM names a broader shift in SAM research: from prompt consumption to prompt engineering, prompt learning, and prompt composition as the principal locus of adaptation (Huang et al., 2024, Chen et al., 2024, Sathish et al., 2023).