Promptable Image Segmentation

• Promptable image segmentation is a family of methods that use external prompts like spatial annotations and textual cues to generate precise segmentation masks.
• It decouples task-specific segmentation from model training, enabling zero-shot and few-shot adaptation across both natural and medical imaging.
• Empirical studies show reduced annotation effort, high generalization, and enhanced interactive usability through multimodal and automated prompting.

Promptable image segmentation comprises a family of methods that deliver segmentation outputs guided by explicit external prompts, including spatial annotations (points, bounding boxes, scribbles, or masks) and/or semantic cues (class names, textual descriptions). The central innovation is the decoupling of task-specific segmentation from model training: the same pretrained model can, at inference, segment arbitrary regions or object classes as dictated by the chosen prompt, supporting flexible, interactive, and generalizable image understanding. This paradigm has reshaped both natural and medical image analysis, enabling zero-shot and few-shot adaptation, multimodal guidance, reduced annotation effort, and new forms of human–AI collaboration. The following sections summarize principles, representative architectures, key methodologies, empirical results, and current limitations.

1. Prompt Modalities and Model Taxonomy

Promptable segmentation models process heterogeneous input prompts to produce segmentation masks. The dominant prompt types and architectural paradigms are:

• Spatial prompts: Individual points (positive/negative), bounding boxes, scribbles, or masks directly encode image regions of interest. These prompts are typically encoded as additional channels, location embeddings, or position-encoded tokens fed into segmentation backbones (Isensee et al., 11 Mar 2025, Li et al., 2024, Wang et al., 2024).
• Semantic prompts: Text (class labels, descriptions, or referring expressions) is encoded by a LLM to bias segmentation, enabling open-set and open-vocabulary capabilities in both natural and medical domains (Chen et al., 26 Jun 2025, Rokuss et al., 14 Nov 2025, Yuan et al., 16 Feb 2025, Lüddecke et al., 2021).
• Multimodal fusion and interaction: Dual-branch or transformer-based models combine spatial and semantic cues, fusing information via attention or explicit selection mechanisms (Xu et al., 25 Mar 2025, Chen et al., 26 Jun 2025).
• Promptable foundation models: Large-scale pretrained models such as SAM/MedSAM, SegAnyPET, and VoxTell accept prompts in various forms and generalize to unseen tasks in a zero-shot or minimally supervised regime (Zhang et al., 20 Feb 2025, Rokuss et al., 14 Nov 2025, Li et al., 2023).

A summary of prompt types and representative models:

Prompt Type	Representative Models	Input Modalities
Points/Boxes	SAM, nnInteractive, ProMISe	Points, boxes, masks
Text	PathSegmentor, VoxTell, CLIPSeg	Free-form language
Dual-modal	BiPrompt-SAM, TPP, ProMaC, INT	Points + text, boxes + text
Sequence	SPT, TPP	Image sequences + prompts
Automated	GeomPrompt, ProMaC, INT	Algorithm/generated prompts

2. Architectural and Algorithmic Frameworks

Advances in promptable segmentation derive from integration of prompt representation, prompt-to-mask reasoning, and cross-modal fusion.

• Prompt encoding: Points and boxes are rasterized or embedded with Gaussian/positional kernels; scribbles and masks are binary mask channels. Language prompts are tokenized and mapped to dense vectors (e.g., via BERT or Qwen3-Embedding) (Rokuss et al., 14 Nov 2025, Chen et al., 26 Jun 2025).
• Backbones: UNet-based (ResEnc-L), transformer-based (ViT-B/H), or hybrid architectures process images, often with separate CNN and transformer streams to capture local and long-range context (Li et al., 2024, Li et al., 2023).
• Prompt fusion: Early concatenation (nnInteractive), cross-attention modules (PathSegmentor, SPT, VoxTell), and explicit selection via Intersection over Union (BiPrompt-SAM) are prevalent (Xu et al., 25 Mar 2025, Rokuss et al., 14 Nov 2025).
• Iterative/interactive refinement: Models such as PRISM and SPT support multi-step prompting and refinement, leveraging past outputs or user corrections (Li et al., 2024, Cheng et al., 2024).
• Selection/gating: Dual-branch MoE-style late fusion enables explicit reasoning about which prompt branch to trust, as in BiPrompt-SAM (Xu et al., 25 Mar 2025).

Algorithmic advances include:

• Prompt selection: Top-k prompt selection (TPS) identifies the most relevant prior examples for current predictions (Cheng et al., 2024).
• Measurement of mask quality and fusion: IoU, CLIP-based semantic alignment, and mask confidence ranking are used to select among candidate masks (Xu et al., 25 Mar 2025, Hu et al., 2024).
• Data-efficient training: Minimal-label strategies train small prompt or classifier modules using tens of labeled examples, employing local search or spiral-guided mask propagation (Karam et al., 23 May 2025).

3. Multimodal and Multi-task Prompt Strategies

Emerging models leverage semantic and spatial prompts simultaneously or autonomously generate prompts:

• Explicit dual-branch selection: BiPrompt-SAM runs point-based segmentation (SAM) and text-based segmentation (EVF-SAM, BEIT-3) in parallel and selects the point mask with highest IoU to the text mask, yielding strong zero-shot performance and substantial reduction in annotation burden versus standard SAM bounding boxes (Xu et al., 25 Mar 2025).
• Automated or guided prompt generation: Frameworks such as ProMaC and INT use multimodal LLMs to mine hallucinated instance-level prompts from task-generic prompts (“polyp,” “camouflaged animal”), iteratively refining prompts via visual contrastive reasoning and semantic mask alignment (Hu et al., 2024, Hu et al., 30 Jan 2025).
• Task-generic pipelines: These methods relax per-instance manual prompt constraints and segment new images using only high-level, universal prompts, relying on vision-language exploration and negative mining to disambiguate target instances (Hu et al., 30 Jan 2025, Hu et al., 2024).
• Class-prompted segmentation: SurgicalSAM eliminates explicit spatial prompts in imagery with high inter-class similarity (e.g., surgical instruments) by learning contrastive prototype-based prompt encoders, tuned with minimal parameters and yielding high class-specific accuracy (Yue et al., 2023).

4. Empirical Benchmarks and Practical Impact

Promptable segmentation models have demonstrated rapid progress on a variety of axes.

• Zero-shot and few-shot performance: Models such as VoxTell and SegAnyPET report state-of-the-art Dice on seen and unseen anatomy or pathologies using only 1–5 points or a single text phrase, outperforming fully supervised and task-specific baselines (Rokuss et al., 14 Nov 2025, Zhang et al., 20 Feb 2025).
• Annotation efficiency: Single-point prompting achieves performance competitive with box- and mask-based annotation while cutting annotation time by a factor of 5–10 in clinical tasks (Xu et al., 25 Mar 2025, Yang et al., 19 Feb 2026).
• Generalization: Promptable segmenters trained on datasets spanning over 1000 anatomical and pathological classes generalize to new unseen structures, rare disease sites, and cross-modal (CT, MRI, PET) input (Rokuss et al., 14 Nov 2025, Zhang et al., 20 Feb 2025).
• Automation and reduced prompt dependency: Multimodal LLM-driven and geometric prompting systems (e.g., ProMaC, INT, GeomPrompt) move towards fully automated pipelines with little or no manual input, exploiting generative and geometric priors to select informative prompts (Hu et al., 30 Jan 2025, Hu et al., 2024, Ball et al., 27 May 2025).
• Interactive usability and toolchain integration: nnInteractive and PRISM deliver plug-ins for major clinical and research viewing software, providing user-friendly, real-time 3D promptable segmentation (Isensee et al., 11 Mar 2025, Li et al., 2024).

Table: Key empirical results for promptable image segmentation models

Model/Method	Input Modality	Benchmark/Dataset	Dice/IoU (%)	Notable Features
BiPrompt-SAM	Point + text	Endovis17, RefCOCO(x)	89.55/81.46, 87.1	IoU-based dual-branch fusion
SegAnyPET	3D points	PETS-5k	91 (seen), 89 (unseen)	Cross-prompt consistency
VoxTell	Free-form text	62K multi-modal volumes	70.9 (mean Dice)	Multi-stage vision-language fusion
nnInteractive	Points, lasso, scribble	120+ datasets	0.90 (Dice)	Early 3D prompting
ProMaC, INT	Automated, text generic	COD, medical, OVS	SOTA per metric	LLM-powered prompt mining

5. Data-Efficient, Domain-Adaptive, and Automated Prompting

Recent research emphasizes data efficiency and domain adaptation:

• Minimal label training: Classifier-driven pipelines achieve promptable segmentation with only 24–32 expert-labeled images in pathology, matching the dice of U-Net ensembles trained on 30–100× more data (Karam et al., 23 May 2025).
• Non-invasive adaptation: ProMISe adapts SAM to new domains with an auto-prompting module (APM) and incremental pattern shifting (IPS) that preserve full promptability without modifying base model weights (Wang et al., 2024).
• Geometry-guided automated prompting: GeomPrompt applies multiscale Hessian-based ridge detection to generate spatial prompts aligned with elongated structures, drastically reducing the number of prompts needed for recall in plant root segmentation (Ball et al., 27 May 2025).
• Detector-guided hybrid prompting: Tiny-YOLOSAM parameterizes the prompt set using fast detectors plus targeted sparse sampling, balancing runtime and coverage for full-scene segmentation (Xu et al., 20 Dec 2025).

6. Extensions: Sequences, 3D, and Multi-View Consistency

Modern promptable segmenters increasingly address more complex data structures:

• Sequential image segmentation: SPT and TPP extend prompting to temporal or volumetric sequences, using transformers with causal or multi-frame attention to condition on past (or adjacent) frames, prior masks, and interactive clicks (Cheng et al., 2024, Yuan et al., 16 Feb 2025).
• 3D and multi-view integration: Models such as MV-SAM achieve 3D-consistent segmentation across multiple views by lifting 2D features into a 3D pointmap and performing cross-attention with 3D prompts and position embeddings, avoiding explicit 3D pretraining (Jeong et al., 25 Jan 2026).
• Multi-click and hybrid prompt adaptation: ProMISe and PRISM demonstrate principled iterative prompting and correction for progressively refining difficult cases and supporting various prompt types (e.g., boxes, scribbles, prior masks) (Wang et al., 2024, Li et al., 2024).

7. Limitations and Challenges

Despite rapid advances, several limitations remain:

• Semantic ambiguity and hallucination: LLM-based prompt generators can hallucinate or mislocalize instance-specific cues; iterative negative mining and VCR strategies mitigate but do not eliminate such errors (Hu et al., 30 Jan 2025, Hu et al., 2024).
• Prompt sensitivity: Performance can depend strongly on prompt phrasing, location, or type; properly engineered contextual prompts yield measurable gains in accuracy (Xu et al., 25 Mar 2025, Chen et al., 26 Jun 2025).
• Computational and integration overhead: Large foundation models and sequential transformer architectures may be computationally intensive; hybrid and distilled architectures like Tiny-YOLOSAM partially address this (Xu et al., 20 Dec 2025).
• Limited class expansion in some pipelines: Certain class-promptable or prototype-based approaches require a pre-enumerated set of target classes; generalization to new concepts or fine-grained substructure often requires further data or tuning (Yue et al., 2023, Rokuss et al., 14 Nov 2025).
• Zero-shot boundaries: Even large-scale models such as VoxTell underperform on truly out-of-distribution structures or textures without minimal fine-tuning data (Rokuss et al., 14 Nov 2025).

Ongoing research explores more robust multimodal prompt fusion, few-shot domain extension, further user-burden reduction, uncertainty calibration, and better alignment between open-vocabulary reasoning and domain-specific precision.