Promptable Concept Segmentation (PCS)

• PCS is a segmentation paradigm that uses external prompts to guide region extraction and handle diverse semantic concepts.
• It integrates various prompt types—such as clicks, boxes, texts, and multimodal inputs—to achieve open-set, user-controllable segmentation.
• PCS leverages foundation models, advanced fusion strategies, and robust training protocols to enhance performance in fields like medical imaging and computer vision.

Promptable Concept Segmentation (PCS) is a paradigm for segmentation tasks in which arbitrary semantic concepts—objects, parts, attributes, or textures—are specified at inference time by user, programmatic, or multimodal prompts. This enables open-set, user-controllable, and generalizable segmentation that transcends fixed taxonomies and addresses requirements in medical imaging, computer vision, language grounding, and interpretable learning. PCS subsumes point, box, region, and text prompts, and includes both vision-only and multimodal (e.g., vision-language) prompting. Modern implementations leverage foundation models, large-scale segmentation datasets, and architectural innovations for prompt encoding, fusion, and interactive simulation.

1. PCS: Definitions and Problem Formulation

Promptable Concept Segmentation generalizes classical semantic segmentation by parameterizing the segmentation task with external prompts, which may specify precise regions (clicks, boxes, scribbles, masks), high-level semantics (text, exemplars), or combinations thereof. The core mapping is defined as: $$(\text{image},\; \text{prompt}) \longmapsto \text{mask}$$ where the prompt can encode an arbitrary concept, possibly unseen during training. In medical imaging, PCS enables efficient interactive workflows, supporting click-based refinement and rapid segmentation of rare findings (Rokuss et al., 29 Aug 2025). In open-world scenarios, PCS allows users to segment any concept via language or example (Liu et al., 12 Oct 2025, Liu et al., 23 May 2025). The model must be robust both to prompt ambiguity and to realistic, incremental human or automated prompting sequences (Rokuss et al., 29 Aug 2025, Li et al., 23 Apr 2024).

2. Prompt Mechanisms and Encoding Strategies

PCS architectures operationalize prompt intake through multiple encoding schemes:

• Spatial (visual) prompts: Binary masks, Euclidean distance transforms (EDT), Gaussian kernels, and logit maps encode user clicks, boxes, or scribbles. EDT encoding yields superior performance to Gaussian kernels for click channels, with sharper spatial gradients critical for effective mask refinement in 3D volumetric data (Rokuss et al., 29 Aug 2025).
• Textual prompts and multimodal fusion: Prompts in natural language are embedded using vision-LLMs, such as CLIP. Advanced PCS models introduce part-aware, task-specific, or dynamic (instance-conditioned) prompts, and fuse them with visual features via cross-attention, adapters, or prompt learners (Han et al., 2023, Liu et al., 12 Oct 2025, Yu et al., 2023).
• Prototype-based and class-prompt encoders: For class-conditional or context-specific PCS, lightweight prototype banks and prompt encoders generate prompt tokens directly from learned prototypes, facilitating efficient class or instance prompting without explicit geometry (Yue et al., 2023).
• Interactive and iterative prompt handling: Models may process sequences of prompts across rounds, updating their prediction and integrating cumulative information (e.g., PRISM, which uses iterative correction, multi-head confidence scoring, and corrective shallow refinement (Li et al., 23 Apr 2024)).

Prompt Type	Encoding Example	PCS Context
Point/Click	EDT, Gaussian channel	Medical interactive (Rokuss et al., 29 Aug 2025)
Box/Scribble	Mask, box channel	3D, iterative (Li et al., 23 Apr 2024)
Text/Language	CLIP, prompt learners	Open-vocab, multimodal (Liu et al., 12 Oct 2025)
Prototype/Class	Prototype-based emb.	Fast adaptation (Yue et al., 2023)
Multi-modal	Visual + Text (joint fusion)	Unified segmentation (Liu et al., 12 Oct 2025)

3. Model Architectures and Fusion Operations

PCS frameworks adapt backbone architectures and introduce prompt-aware encoders-decoders to support flexible, robust segmentation:

• CNN/Transformer hybrids: For 3D volumes, local (convolutional) and global (transformer) feature extractors are fused to capture anatomical structure at different scales, supporting robust prompt integration (Li et al., 23 Apr 2024).
• Cross-attention and blending: Fusion of prompt (visual or textual) and image features is realized via cross-attention, feature adapters, and aligned token mixing. The image-prompt aligner in COSINE jointly refines all modalities, enabling strong open-vocabulary segmentation (Liu et al., 12 Oct 2025).
• Variant-specific adaptations: In medical imaging, 3D ViTs and U-Nets are extended with additional prompt channels, cross-attention layers, and iterative decoding for robust prompt intake and correction (Rokuss et al., 29 Aug 2025, Li et al., 23 Apr 2024). In part segmentation for 3D shapes, dual-branch encoders handle geometrically structured tokens, and prompt embedding leverages 3D positional encodings and triplane feature fields (Zhu et al., 26 Sep 2025).
• Prompt learning modules and regularization: Task-specific prompt alignment and boundary-aware modules (e.g., point matching) refine the mapping from prompt embedding to mask output. In language-grounded segmentation, explicit regularization connects mask prompts to linguistic dependencies for compositional and consistent mask extraction (Liu et al., 23 May 2025).

4. Training, Simulation, and Optimization Protocols

PCS models employ specialized training protocols to generalize across prompt types and semantic targets:

• Online prompt simulation: Training samples include randomly sampled user interaction prompts (points, boxes), dynamically encoded as additional channels, to avoid overfitting and ensure robustness under varied prompting (Rokuss et al., 29 Aug 2025).
• Hybrid loss functions: Standard segmentation losses (Dice, cross-entropy) are complemented by auxiliary objectives: prompt-alignment loss, uncertainty-guided rectification for noisy labels, consistency regularization across simulated prompts, and prompt-specific boundary terms for sharper edges (Zhang et al., 20 Feb 2025, Li et al., 23 Apr 2024, Kim et al., 14 Mar 2024, Han et al., 2023).
• Contrastive and cross-modal loss: For vision-language PCS, contrastive losses align image and prompt (text or prototype) features in joint embedding spaces, improving open-set and few-shot generalization (Han et al., 2023, Yu et al., 2023).
• Efficient distillation and tuning: To make large PCS models usable on-device, hierarchical distillation schemes train compact students by aligning features, temporal memory, and end-to-end output to a large PCS teacher under full prompt-in-the-loop scenarios (Zeng et al., 19 Nov 2025).

5. Benchmark Datasets and Performance Outcomes

PCS evaluation spans diverse domains, with each setting emphasizing different strengths:

• Medical imaging (PET/CT, MRI): Simulation of user prompts demonstrates that EDT-based 3D promptable models reduce both false positives (–4.8%) and false negatives (–62.9%) compared to automated baselines in multi-tracer, multi-center datasets (Rokuss et al., 29 Aug 2025). SegAnyPET achieves 90.49% Dice on seen organs and 89.04% on zero-shot unseen organs using only one prompt (Zhang et al., 20 Feb 2025).
• Few-shot and open-vocabulary part segmentation: Part-aware prompt learning in CLIP-based backbones yields 2–3% mIoU improvements on PartImageNet and cross-domain generalization to Pascal-Part (Han et al., 2023). 3D PartSAM outperforms 2D-derived approaches by 12–20% mIoU in both interactive and automatic open-world part segmentation (Zhu et al., 26 Sep 2025).
• Surgical instrument and general open-set segmentation: Promptable frameworks leveraging multimodal encoders and prompt mixture strategies demonstrate state-of-the-art performance on EndoVis and Cholecseg8k datasets (e.g., Ch_IoU 79.90% vs prior 72%) (Zhou et al., 2023). Segment Anyword, operating in a training-free regime, achieves 52.5 mIoU (Pascal Context-59) and 67.4 mIoU (GranDF) without per-concept adaptation (Liu et al., 23 May 2025).

Domain	Dataset/Metric	PCS Top Metric	Preceding Baseline
PET/CT	autoPET IV Dice (last)	76.35%	68.33% (autoPET III)
3D Parts	PartObjaverse-Tiny IoU@10	87.6% (PartSAM)	73.9% (Point-SAM)
Open-Vocab	Pascal Context-59 (mIoU)	52.5 (Anyword)	45.7 (baseline)
Med. 3D	PRISM-ultra (Colon Dice)	93.8%	67.2% (PRISM-plain)

6. Multimodal and Unified PCS Systems

• Unified multi-modal segmentation: COSINE consolidates text and visual prompt streams via feature-aligned architectures, outperforming prior art on LVIS (few-shot), ADE20K (open-vocab), and referring segmentation. Inference-time fusion of text and visual prompts yields further mIoU gains (36.3 vs 35.7 on ADE20K, multi-modal vs text only) (Liu et al., 12 Oct 2025).
• Promptable video segmentation: SAM3-architecture PCS supports detection, segmentation, and tracking of concept instances in both images and videos—guided by text, exemplar crops, or their fusion—scaling to on-device efficient models via hierarchical distillation (Zeng et al., 19 Nov 2025).
• Generalization and interactive scenarios: Task-generic PCS with automatic prompt refinement (ProMaC) reduces dependency on manual prompts, leveraging MLLM hallucinations and prompt-masking cycles for high accuracy on camouflaged, medical, and transparent object benchmarks (Hu et al., 27 Aug 2024). Language guidance and visual reasoning are further used for clustering and regularizing mask proposals (Liu et al., 23 May 2025).

7. Interpretability and Future Directions

PCS facilitates interpretable and transparent segmentation pipelines:

• Concept-based decompositions: PCS can reveal the compositional structure of prompt embeddings, decomposing them into human-readable concepts via provable matrix factorizations. Submodular selection and concept attribution metrics confirm faithfulness and semantic interpretability (Chen et al., 2 Dec 2024).
• Open problems: Robust handling of prompt ambiguity, overlapping objects, and very small/thin structures remains challenging, as does the speed of complex PCS pipelines. Future research involves adaptive prompt encoding, efficient multi-modal fusion, larger concept pools, and joint learning with large language–vision models. Real-world deployments increasingly demand on-device, real-time PCS realizations, met by recent advances in knowledge distillation and compact backbone design (Zeng et al., 19 Nov 2025).

PCS thus represents a foundational shift toward interactive, generalizable, and interpretable segmentation across modalities, domains, and prompt forms.