Cascaded Prompts for SAM (CPS)

Updated 20 October 2025

Cascaded Prompts for SAM (CPS) is a multi-stage, multi-modal strategy that refines segmentation outputs iteratively using diverse prompt types.
It integrates cues from points, masks, boxes, and language inputs, significantly improving segmentation accuracy and boundary precision.
CPS architectures enhance robustness, efficiency, and automation across domains such as medical imaging, geospatial analysis, and industrial defect detection.

Cascaded Prompts for SAM (CPS) refers to a family of strategies that structure the input prompting process for the Segment Anything Model (SAM) into multiple, mutually reinforcing stages or modalities. This concept encompasses workflows that move beyond single, static prompt inputs—integrating hierarchies of prompt types (e.g., points, masks, boxes, language cues), sequential refinement steps, feedback loops, or explicit decision-level fusion. CPS approaches have demonstrated tangible improvements in segmentation accuracy, boundary refinement, and task automation across domains such as medical imaging, geospatial analysis, industrial defect detection, and open-vocabulary segmentation. The following sections summarize major CPS principles, representative architectures, performance implications, and their relationship to foundational prompting paradigms.

1. Core Principles of Cascaded Prompts

CPS architectures systematically orchestrate prompt delivery to SAM via multi-stage or multi-modal processes. Rather than relying on isolated point or box prompts supplied either manually or heuristically, they aim to:

Integrate prompt types drawn from complementary modalities (points, boxes, masks, text, VLM/CLIP embeddings).
Structure prompt provision through sequential modules, where each prompt (or set of prompts) is derived or refined based on prior segmentation results or auxiliary models.
Employ feedback, selection, or fusion mechanisms that weight, choose, or adapt multiple candidate prompt responses.
Reduce dependence on expert or manual inputs by leveraging automatic prompt generation, iterative self-prompting, or adaptation based on image features.

These design choices address limitations of vanilla SAM, including prompt sensitivity, inability to process complex domains (e.g., 3D images or ambiguous object boundaries), and limited semantic expressivity.

2. Representative Architectures and Methodologies

Automated Prompt Generators and Layered Prompting

AutoProSAM (Li et al., 2023) eliminates manual prompt input for 3D medical segmentation by introducing an Auto Prompt Generator (APG) module. A parameter-efficient, 3D-adapted encoder is coupled with the APG to extract and refine features, directly generating input cues (prompts) for volumetric data. This approach cascades information from hierarchical feature representations, implicitly serving as a chain of prompt refinements, and can be extended to multi-stage frameworks.

GeoSAM (Sultan et al., 2023) combines automatically generated sparse (point) prompts—sampled from task-specific CNN pseudo-labels—with dense prompt embeddings derived from soft masks. The mask serves as a secondary spatial context provider for SAM, with both prompt sets presented in a staged (cascaded) fashion, which has been empirically shown to boost mIoU by at least 5% on mobility infrastructure segmentation.

SAM-SP (Zhou et al., 22 Aug 2024), Self-Prompt-SAM (Xie et al., 2 Feb 2025), and OP-SAM (Mao et al., 22 Jul 2025) implement explicit iterative prompt evolution. In SAM-SP, the model’s own prior output mask is fed back into the prompt encoder as a new prompt, generating a refined mask in the next step. This cascade continues for multiple iterations, optionally incorporating self-distillation terms to improve intermediate outputs. Self-Prompt-SAM incorporates a multi-scale mask generator to automatically identify prompts, which are then sequenced through bounding box and distance-transform point prompt derivation. OP-SAM orchestrates a chain involving semantic prior generation through cross- and self-correlation (CPG), multi-scale prior fusion (SPF), and iterative Euclidean Prompt Evolution (EPE), using segmentation feedback at each round to adapt future prompt queries.

In EVF-SAM (Zhang et al., 28 Jun 2024) and BiPrompt-SAM (Xu et al., 25 Mar 2025), CPS is expressed as simultaneous, parallel provision of image and text prompts, whose outputs are subsequently fused. EVF-SAM employs early vision-language fusion to augment prompt embeddings, improving mask alignment for referring expression tasks. BiPrompt-SAM advances this by explicitly selecting the best spatial mask (from a point-prompted set) that maximally overlaps with a semantically-guided mask (from a text-prompt branch) via Intersection over Union (IoU), a form of decision-level cascading.

SAM-CP (Chen et al., 23 Jul 2024) introduces composable prompts in a cascading query-key framework, where semantic (Type-I) and instance-merging (Type-II) cascades operate jointly to assign meaning and merge SAM's over-segmented regions, with open-vocabulary support via language embeddings from CLIP.

The CPS module in zero-shot anomaly detection (Hou et al., 13 Oct 2025) systematically fuses point, logit, and box prompts through iterative passes with a lightweight decoder. Each round refines previous segmentation outputs; dense logit outputs are re-ingested as dense prompts, and bounding box information is successively added to localize the anomalous regions, yielding a significant improvement of over 10% in $F_1$ -max on the Visa dataset relative to prior methods.

In the OVCOS framework (Zhao et al., 24 Jun 2025), a cascaded architecture leverages a VLM to generate text/vision prompts, which steer the segmentation decoder (SAM) before using segmentation output itself as a spatial prior for downstream open-vocabulary classification—thereby forming a two-stage, semantically consistent prompt cascade.

3. Technical Implementations and Performance Metrics

CPS frameworks commonly involve the following technical building blocks:

Prompt generator modules (often CNNs, Transformer-derived architectures, or learned adapters) that derive cue locations or types.
Fusion and selection mechanisms (e.g., attention, IoU-based gating, or affinity propagation via dynamic cross-attention layers as in SAM-CP).
Iterative or staged mask refinement, either by explicitly chaining prompt decoders or by providing dense outputs (logits or masks) from one stage as prompts to subsequent call(s) to the SAM decoder.
Auxiliary loss terms (e.g., self-distillation, preference optimization) that tie together the cascade and ensure information is blended rather than discarded at each stage.

Performance improvements are consistently reported across domains:

Method	Domain	mIoU/Dice	Notable Improvement
AutoProSAM	3D Medical CT	87.15 (Dice)	+3–6% over MedSAM, etc.
GeoSAM	Geospatial	+5% mIoU	Up to +5% over baseline
CPS (Visa Anomaly)	Industrial	+10.3% $F_1$	Best AP, $F_1$
BiPrompt-SAM	Medical/Natural	81.46/89.55%	SOTA zero-shot
OP-SAM	Polyp	76.93% (IoU)	+11.44% over prior SOTA

These improvements are directly tied to the staged prompt generation, automated selection, or hybrid ingestion pipelines that are characteristic of CPS.

4. Domain-Specific Applications and Adaptations

In medical imaging, CPS enables automation of expert-dependent tasks, multi-organ segmentation, and robust adaptation to 3D and low SNR modalities. For example, AutoProSAM's APG and parameter-efficient 3D adaptation address both domain gap and annotation burden (Li et al., 2023). In weakly supervised landslide or geographic tasks, APSAM and GeoSAM use cascaded multi-modal prompts to compensate for coarse pseudo-labels and improve weakly supervised extraction accuracy (Wang et al., 23 Jan 2025, Sultan et al., 2023).

In open-vocabulary and industrial settings, CPS facilitates the inclusion of language-driven semantics (via CLIP or VLMs) and staged refinement for ambiguous regions (as in OVCOS and anomaly detection), where hybrid prompt types are cascaded to progressively pin down both semantic and spatial uncertainty (Zhao et al., 24 Jun 2025, Hou et al., 13 Oct 2025).

5. Implications for Robustness, Efficiency, and Automation

CPS methods not only improve segmentation accuracy but also contribute to:

Robustness: By decoupling coarse localization from boundary refinement (or by using uncertainty-aware prompt encoders, as in ProSAM (Wang et al., 27 Jun 2025)), cascaded schemes hedge against prompt misplacement and domain shifts.
Efficiency: Approaches such as AoP-SAM (Chen et al., 17 May 2025) integrate lightweight prompt predictors and eliminate redundant or superfluous prompts via adaptive filtering, reducing computational burden relative to grid-based or manual input methods.
Automation: Self-prompting (SAM-SP (Zhou et al., 22 Aug 2024), Self-Prompt-SAM (Xie et al., 2 Feb 2025)) and auto-prompting (AutoProSAM, APSAM) architectures obviate the need for continual expert annotation, making SAM viable in high-throughput and real-time applications across several domains.
Generalizability: By supporting multi-stage, multi-modal input, CPS approaches remain applicable in both closed-set (medical, industrial) and open-vocabulary (panoptic or referring segmentation) settings.

6. Synthesis and Outlook

Cascaded Prompts for SAM bring together automatic, hierarchical, and multi-modal prompt strategies—operating as pipelines where each stage provides increasingly refined, semantically rich, or context-adaptive guidance to SAM's mask decoder. Cross-domain evidence, from 3D medical imaging to geospatial and industrial anomaly detection, demonstrates that these frameworks yield superior segmentation, improved boundary quality, better generalization, and significant annotation cost savings.

Looking forward, further research on CPS will likely emphasize learnable prompt fusion mechanisms, adaptive cascades driven by uncertainty, and collaboration with large-scale multimodal databases—potentially integrating language, spatial, and temporal prompts in a unified, scalable fashion. This trajectory situates CPS as a cornerstone methodology in next-generation, foundation model-based segmentation workflows.