Point Prompt Defender for SAM

• Point Prompt Defender is an adversarial RL-based framework that refines point prompts for SAM to substantially improve segmentation quality.
• It employs a two-agent attack-for-defense paradigm where an attacker degrades segmentation while a defender suppresses harmful prompts using Dice feedback.
• The plug-and-play, task-agnostic method utilizes dual-space graphs and DQNs to optimize prompt configurations without retraining SAM.

Searching arXiv for the cited work to ground the article in current papers. Point Prompt Defender (PPD) is an adversarial, reinforcement-learning-based framework for automatically optimizing point prompts for the Segment Anything Model (SAM). It is formulated around the observation that SAM’s segmentation quality is highly sensitive to the quality, location, and composition of point prompts, while many existing automatic prompt strategies remain heuristic, static, task-specific, or supervised. PPD addresses this by introducing an attack-for-defense paradigm in which an attacker agent learns to activate prompts that degrade SAM’s segmentation and a defender agent learns to suppress harmful prompts and restore or improve accuracy. The framework is task-agnostic, plug-and-play, and training-free with respect to SAM itself; during inference, only the defender is deployed to refine arbitrary coarse prompt sets without retraining SAM (Liu et al., 23 Sep 2025).

1. Problem setting and conceptual scope

SAM formulates segmentation as “image + prompt $\rightarrow$ mask,” and its performance depends strongly on the prompt set supplied to the model. Good positive points lie inside the target object and avoid boundaries and ambiguous regions, while good negative points lie outside the object and suppress distractors. Poor points, including points on the wrong side of boundaries or on confusing textures, can seriously degrade IoU and Dice. The main research goal of PPD is to design a task-agnostic, RL-based mechanism that can automatically refine arbitrary point prompt sets to improve SAM’s segmentation by using SAM’s own segmentation feedback and without retraining SAM (Liu et al., 23 Sep 2025).

The motivation follows from limits of prior automatic prompt strategies. The reported baselines include uniform or grid sampling inside or outside coarse masks, geometry-based sampling on edges, centroids, or salient regions, and feature matching or pseudo-label-derived points. These strategies are described as static, because they do not adapt to SAM’s output or to local segmentation failures; task-specific or supervised, because many require finetuning on labeled downstream data; and sensitive to initial prompts, because poor or noisy initialization can collapse performance. This constrains scalability to new domains, noisy or low-quality initial prompts, and training-free deployment scenarios (Liu et al., 23 Sep 2025).

Within this usage, “Point Prompt Defender” names a segmentation-time prompt optimizer, not a security classifier. A plausible implication is that the term is local to this line of work rather than a standardized label shared across all prompt-defense literatures.

2. Attack-for-defense formulation

PPD uses a two-agent adversarial reinforcement learning scheme. The attacker agent learns to activate prompts that maximally degrade SAM’s segmentation quality, and the defender agent learns to deactivate harmful prompts and restore or improve segmentation quality. The framework is explicitly described as an attack-for-defense paradigm: training a strong adversary creates difficult prompt configurations, and the defender learns robust strategies that generalize to noisy prompts encountered at test time (Liu et al., 23 Sep 2025).

The training pipeline has two phases. In the environment construction stage, an input image $X$ and its ground-truth mask $M$ are used to build a dual-space graph from image patches, and ideal prompts are generated from $M$ by sampling positive points inside the mask and negative points outside it. In the attack phase, the attacker starts from the ideal prompt pool $P_i$, activates a subset of inactive prompts, runs SAM with the new active set, receives a reward from the decrease in Dice score, and updates its Deep Q-Network. In the defense phase, the defender receives the attacker’s final prompt configuration $P_{\text{att}}$, deactivates a subset of active prompts, runs SAM with refined prompts, receives a reward from the increase in Dice score, and updates its own DQN (Liu et al., 23 Sep 2025).

Inference differs sharply from training. Only the defender is used, and the attacker, ideal prompts, and ground-truth masks are not needed. The defender takes any coarse or noisy prompt set, including prompts from heuristic feature matching, coarse segmentation or detection, manual user clicks, or other sources, and selects which active prompts to keep or deactivate before the refined set is fed into SAM. SAM itself remains unchanged, so PPD functions as a plug-and-play prompt refinement module rather than a modified segmentation backbone (Liu et al., 23 Sep 2025).

3. Dual-space graph environment

The reinforcement-learning environment is a dual-space heterogeneous graph that encodes both semantic and spatial relations among image patches or prompt locations. Given an image $X$, PPD partitions it into patches

$x = \{x_1, x_2, \ldots, x_n\},$

and each patch $x_i$ is encoded by DINOv2 into a feature vector

$$f_i = \text{DINOv2}(x_i), \quad f = \{f_1, f_2, \ldots, f_n\}.$$

These features provide the semantic representations used to construct the graph (Liu et al., 23 Sep 2025).

Two distance matrices are then defined. The semantic, or feature, distance is

$X$0

and the physical, or spatial, distance is

$X$1

where $X$2 denotes the geometric center of patch $X$3. The resulting graph is

$X$4

with $X$5 as nodes, $X$6 as edges weighted by semantic similarity, and $X$7 as edges weighted by physical proximity (Liu et al., 23 Sep 2025).

The graph is prompt-aware. Ideal positive and negative prompts are sampled from the ground-truth mask $X$8: points inside $X$9 are positive, points outside $M$0 are negative, and uniform sampling at fixed intervals ensures broad coverage. Prompts are attached to graph nodes, and node features may include the DINOv2 feature $M$1, prompt label, and activation status. This gives the agents a structured state rather than a set of isolated pixels, so actions can be conditioned on both local prompt attributes and their semantic or spatial neighborhoods (Liu et al., 23 Sep 2025).

4. Reinforcement-learning formulation and optimization

PPD is presented as an MDP in which prompt activation status is the controllable variable. The state includes the current dual-space graph $M$2, the prompt pool $M$3, and for each prompt $M$4, its type, associated patch, and status. The attacker’s action space at time $M$5 is the set of inactive prompts,

$M$6

and the defender’s action space is the set of active prompts,

$M$7

The attacker activates prompts; the defender deactivates prompts (Liu et al., 23 Sep 2025).

Segmentation quality is measured by the Dice coefficient,

$M$8

with $M$9 and $M$0. Let $M$1 and $M$2. The attacker reward is

$M$3

so harmful actions that reduce Dice receive positive reward. The defender reward is

$M$4

so prompt suppression is rewarded when segmentation quality improves (Liu et al., 23 Sep 2025).

Both agents are trained with DQNs. Each maintains a Q-network $M$5 and a target network $M$6, and optimization uses the standard temporal-difference loss

$M$7

The reported training details are: Adam optimizer with learning rate $M$8, batch size $M$9, $P_i$0 episodes, per-episode environment steps randomly sampled between $P_i$1 and $P_i$2, target network updates every $P_i$3 environment steps, and an $P_i$4-greedy exploration policy with $P_i$5 annealed linearly from $P_i$6 to $P_i$7 (Liu et al., 23 Sep 2025).

A common misconception is that PPD is a new SAM training procedure. It is not. The defender policy is trained against segmentation-quality feedback, but SAM remains exactly as originally pretrained. Another misconception is that the defender generates prompts from scratch. The paper instead describes it as a refinement mechanism that suppresses or reconfigures prompts, and notes that extremely poor cold-start prompt sets may still yield poor segmentation (Liu et al., 23 Sep 2025).

5. Empirical behavior across domains

PPD is trained on $P_i$8 diverse images from FSS-1000 and evaluated without retraining on PASCAL VOC, ISIC, and Kvasir. The top part of the main results establishes the attack-for-defense effect under ideal prompts derived from ground truth: the attacker sharply degrades performance, and the defender recovers most of the loss. For example, with ideal prompts, VOC reaches mDSC $P_i$9 and mIoU $P_{\text{att}}$0; after attack, VOC drops to mDSC $P_{\text{att}}$1 and mIoU $P_{\text{att}}$2; after defense, VOC recovers to mDSC $P_{\text{att}}$3 and mIoU $P_{\text{att}}$4. Analogous recoveries are reported for ISIC and Kvasir (Liu et al., 23 Sep 2025).

The bottom part of the same table evaluates training-free initial prompts from feature matching. Here PPD functions as a task-agnostic refinement stage rather than an oracle recovery mechanism.

Setting	Dataset	mDSC / mIoU
Feature matching	VOC	41.3 / 34.8
Feature matching + PPD	VOC	69.1 / 60.3
Feature matching	ISIC	66.4 / 55.0
Feature matching + PPD	ISIC	76.3 / 64.2
Feature matching	Kvasir	31.4 / 21.5
Feature matching + PPD	Kvasir	54.8 / 44.9

These gains are especially large on VOC and Kvasir, where mDSC improves by $P_{\text{att}}$5 and $P_{\text{att}}$6, respectively, and mIoU improves by $P_{\text{att}}$7 and $P_{\text{att}}$8. The same section reports that PPD is evaluated without retraining on both natural scenes and medical images, which is the basis for its task-agnostic characterization (Liu et al., 23 Sep 2025).

The comparison with prior SAM-based one-shot segmentation methods further situates the method. On PASCAL VOC, FM-PPD reports $P_{\text{att}}$9 and achieves the best mDSC while slightly trailing Matcher in mIoU. On ISIC, FM-PPD reports $X$0 and is the best method. On Kvasir, FM-PPD reaches $X$1, substantially above Matcher, GBMSeg, and FM-PPO. The paper attributes the difference from FM-PPO to the attack-for-defense design and to rewards based directly on SAM’s segmentation feedback rather than only on inter-prompt relations (Liu et al., 23 Sep 2025).

6. Interpretability, limitations, and related usages of the term

PPD is described as interpretable because its actions correspond directly to which prompts are kept or suppressed. Visualizations of active and deactivated prompts show that harmful points near confusing backgrounds or object boundaries are often turned off, while helpful points in core object regions are retained. It is described as flexible because it does not modify SAM and can sit on top of feature matching, heuristic sampling, one-shot schemes, or manual interactive points. The same section suggests conceptual compatibility with other prompt-based models such as SAM2 and SegGPT, wherever point prompts govern segmentation output (Liu et al., 23 Sep 2025).

Its assumptions and limits are explicit. The defender requires reasonably informative initial prompts, since it refines prompts rather than creating them from scratch. It relies on DINOv2 features to construct a general-purpose graph; if those features are poor in a niche domain, the graph may be less informative. Training requires ground-truth masks because Dice-based rewards are computed against $X$2. The paper also notes sensitivity to very poor cold-start prompt sets and a computational burden arising from two agents, DQN training, DINOv2 feature extraction, and repeated SAM calls (Liu et al., 23 Sep 2025).

The phrase “prompt defender” appears in several adjacent literatures, but with different referents. In LLM security, it can denote prompt injection detectors or prompt-layer safety mechanisms, including a local multiagent classifier in CourtGuard (Wu et al., 20 Oct 2025), suffix-based jailbreak defense in Defensive Prompt Patch (Xiong et al., 2024), deployable prompt injection detection in PromptShield (Jacob et al., 25 Jan 2025), unified detection of prompt injection, backdoor, and adversarial attacks in UniGuardian (Lin et al., 18 Feb 2025), early-exit jailbreak defense in EEG-Defender (Zhao et al., 2024), and agent-facing sanitization in PromptArmor (Shi et al., 21 Jul 2025). In 3D robustness, related “defender” language appears in point-cloud defenses such as Defense-PointNet (Zhang et al., 2020), DUP-Net (Zhou et al., 2018), PointDP (Sun et al., 2022), and structured declarative classifiers such as LPC (Li et al., 2022). A plausible implication is that PPD’s distinctiveness lies less in the word “defender” than in its specific combination of SAM prompt optimization, dual-space graph modeling, and adversarial RL.