PerSense++: Enhanced Instance Segmentation

• The paper demonstrates that PerSense++ enhances instance segmentation by introducing diversity-aware exemplar selection, a hybrid IDM, and an Irrelevant Mask Rejection Module (IMRM).
• It employs a density-map-driven approach to generate instance-level point prompts, effectively handling cluttered, occluded, and densely populated scenes.
• Empirical results show notable mIoU improvements—with gains up to +2.3 mIoU—and reduced MAE and RMSE on challenging dense image benchmarks.

Searching arXiv for PerSense++ and closely related papers to ground the article in current sources. arxiv_search query="PerSense++ personalized instance segmentation dense images" max_results=10 sort_by="relevance" PerSense++ is an enhanced, fully training-free, model-agnostic framework for personalized instance segmentation in dense images. It extends PerSense by addressing the problem of segmenting all instances of a user-specified concept in cluttered, occluded, densely populated scenes from a single support example, while avoiding manual point or box prompting at test time. Its central mechanism is to generate instance-level point prompts from a density map and use them to drive a segmentation decoder such as SAM; PerSense++ strengthens this pipeline with diversity-aware exemplar selection, a hybrid Instance Detection Module, and an Irrelevant Mask Rejection Module (IMRM) (Siddiqui et al., 20 Aug 2025).

1. Problem formulation and operating regime

PerSense++ is defined for personalized instance segmentation in dense visual scenes: given a one-shot support image for a target concept or class, the system must segment all instances of that concept in a query image containing many tightly packed objects (Siddiqui et al., 20 Aug 2025). The paper emphasizes dense industrial and medical scenarios such as vegetables on a conveyor belt or cells in microscopy, where standard promptable segmentation methods are vulnerable to occlusion, overlap, background clutter, scale variation, and semantic ambiguity (Siddiqui et al., 20 Aug 2025).

The framework is motivated by the observation that box prompts are often too coarse in dense layouts. Adjacent objects can merge within the same box, boundaries are hard to isolate, and irrelevant regions can receive high-confidence masks. PerSense therefore uses density maps to localize likely instance centers and converts those maps into point prompts; PerSense++ keeps that design but reinforces the stages where the original system was fragile: exemplar selection for density-map generation, instance separation within complex density patterns, and rejection of spurious masks (Siddiqui et al., 2024, Siddiqui et al., 20 Aug 2025).

The system is explicitly one-shot, end-to-end, training-free, and model-agnostic. “Training-free” means that no task-specific fine-tuning is performed for dense personalized segmentation; “model-agnostic” means that the pipeline is plug-and-play with off-the-shelf components such as GroundingDINO, DSALVANet or CounTR for density-map generation, and SAM or MedSAM for segmentation (Siddiqui et al., 2024, Siddiqui et al., 20 Aug 2025).

2. Baseline PerSense pipeline

PerSense++ inherits the baseline PerSense pipeline and then augments it. The baseline begins with a support image and a coarse support mask. The masked support image is sent to a class-label extractor with the prompt “Name the object in the image?”, and the extracted noun is used as the target class label (Siddiqui et al., 2024). That label is passed to a grounding detector, typically with the “all + class-name” formulation, to retrieve target-object detections in the query image (Siddiqui et al., 2024).

A query-support similarity prior is then computed from encoder features. In the baseline formulation,

$$S_{\text{score}}(Q, S_{\text{supp}}) = \text{sim}(f(Q), f(S_{\text{supp}})),$$

and the highest-confidence grounding box $B_{\max}$ is used to select a positive location prior

$$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$

The mask produced from this initial step is converted into an exemplar and sent to a density map generator (DMG), such as DSALVANet or CounTR, to produce a density map of likely target-instance locations (Siddiqui et al., 20 Aug 2025).

The baseline Instance Detection Module (IDM) is contour-based. It thresholds the grayscale density map into a binary map, applies $3 \times 3$ morphological erosion, extracts contours, and models contour areas using

$A_{\text{ctr}} \sim \mathcal{N}(\mu, \sigma^2),$

with composite contours treated as outliers above

$T_{\text{comp}} = \mu + 2\sigma.$

A distance transform is then used to split merged regions into child contours, and contour centroids become candidate point prompts (Siddiqui et al., 20 Aug 2025). This stage is the core bridge between counting-style density estimation and point-prompt segmentation.

Candidate prompts are filtered by the Point Prompt Selection Module (PPSM). PPSM scores points by query-support similarity and applies an adaptive threshold scaled by estimated object count $C$:

$T_{\text{adapt}} = \frac{S_{\max}}{C/k},$

with $k=\sqrt{2}$ chosen empirically; when $C=1$, the maximum-score point is selected directly (Siddiqui et al., 20 Aug 2025). The design rationale is that similarity statistics are easily corrupted by background regions, so the threshold is not estimated from score mean and variance. In the original PerSense, a one-pass feedback loop then selects the top $B_{\max}$0 exemplars from the initial segmentation masks using SAM mask scores, with $B_{\max}$1, and regenerates the density map to improve final prompt quality (Siddiqui et al., 20 Aug 2025).

3. Architectural extensions introduced by PerSense++

PerSense++ adds three components to the baseline: diversity-aware exemplar selection, a hybrid IDM, and IMRM (Siddiqui et al., 20 Aug 2025). Together they target three specific shortcomings observed in PerSense: redundancy in exemplar selection when only SAM score is used, missed instances inside composite density blobs, and false-positive masks that survive PPSM.

A concise comparison is given below.

Stage	PerSense	PerSense++
Exemplar refinement	Top-$B_{\max}$2 feedback using SAM mask scores	Diversity-aware exemplar selection using feature diversity, scale diversity, and weighted scoring
IDM	Contour-based prompt extraction	Hybrid contour-plus-peak prompt extraction
Final mask filtering	PPSM only	PPSM plus IMRM

Diversity-aware exemplar selection

The PerSense++ feedback mechanism does not rely on SAM score alone. Candidate exemplars are embedded with a ResNet50 backbone and RoIAlign,

$B_{\max}$3

standardized, and clustered with $B_{\max}$4-means to promote feature diversity (Siddiqui et al., 20 Aug 2025). SAM-score filtering is still used, but only after clustering, with a fixed threshold $B_{\max}$5. The paper explicitly notes that SAM scores are not semantically aware: a high-scoring mask may correspond to an irrelevant region, such as a sticker on an eggplant instead of the eggplant itself (Siddiqui et al., 20 Aug 2025).

Selection is then driven by a weighted score

$B_{\max}$6

where the terms encode cluster-centrality similarity, reference-exemplar similarity, area deviation, and aspect-ratio deviation (Siddiqui et al., 20 Aug 2025). Scale diversity is enforced by partitioning candidates into small, medium, and large bins using the $B_{\max}$7 and $B_{\max}$8 percentiles of bounding-box area and selecting the top-scoring exemplar from each bin (Siddiqui et al., 20 Aug 2025). This makes the feedback exemplars more diverse and more representative for DM generation.

Hybrid IDM

PerSense++ supplements contour-based prompt generation with peak detection. The motivation is that contour splitting can fail inside tightly packed blobs even when local maxima remain visible in the density map (Siddiqui et al., 20 Aug 2025). Peak detection uses the grayscale density map $B_{\max}$9 and thresholds it with

$$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$0

after which local peaks are identified and added to the contour-derived prompts:

$$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$1

This union is intended to capture both merged regions that can be separated by contour analysis and dense blobs where only local maxima remain discriminative (Siddiqui et al., 20 Aug 2025).

Irrelevant Mask Rejection Module

IMRM is a post-decoder spatial outlier filter. It operates on mask areas

$$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$2

computes an IQR-based cutoff

$$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$3

and also estimates a majority-cluster mean and standard deviation, which are combined into a final threshold

$$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$4

A mask is retained iff $$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$5 (Siddiqui et al., 20 Aug 2025). Because genuine objects may be large under perspective distortion, a flagged outlier is preserved if it has high IoU with a grounded detection box, e.g. $$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$6 (Siddiqui et al., 20 Aug 2025). The module therefore discards spatially inconsistent masks while preserving geometrically plausible large instances.

4. Benchmarking and empirical results

PerSense++ is evaluated on PerSense-D, a benchmark dedicated to personalized segmentation in dense images. The PerSense++ paper reports 717 images across 28 object categories, with 36,837 annotated objects, an average of 53 instances per image, a range from 7 to 573 instances per image, and average resolution $$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$7 (Siddiqui et al., 20 Aug 2025). The benchmark supports both class-wise and density-based evaluation, with images grouped into Low ($$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$8), Medium ($$P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).$$9), and High ($3 \times 3$0) density bins (Siddiqui et al., 20 Aug 2025). The same paper also reports dense subsets derived from standard benchmarks: COCO-20$3 \times 3$1 with 9,184 dense images and LVIS-92$3 \times 3$2 with 7,204 dense images (Siddiqui et al., 20 Aug 2025).

The headline quantitative result is that PerSense++ improves substantially over PerSense on dense data, especially on PerSense-D. The most salient reported results are summarized below.

Evaluation setting	Reported performance
PerSense-D, DMG1	70.96 mIoU $3 \times 3$3 77.45 mIoU
PerSense-D, DMG2	71.61 mIoU $3 \times 3$4 81.35 mIoU
COCO-20$3 \times 3$5	+0.7 mIoU over PerSense
LVIS-92$3 \times 3$6	+1.2 mIoU over PerSense
COCO-20$3 \times 3$7	+2.1 mIoU over PerSense
LVIS-92$3 \times 3$8	+2.3 mIoU over PerSense

These results are accompanied by improved density-map quality. For DMG2 on PerSense-D, MAE drops from 16.17 to 11.48 and RMSE drops from 40.76 to 24.67 under PerSense++ (Siddiqui et al., 20 Aug 2025). The paper highlights that gains are larger on dense subsets than on full benchmark splits, which it interprets as evidence that the new modules specifically help in crowded scenes (Siddiqui et al., 20 Aug 2025).

Relative to earlier baselines, the original PerSense already outperformed PerSAM, PerSAM-F, Matcher, and Grounded-SAM on PerSense-D (Siddiqui et al., 2024). The PerSense++ paper retains that context and reports that PerSense achieves 71.61% overall class-wise mIoU on PerSense-D, outperforming PerSAM by +47.16, PerSAM-F by +42.27, SegGPT by +16.11, Matcher by +8.83, and Grounded-SAM by +5.69 (Siddiqui et al., 20 Aug 2025). PerSense++ then extends that margin further.

Ablation results isolate the contribution of the three new components. On PerSense++, diversity-aware exemplar selection contributes +3.80 mIoU, Hybrid IDM contributes +2.72 mIoU, and IMRM contributes +3.22 mIoU (Siddiqui et al., 20 Aug 2025). The paper also reports that performance in PerSense saturates after 4 exemplars and that more feedback iterations do not improve accuracy but increase runtime (Siddiqui et al., 20 Aug 2025).

5. Failure modes, limitations, and interpretive scope

The principal limitation is explicit: the entire pipeline depends critically on the quality of the generated density map (Siddiqui et al., 20 Aug 2025). If target instances are absent from the DM, then IDM, PPSM, and IMRM cannot recover them later. This failure mode is consistent across the PerSense and PerSense++ descriptions: false positives can be suppressed by prompt filtering and outlier rejection, but missed objects from DM generation are much harder to recover (Siddiqui et al., 2024, Siddiqui et al., 20 Aug 2025).

The method is also most effective in dense scenes. On sparse benchmarks, the reported gains are smaller, and the PerSense++ paper states that density maps are less informative there (Siddiqui et al., 20 Aug 2025). This does not mean the method is limited to dense scenes exclusively—PerSense remains competitive on COCO-20$3 \times 3$9 and LVIS-92$A_{\text{ctr}} \sim \mathcal{N}(\mu, \sigma^2),$0—but the empirical advantage is strongest where crowding provides richer density cues (Siddiqui et al., 20 Aug 2025).

A further implication of the design is that PerSense++ improves robustness without introducing task-specific learning. All added modules are deterministic or heuristic post-processing stages: feature extraction and clustering for exemplar selection, thresholding and morphology for Hybrid IDM, and statistical outlier analysis for IMRM (Siddiqui et al., 20 Aug 2025). This suggests a modular alternative to training-based dense segmentation, although the same evidence also indicates that the framework remains bounded by the reliability of its upstream off-the-shelf modules.

6. Relation to PerSense and distinction from similarly named systems

PerSense++ belongs to the PerSense line of work on personalized instance segmentation in dense images. The original PerSense introduced the training-free, model-agnostic, one-shot pipeline based on density maps, IDM, PPSM, feedback-based exemplar refinement, and the PerSense-D benchmark (Siddiqui et al., 2024). PerSense++ should therefore be understood not as a separate research direction, but as an architectural extension that hardens the baseline at three identified failure points (Siddiqui et al., 20 Aug 2025).

The name should not be conflated with several similarly named systems in unrelated domains. “PerSE” concerns personalized evaluation of open-ended text generation rather than visual segmentation (Wang et al., 2023). “PerfSense” is a lightweight, zero-shot multi-agent LLM framework for identifying performance-sensitive configurations in software systems (Wang et al., 2024). “Perception Programs” ($A_{\text{ctr}} \sim \mathcal{N}(\mu, \sigma^2),$1) address representation of visual tool outputs for multimodal LLMs and, in the provided source, are explicitly described as not presenting PerSense++ by name (Janjua et al., 14 Apr 2026). Within the vision literature, the technically relevant lineage is therefore PerSense $A_{\text{ctr}} \sim \mathcal{N}(\mu, \sigma^2),$2 PerSense++ rather than any of these adjacent uses of related nomenclature (Siddiqui et al., 2024, Siddiqui et al., 20 Aug 2025).