---
title: 'PerSense++: Enhanced Instance Segmentation'
url: https://www.emergentmind.com/topics/persense-198b2e79-307e-4e63-a9d6-68e4ea748293
type: topic
---

# PerSense++: Enhanced Instance Segmentation

Searching arXiv for PerSense++ and closely related papers to ground the article in current sources.
arxiv_search query="PerSense++ personalized instance segmentation dense images" max_results=10 sort_by="relevance"
PerSense++ is an enhanced, fully training-free, model-agnostic framework for personalized instance segmentation in dense images. It extends PerSense by addressing the problem of segmenting all instances of a user-specified concept in cluttered, occluded, densely populated scenes from a single support example, while avoiding manual point or box prompting at test time. Its central mechanism is to generate instance-level point prompts from a density map and use them to drive a segmentation decoder such as SAM; PerSense++ strengthens this pipeline with diversity-aware exemplar selection, a hybrid Instance Detection Module, and an Irrelevant Mask Rejection Module (IMRM) [2508.14660].

## 1. Problem formulation and operating regime

PerSense++ is defined for personalized instance segmentation in dense visual scenes: given a one-shot support image for a target concept or class, the system must segment all instances of that concept in a query image containing many tightly packed objects [2508.14660]. The paper emphasizes dense industrial and medical scenarios such as vegetables on a conveyor belt or cells in microscopy, where standard promptable segmentation methods are vulnerable to occlusion, overlap, background clutter, scale variation, and semantic ambiguity [2508.14660].

The framework is motivated by the observation that box prompts are often too coarse in dense layouts. Adjacent objects can merge within the same box, boundaries are hard to isolate, and irrelevant regions can receive high-confidence masks. PerSense therefore uses density maps to localize likely instance centers and converts those maps into point prompts; PerSense++ keeps that design but reinforces the stages where the original system was fragile: exemplar selection for density-map generation, instance separation within complex density patterns, and rejection of spurious masks [2405.13518; 2508.14660].

The system is explicitly one-shot, end-to-end, training-free, and model-agnostic. “Training-free” means that no task-specific fine-tuning is performed for dense personalized segmentation; “model-agnostic” means that the pipeline is plug-and-play with off-the-shelf components such as GroundingDINO, DSALVANet or CounTR for density-map generation, and SAM or MedSAM for segmentation [2405.13518; 2508.14660].

## 2. Baseline PerSense pipeline

PerSense++ inherits the baseline PerSense pipeline and then augments it. The baseline begins with a support image and a coarse support mask. The masked support image is sent to a class-label extractor with the prompt “Name the object in the image?”, and the extracted noun is used as the target class label [2405.13518]. That label is passed to a grounding detector, typically with the “all + class-name” formulation, to retrieve target-object detections in the query image [2405.13518].

A query-support similarity prior is then computed from encoder features. In the baseline formulation,
$$
S_{\text{score}}(Q, S_{\text{supp}}) = \text{sim}(f(Q), f(S_{\text{supp}})),
$$
and the highest-confidence grounding box \(B_{\max}\) is used to select a positive location prior
$$
P_{\max} = \arg\max_{P \in B_{\max}} S_{\text{score}}(P, S_{\text{supp}}).
$$
The mask produced from this initial step is converted into an exemplar and sent to a density map generator (DMG), such as DSALVANet or CounTR, to produce a density map of likely target-instance locations [2508.14660].

The baseline Instance Detection Module (IDM) is contour-based. It thresholds the grayscale density map into a binary map, applies \(3 \times 3\) morphological erosion, extracts contours, and models contour areas using
$$
A_{\text{ctr}} \sim \mathcal{N}(\mu, \sigma^2),
$$
with composite contours treated as outliers above
$$
T_{\text{comp}} = \mu + 2\sigma.
$$
A distance transform is then used to split merged regions into child contours, and contour centroids become candidate point prompts [2508.14660]. This stage is the core bridge between counting-style density estimation and point-prompt segmentation.

Candidate prompts are filtered by the Point Prompt Selection Module (PPSM). PPSM scores points by query-support similarity and applies an adaptive threshold scaled by estimated object count \(C\):
$$
T_{\text{adapt}} = \frac{S_{\max}}{C/k},
$$
with \(k=\sqrt{2}\) chosen empirically; when \(C=1\), the maximum-score point is selected directly [2508.14660]. The design rationale is that similarity statistics are easily corrupted by background regions, so the threshold is not estimated from score mean and variance. In the original PerSense, a one-pass feedback loop then selects the top \(m\) exemplars from the initial segmentation masks using SAM mask scores, with \(m=4\), and regenerates the density map to improve final prompt quality [2508.14660].

## 3. Architectural extensions introduced by PerSense++

PerSense++ adds three components to the baseline: diversity-aware exemplar selection, a hybrid IDM, and IMRM [2508.14660]. Together they target three specific shortcomings observed in PerSense: redundancy in exemplar selection when only SAM score is used, missed instances inside composite density blobs, and false-positive masks that survive PPSM.

A concise comparison is given below.

| Stage | PerSense | PerSense++ |
|---|---|---|
| Exemplar refinement | Top-\(m\) feedback using SAM mask scores | Diversity-aware exemplar selection using feature diversity, scale diversity, and weighted scoring |
| IDM | Contour-based prompt extraction | Hybrid contour-plus-peak prompt extraction |
| Final mask filtering | PPSM only | PPSM plus IMRM |

### Diversity-aware exemplar selection

The PerSense++ feedback mechanism does not rely on SAM score alone. Candidate exemplars are embedded with a ResNet50 backbone and RoIAlign,
$$
\mathbf{z}_i = \Psi(I, b_i) \in \mathbb{R}^d,
$$
standardized, and clustered with \(k\)-means to promote feature diversity [2508.14660]. SAM-score filtering is still used, but only after clustering, with a fixed threshold \(T_{\text{SAM}} = 0.8\). The paper explicitly notes that SAM scores are not semantically aware: a high-scoring mask may correspond to an irrelevant region, such as a sticker on an eggplant instead of the eggplant itself [2508.14660].

Selection is then driven by a weighted score
$$
\Phi(b_i) = w_1 S_{\text{cent}, i} + w_2 S_{\text{ref}, i} - w_3 \log(\Delta A_i) - w_4 \Delta A_{d, i},
$$
where the terms encode cluster-centrality similarity, reference-exemplar similarity, area deviation, and aspect-ratio deviation [2508.14660]. Scale diversity is enforced by partitioning candidates into small, medium, and large bins using the \(P_{33}\) and \(P_{66}\) percentiles of bounding-box area and selecting the top-scoring exemplar from each bin [2508.14660]. This makes the feedback exemplars more diverse and more representative for DM generation.

### Hybrid IDM

PerSense++ supplements contour-based prompt generation with peak detection. The motivation is that contour splitting can fail inside tightly packed blobs even when local maxima remain visible in the density map [2508.14660]. Peak detection uses the grayscale density map \(G(x,y)\) and thresholds it with
$$
T_{\text{peak}} = \mu_G + \alpha \cdot \sigma_G,
$$
after which local peaks are identified and added to the contour-derived prompts:
$$
\mathcal{Q}_{\text{hybrid}} = \mathcal{Q}_{\text{contour}} \cup \mathcal{Q}_{\text{peak}}.
$$
This union is intended to capture both merged regions that can be separated by contour analysis and dense blobs where only local maxima remain discriminative [2508.14660].

### Irrelevant Mask Rejection Module

IMRM is a post-decoder spatial outlier filter. It operates on mask areas
$$
a_i = \sum_{x,y} m_i(x,y),
$$
computes an IQR-based cutoff
$$
T_{\text{IQR}} = Q_3 + 2 \cdot \text{IQR},
$$
and also estimates a majority-cluster mean and standard deviation, which are combined into a final threshold
$$
T = (\mu_{\text{maj}} + 2 \cdot \sigma_{\text{maj}}) + T_{\text{IQR}}.
$$
A mask is retained iff \(a_i \leq T\) [2508.14660]. Because genuine objects may be large under perspective distortion, a flagged outlier is preserved if it has high IoU with a grounded detection box, e.g. \(\text{IoU} \geq 0.8\) [2508.14660]. The module therefore discards spatially inconsistent masks while preserving geometrically plausible large instances.

## 4. Benchmarking and empirical results

PerSense++ is evaluated on PerSense-D, a benchmark dedicated to personalized segmentation in dense images. The PerSense++ paper reports 717 images across 28 object categories, with 36,837 annotated objects, an average of 53 instances per image, a range from 7 to 573 instances per image, and average resolution \(839 \times 967\) [2508.14660]. The benchmark supports both class-wise and density-based evaluation, with images grouped into Low (\(C_I \leq 30\)), Medium (\(30 < C_I \leq 60\)), and High (\(C_I > 60\)) density bins [2508.14660]. The same paper also reports dense subsets derived from standard benchmarks: COCO-20\(^d\) with 9,184 dense images and LVIS-92\(^d\) with 7,204 dense images [2508.14660].

The headline quantitative result is that PerSense++ improves substantially over PerSense on dense data, especially on PerSense-D. The most salient reported results are summarized below.

| Evaluation setting | Reported performance |
|---|---|
| PerSense-D, DMG1 | 70.96 mIoU \(\rightarrow\) 77.45 mIoU |
| PerSense-D, DMG2 | 71.61 mIoU \(\rightarrow\) 81.35 mIoU |
| COCO-20\(^i\) | +0.7 mIoU over PerSense |
| LVIS-92\(^i\) | +1.2 mIoU over PerSense |
| COCO-20\(^d\) | +2.1 mIoU over PerSense |
| LVIS-92\(^d\) | +2.3 mIoU over PerSense |

These results are accompanied by improved density-map quality. For DMG2 on PerSense-D, MAE drops from 16.17 to 11.48 and RMSE drops from 40.76 to 24.67 under PerSense++ [2508.14660]. The paper highlights that gains are larger on dense subsets than on full benchmark splits, which it interprets as evidence that the new modules specifically help in crowded scenes [2508.14660].

Relative to earlier baselines, the original PerSense already outperformed PerSAM, PerSAM-F, Matcher, and Grounded-SAM on PerSense-D [2405.13518]. The PerSense++ paper retains that context and reports that PerSense achieves 71.61% overall class-wise mIoU on PerSense-D, outperforming PerSAM by +47.16, PerSAM-F by +42.27, SegGPT by +16.11, Matcher by +8.83, and Grounded-SAM by +5.69 [2508.14660]. PerSense++ then extends that margin further.

Ablation results isolate the contribution of the three new components. On PerSense++, diversity-aware exemplar selection contributes +3.80 mIoU, Hybrid IDM contributes +2.72 mIoU, and IMRM contributes +3.22 mIoU [2508.14660]. The paper also reports that performance in PerSense saturates after 4 exemplars and that more feedback iterations do not improve accuracy but increase runtime [2508.14660].

## 5. Failure modes, limitations, and interpretive scope

The principal limitation is explicit: the entire pipeline depends critically on the quality of the generated density map [2508.14660]. If target instances are absent from the DM, then IDM, PPSM, and IMRM cannot recover them later. This failure mode is consistent across the PerSense and PerSense++ descriptions: false positives can be suppressed by prompt filtering and outlier rejection, but missed objects from DM generation are much harder to recover [2405.13518; 2508.14660].

The method is also most effective in dense scenes. On sparse benchmarks, the reported gains are smaller, and the PerSense++ paper states that density maps are less informative there [2508.14660]. This does not mean the method is limited to dense scenes exclusively—PerSense remains competitive on COCO-20\(^i\) and LVIS-92\(^i\)—but the empirical advantage is strongest where crowding provides richer density cues [2508.14660].

A further implication of the design is that PerSense++ improves robustness without introducing task-specific learning. All added modules are deterministic or heuristic post-processing stages: feature extraction and clustering for exemplar selection, thresholding and morphology for Hybrid IDM, and statistical outlier analysis for IMRM [2508.14660]. This suggests a modular alternative to training-based dense segmentation, although the same evidence also indicates that the framework remains bounded by the reliability of its upstream off-the-shelf modules.

## 6. Relation to PerSense and distinction from similarly named systems

PerSense++ belongs to the PerSense line of work on personalized instance segmentation in dense images. The original PerSense introduced the training-free, model-agnostic, one-shot pipeline based on density maps, IDM, PPSM, feedback-based exemplar refinement, and the PerSense-D benchmark [2405.13518]. PerSense++ should therefore be understood not as a separate research direction, but as an architectural extension that hardens the baseline at three identified failure points [2508.14660].

The name should not be conflated with several similarly named systems in unrelated domains. “PerSE” concerns personalized evaluation of open-ended text generation rather than visual segmentation [2310.03304]. “PerfSense” is a lightweight, zero-shot multi-agent LLM framework for identifying performance-sensitive configurations in software systems [2406.12806]. “Perception Programs” (\(P^2\)) address representation of visual tool outputs for multimodal language models and, in the provided source, are explicitly described as not presenting PerSense++ by name [2604.12896]. Within the vision literature, the technically relevant lineage is therefore PerSense \(\rightarrow\) PerSense++ rather than any of these adjacent uses of related nomenclature [2405.13518; 2508.14660].

Source: https://www.emergentmind.com/topics/persense-198b2e79-307e-4e63-a9d6-68e4ea748293