Instance-Specific Prototype Proposal (IPP)

• Instance-Specific Prototype Proposal (IPP) is a design pattern that customizes prototype mechanisms for individual input instances, enabling adaptive representations in diverse tasks.
• Multiple formulations—ranging from distance metrics to prototype mixtures—allow IPP to integrate with architectures for object detection, segmentation, and continual learning.
• IPP methods leverage contrastive, self-supervised, and adaptive losses to achieve performance gains in unknown detection, segmentation quality, and instance classification.

Instance-Specific Prototype Proposal (IPP) denotes a family of mechanisms in which prediction is organized around prototypes that are selected, generated, adapted, or weighted for a particular input instance. The term appears explicitly in Self-supervised Instance-adaptive Prototype Learning (SIPL), where an instance-specific prototype proposal is paired with a common prototype proposal for medical image segmentation, and closely related constructions appear under different names in open-set object detection, new intent discovery, online video instance segmentation, weakly supervised semantic segmentation, referring segmentation, continual graph learning, prototype-based prompt learning, 3D instance segmentation, and 3D panoptic scene completion (Liang et al., 10 Jul 2025, Zhou et al., 2022, Zhang et al., 2024, He et al., 2023, Chen et al., 2022, Liu et al., 2022, 2505.10040, Zhang et al., 2022, Royen et al., 2024, Gross et al., 25 Jun 2025).

1. Definition and conceptual scope

In the most explicit formulation, SIPL defines IPP as a per-input prototype mechanism that complements a Common Prototype Proposal (CPP). CPP captures common visual patterns, whereas IPP is tailored to each input by re-weighting intermediate features according to confidence scores generated hierarchically by a transformer decoder; the method also introduces a self-supervised filtering strategy to prioritize foreground pixels during decoder training (Liang et al., 10 Jul 2025).

Across the broader literature, the same organizing idea often appears without the exact phrase “Instance-Specific Prototype Proposal.” Openset RCNN states that it does not use the phrase explicitly, yet its Prototype Learning Network computes per-proposal distances to class prototypes and uses those instance-specific distance profiles for known/unknown decisions. RAP for new intent discovery likewise uses hard instance-to-prototype assignments, interpolation-based soft prototype targets, and adaptive prototype dispersion. SIPE builds image-specific prototypes from seed pixels for each image, and the referring segmentation framework based on Instance-Specific Feature Propagation treats each grid-anchored feature as an instance-specific representation that is propagated and selected. In this sense, IPP is best understood as a cross-task design pattern rather than a single standardized algorithmic template (Zhou et al., 2022, Zhang et al., 2024, Chen et al., 2022, Liu et al., 2022).

A recurring distinction in these works is between two regimes. One regime keeps prototypes global but makes the compatibility computation instance-specific, as in prototype distances, affinity scores, or prototype mixtures. The other regime generates or updates prototypes directly from the current instance or scene, as in image-specific prototypes, propagated query–proposal pairs, or context-adaptive instance proposals. The surveyed papers suggest that both regimes fall naturally under the IPP umbrella when the decisive computation is conditioned on the particular instance under consideration (Zhou et al., 2022, Zhang et al., 2022, He et al., 2023, Gross et al., 25 Jun 2025).

2. Geometric and mathematical principles

The geometric core of IPP is an instance–prototype interaction law. In Openset RCNN, a proposal feature $\mathbf{f}_i \in \mathbb{R}^{1024}$ is mapped to an embedding $\mathbf{z}_i \in \mathbb{R}^{256}$, compared with class prototypes $\mathbf{P}_j$, and scored by cosine distance

$$D_{ij} = 1 - \frac{\mathbf{z}_i \cdot \mathbf{P}_j}{\|\mathbf{z}_i\|\,\|\mathbf{P}_j\|}.$$

The nearest prototype is $j^\ast=\arg\min_j D_{ij}$, while unknown detection uses the rule $\min_j D_{ij} > T_u$. Here the instance-specific prototype proposal is the full distance vector $\{D_{ij}\}_{j=1}^K$, with rejection implemented as failure to fit any known prototype neighborhood (Zhou et al., 2022).

RAP and IPAL use a related but affinity-based formulation. RAP employs a prototype softmax

$$\mathcal{L}_p = - \sum_{i=1}^{N_y} \log \frac{\exp(z_i \cdot u_{y_i} / T)}{\sum_{k=1}^C \exp(z_i \cdot u_k / T)},$$

so each instance is attracted to its assigned prototype and repelled from the others. IPAL similarly defines instance–prototype affinity by the dot product $\mathcal{F}_\theta(x)^\top \mu_k$ and builds Prototype Contrastive Learning around positive class prototypes and negative prototypes from other classes. These constructions make prototype interaction explicit at the instance level even when prototypes themselves are global class summaries (Zhang et al., 2024, 2505.10040).

A second formulation replaces hard nearest-prototype assignment by prototype mixtures. In PTP, an image prototype bank $\{\mathcal{P}_k\}_{k=1}^K$ and a prompt prototype bank $\mathbf{z}_i \in \mathbb{R}^{256}$0 are coupled by instance-specific weights

$\mathbf{z}_i \in \mathbb{R}^{256}$1

and the final prediction is

$\mathbf{z}_i \in \mathbb{R}^{256}$2

Here IPP is realized as an instance-specific convex weighting over a small dictionary of prompt prototypes rather than as explicit per-instance prototype generation (Zhang et al., 2022).

A third formulation uses shared prototypes as a basis and lets the instance contribute coefficients. In joint prototype and coefficient prediction for 3D instance segmentation, each candidate instance mask is

$\mathbf{z}_i \in \mathbb{R}^{256}$3

where $\mathbf{z}_i \in \mathbb{R}^{256}$4 are shared scene-level prototypes and $\mathbf{z}_i \in \mathbb{R}^{256}$5 are seed-specific coefficients. This is IPP in a linear-combination form: the instance-specific proposal is not a prototype vector alone, but a coefficient-conditioned instantiation of a prototype basis (Royen et al., 2024).

Weakly supervised segmentation introduces a fourth geometric pattern. SIPE computes an image-specific prototype for class $\mathbf{z}_i \in \mathbb{R}^{256}$6 by averaging hierarchical features over seed pixels,

$\mathbf{z}_i \in \mathbb{R}^{256}$7

and then produces an image-specific CAM by cosine similarity between each pixel feature and $\mathbf{z}_i \in \mathbb{R}^{256}$8. This mechanism turns per-image feature geometry into an image-specific prototype field for localization (Chen et al., 2022).

3. Architectural realizations

One major architectural family uses latent prototype spaces for classification or rejection. Openset RCNN couples a classification-free region proposal network with a Prototype Learning Network. The former removes the RPN classifier and defines objectness purely from regression cues, $\mathbf{z}_i \in \mathbb{R}^{256}$9, while the latter imposes a compact prototype geometry for known categories and treats unknowns as the complementary region of all known prototype neighborhoods. This decouples class-agnostic objectness from prototype-based open-set reasoning (Zhou et al., 2022).

A second family uses dynamic or persistent instance slots as prototypes. InsPro propagates an instance query–proposal pair $\mathbf{P}_j$0 through time, where the query is an appearance prototype and the proposal is a spatial hypothesis. The pair is updated by multi-stage segmentation heads, temporally aggregated through a feature bank, and trained with Temporally Consistent Matching so that a fixed query index remains bound to the same object across frames. The result is an online video instance segmentation system in which prototype persistence replaces explicit tracking (He et al., 2023).

IPFormer adopts a scene-conditioned variant of the same principle. Rather than using fixed learned instance queries, it derives context-adaptive instance proposals $\mathbf{P}_j$1 from visible voxel features after deformable cross-attention and self-attention. These proposal embeddings are then refined by cross-attention with voxel features and used both for instance classification and for instance–voxel affinity computation $\mathbf{P}_j$2. In this architecture the prototype is explicitly a proposal slot generated from the observed scene at train and test time (Gross et al., 25 Jun 2025).

A third family is grid-anchored. In referring segmentation, the Instance Extraction Module converts fused vision–language features into an ISF map $\mathbf{P}_j$3, with one instance-specific feature per grid cell. The Feature Propagation Module then propagates information across this grid in four bidirectional directions, and the resulting identifying map selects the target instance. The selected grid index also selects the corresponding mask channel, making the grid feature a prototype, a proposal, and a selector simultaneously (Liu et al., 2022).

A fourth family is prototype generation from image-specific evidence. SIPE first performs structure-aware seed locating via pixelwise structure maps and CAM overlap, then averages hierarchical features inside the selected seed region to form image-specific prototypes. SIPL advances a closely related logic for medical image segmentation by explicitly combining CPP and IPP; the abstract states that the IPP is generated by confidence-guided re-weighting of intermediate features and that the confidence scores are produced hierarchically by a transformer decoder (Chen et al., 2022, Liang et al., 10 Jul 2025).

4. Training objectives and supervisory signals

IPP methods rarely rely on task loss alone. Openset RCNN trains its prototype geometry with a double-margin contrastive loss that enforces intra-class compactness around the positive prototype and inter-class separation from the hardest negative prototype. RAP similarly combines a prototypical contrastive objective with interpolation training, so virtual samples are attracted to mixed prototype targets, and supplements that with Adaptive Prototypical Dispersing Learning, which pushes nearby prototypes apart more strongly through the adaptive term $\mathbf{P}_j$4 (Zhou et al., 2022, Zhang et al., 2024).

In continual graph learning, IPAL uses Prototype Contrastive Learning as the primary feature-shaping loss, but its distinctive contribution is Instance-Prototype Affinity Distillation. Rather than distilling raw features, IPAD matches old and new instance–prototype affinities on synthetic features mixed with class prototypes. The same framework augments prototype replay with Topology-Integrated Gaussian Prototypes, in which prototype means and covariances are weighted by PageRank-derived node importance, and adds Decision Boundary Perception by mining high-entropy boundary instances as extra negatives (2505.10040).

Several systems enforce prototype specialization through structured assignment losses. InsPro uses Hungarian matching on the first frame and propagates that assignment across subsequent frames in a batch, which makes each query index correspond to a consistent object track; it also introduces a Box Deduplication Loss to keep unmatched proposals spatially separated. IPFormer similarly uses Hungarian matching between predicted instance proposals and ground-truth instances, then optimizes cross-entropy, Dice, and focal losses over matched masks. In the 3D prototype–coefficient model, supervision is seed-specific: the mask predicted from a sampled point must match the ground-truth instance that contains that point (He et al., 2023, Gross et al., 25 Jun 2025, Royen et al., 2024).

Self-supervision and regularization are also central. SIPE adds a General-Specific Consistency loss between general CAM and image-specific CAM, allowing classifier-weight CAMs and image-specific prototypes to correct one another. PTP constrains image prototypes by the pair of regularizers $\mathbf{P}_j$5 and $\mathbf{P}_j$6, forcing each prototype to stay near some training example while covering the training data manifold. SIPL further introduces a self-supervised filtering strategy to prioritize foreground pixels when learning confidence maps for IPP generation (Chen et al., 2022, Zhang et al., 2022, Liang et al., 10 Jul 2025).

5. Empirical behavior across application domains

In open-set object detection, the prototype view produces measurable gains on both unknown identification and unknown recall. On VOC-COCO-80 T1, Openset RCNN reports $\mathbf{P}_j$7, compared with $\mathbf{P}_j$8 for OpenDet and $\mathbf{P}_j$9 for Faster R-CNN. On GraspNet-Test-6, it reports $$D_{ij} = 1 - \frac{\mathbf{z}_i \cdot \mathbf{P}_j}{\|\mathbf{z}_i\|\,\|\mathbf{P}_j\|}.$$0, compared with $$D_{ij} = 1 - \frac{\mathbf{z}_i \cdot \mathbf{P}_j}{\|\mathbf{z}_i\|\,\|\mathbf{P}_j\|}.$$1 for OpenDet; the ablation on GraspNet-Test-6 further shows that PLN is the dominant contributor to unknown identification, while CF-RPN improves the proposal distribution (Zhou et al., 2022).

In clustering-oriented settings, prototype structure primarily affects representation quality. RAP reports an average $$D_{ij} = 1 - \frac{\mathbf{z}_i \cdot \mathbf{P}_j}{\|\mathbf{z}_i\|\,\|\mathbf{P}_j\|}.$$2 improvement over current state-of-the-art methods on CLINC, BANKING, and StackOverflow, and its ablations show that removing either the attracting or the dispersing term degrades performance; the paper also reports lower within-cluster distances and larger between-cluster distances than competing methods. IPAL reports superior results on four continual node-classification benchmarks and frames its contribution as a better trade-off between plasticity and stability (Zhang et al., 2024, 2505.10040).

For video and scene understanding, persistent or adaptive prototype proposals substantially improve association quality. InsPro with a ResNet-50 backbone achieves 43.2 AP on YouTube-VIS 2019 and 37.6 AP on YouTube-VIS 2021. Its ablation trajectory on YouTube-VIS 2019 rises from 24.0 AP without propagation or TCM, to 36.3 AP with propagated query–proposal pairs, 37.4 AP with TCM, 38.4 AP with box deduplication, and 40.2 AP with intra-query attention at $$D_{ij} = 1 - \frac{\mathbf{z}_i \cdot \mathbf{P}_j}{\|\mathbf{z}_i\|\,\|\mathbf{P}_j\|}.$$3. IPFormer reports that dynamically deriving instance proposals from image context yields a 3.62% increase in PQ-All and an average improvement of 18.65% in combined Thing-metrics over random initialization, while reducing runtime by more than $$D_{ij} = 1 - \frac{\mathbf{z}_i \cdot \mathbf{P}_j}{\|\mathbf{z}_i\|\,\|\mathbf{P}_j\|}.$$4 relative to prior PSC systems (He et al., 2023, Gross et al., 25 Jun 2025).

In segmentation from weak or indirect supervision, image- or instance-specific prototypes improve completeness. SIPE raises localization mIoU on PASCAL VOC 2012 train from 50.1% for CAM to 53.2% with IPE alone and to 58.6% with IPE plus GSC; with denseCRF, localization reaches 64.7%. In final semantic segmentation, SIPE reports 68.2% mIoU on VOC val and 69.5% on VOC test with a ResNet-38 backbone. In 3D instance segmentation, the prototype–coefficient method reports 32.9% faster inference than the prior state of the art and an over 20-fold reduction in inference-time variance, while achieving the best mRec and mPrec on S3DIS-blocks. In few-shot recognition, PTP shows that prototype-weighted prompting can be competitive with or better than both task-level and instance-level prompt baselines; on Caltech101 with CLIP it reports 91.93 at 1-shot and 96.11 at 16-shot, compared with 89.12 and 93.34 for a single soft prompt (Chen et al., 2022, Royen et al., 2024, Zhang et al., 2022).

6. Limitations, misconceptions, and open directions

A common misconception is that IPP requires an explicitly generated prototype vector for every test instance. The surveyed literature shows a broader reality. Openset RCNN, PTP, RAP, and IPAL all realize instance-specific prototype behavior through distances, affinities, or mixtures over global prototypes, without constructing a separate prototype parameter for each instance. Conversely, InsPro, SIPE, SIPL, and IPFormer do generate per-instance or per-scene prototype-like entities directly from current observations. This suggests that IPP is better defined by instance-conditioned prototype interaction than by any single implementation detail (Zhou et al., 2022, Zhang et al., 2024, 2505.10040, Zhang et al., 2022, He et al., 2023, Chen et al., 2022, Liang et al., 10 Jul 2025, Gross et al., 25 Jun 2025).

Several technical limitations recur. Openset RCNN notes threshold sensitivity in the unknown decision rule and emphasizes that incomplete annotation biases unknown-object evaluation, motivating its GraspNet reorganization. RAP identifies pseudo-label reliability from k-means initialization as a core challenge. PTP is limited by the semantics available in class names and by the latent knowledge already present in the pretrained vision–LLM, with FGVCAircraft highlighted as a difficult case. InsPro reports difficulty with small objects and finite temporal memory. IPFormer is fast at inference but memory-intensive in training, and the 3D prototype–coefficient model remains block-based with fixed prototype and seed counts (Zhou et al., 2022, Zhang et al., 2024, Zhang et al., 2022, He et al., 2023, Gross et al., 25 Jun 2025, Royen et al., 2024).

The literature also points to converging future directions. Several papers suggest moving beyond single-prototype-per-class assumptions toward richer local or multi-prototype structure; RAP discusses prototype multiplicity per intent, IPAL motivates instance-aware refinements under topology and drift, and SIPL explicitly combines global and instance-adaptive prototypes. Other suggested directions include adaptive thresholding and better unknown modeling in open-set detection, hierarchical or multimodal proposal generation in 3D scene completion, and confidence-weighted or split–merge prototype evolution in continual learning and intent discovery. A plausible implication is that future IPP research will be organized less around whether prototypes are “global” or “instance-specific,” and more around how prototype geometry, proposal generation, assignment uncertainty, and task supervision are coupled in a single trainable system (Liang et al., 10 Jul 2025, Zhang et al., 2024, 2505.10040, Gross et al., 25 Jun 2025, Zhou et al., 2022).