Instance-Adaptive Prototypes Overview
- Instance-adaptive prototypes are dynamic representations that adjust based on query or instance inputs, contrasting with static class centroids.
- They employ mechanisms like query-conditioned transformation, online revision, and adaptive scaling to improve performance in segmentation and detection tasks.
- These methods address fixed prototype limitations by refining representations in real time, as evidenced by significant improvements in mIoU and other metrics.
Searching arXiv for papers on instance-adaptive prototypes and closely related prototype adaptation methods. arXiv search query: "instance-adaptive prototype OR prototype adaptation few-shot segmentation point cloud semantic segmentation cross-domain generalized zero-shot prototype replay"
Instance-adaptive prototypes are prototype representations whose content, weighting, or geometry is conditioned on the current instance, query, episode, or evolving feature space rather than remaining fixed class centroids. Across recent arXiv literature, the term covers several closely related constructions: query-specific prototypes for few-shot point cloud segmentation, query-derived class-agnostic prototypes for few-shot semantic segmentation, query-aware support prototypes for few-shot object detection, prompt-derived virtual semantic prototypes for sentence embedding, online adaptive hyperspherical prototypes for novel category discovery, region-bound adaptive binding prototypes for generalized referring expression segmentation, and representation-shift-aware prototype replay for continual segmentation (Liu et al., 2024, Chen et al., 2021, Lu et al., 2022, Zeng et al., 2022, Zhang et al., 2022, Li et al., 2024, Zhu et al., 2024). The unifying idea is that a prototype is not treated as immutable: it is re-estimated, refined, selected, or reweighted so that it better matches the current input and the current optimization regime.
1. Conceptual scope
Static prototype methods typically assume that a support-derived average, a learned class anchor, or a stored memory item is sufficiently representative for all future instances. The recent literature repeatedly identifies failure modes of that assumption. In few-shot point cloud segmentation, static support prototypes can fail when there is a significant distribution shift between support and query features, especially when objects of the same class vary in scale, appearance, and feature statistics (Liu et al., 2024). In few-shot semantic segmentation, single foreground prototypes can produce biased classification because training treats many novel-class objects as background, which yields incomplete feature comparisons (Chen et al., 2021). In cross-domain few-shot classification, a frozen-backbone pipeline that applies the same transformation to image embeddings and class prototypes is reported to ignore a natural prototype–image gap (Tian et al., 2024). In class incremental semantic segmentation, replay with fixed prototypes is reported to mismatch the continuously updated representation of old classes (Zhu et al., 2024).
A compact way to organize the area is to distinguish what exactly is being adapted.
| Setting | Adaptive object | Representative formulation |
|---|---|---|
| Few-shot segmentation | Query-specific or region-specific prototypes | DPA, APANet, MABP (Liu et al., 2024, Chen et al., 2021, Li et al., 2024) |
| Few-shot detection and classification | Query-aware support prototypes or separate prototype/image transforms | ICPE, CoPA, APPL (Lu et al., 2022, Tian et al., 2024, Heidari et al., 2023) |
| Unsupervised and zero-shot representation learning | Per-instance semantic anchors or online updated class anchors | ConPVP, adaptive prototype learning, PCE-GZSL (Zeng et al., 2022, Zhang et al., 2022, Paul et al., 2023) |
| Continual learning and memory systems | Replayed, compensated, or memorized prototypes | Adapter, MPNet (Zhu et al., 2024, He et al., 2020) |
| Interpretability and case-based reasoning | Instance-specific informative overlaps with prototypes | “alike parts” (Karolczak et al., 8 May 2025) |
This suggests that “instance-adaptive” is not a single algorithmic template. In the literature, it denotes at least four regimes: query-conditioned prototype transformation, query-derived prototype construction, online prototype revision from incoming instances, and explanation-time identification of the prototype parts most relevant to a specific case.
2. Core mechanisms of adaptation
A common mechanism is explicit prototype transformation conditioned on the current query. In Dynamic Prototype Adaptation, a vanilla support prototype is rectified by a query-conditioned transformation,
with
followed by prototype-to-query attention,
and residual FFN refinement
The resulting task-specific prototypes are then used in non-parametric segmentation through a distance-based softmax with , while a distillation term
encourages consistency between early and deeper adaptation stages (Liu et al., 2024).
A second mechanism is query-derived prototype construction. APANet introduces a class-specific branch and a class-agnostic branch. The support foreground prototype is obtained by mask average pooling,
$\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$
while query-derived class-agnostic prototypes are constructed by k-means on high-level query features and then pooled over the induced regions,
$\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$
The two branches share the same comparison function , and optimization uses
0
so that foreground and background both participate in the metric (Chen et al., 2021).
A third mechanism is query-aware support enhancement before prototype aggregation. In Information-Coupled Prototype Elaboration, a conditional information coupling module computes
1
builds a coupling mask
2
and injects the coupled signal via
3
A subsequent prototype dynamic aggregation module computes image-specific prototypes
4
with inter-image fusion
5
producing prototypes tailored to each query image (Lu et al., 2022).
A fourth mechanism replaces geometric averaging with semantic prompting or adaptive scaling. ConPVP assigns each sentence its own prompt-derived positive and negative semantic prototypes,
6
and optimizes an InfoNCE-style prototypical loss that attracts 7 to 8 while repelling it from 9 and the prototypes of other sentences (Zeng et al., 2022). In generalized zero-shot learning, PCE-GZSL instead uses learnable class prototypes as anchors in projection space and modulates pairwise hardness through instance-adaptive scales
0
inside a margin-based prototypical contrastive loss (Paul et al., 2023).
3. Segmentation and detection
Segmentation is the domain in which instance-adaptive prototypes have been articulated most explicitly. In few-shot point cloud segmentation, DPA keeps the prototypical classifier non-parametric but modifies the prototype itself so that support prototypes are aligned to query geometry and semantics. On S3DIS, its mean IoU reaches 1 in the 2-way 3-shot setting versus QGPA 4; on ScanNet, the corresponding numbers are 5 versus 6, yielding the reported gains of 7 and 8 (Liu et al., 2024).
In few-shot semantic segmentation, APANet addresses a different failure mode: incomplete feature comparisons. Its adaptive component is primarily the query-conditioned class-agnostic branch, which generates multiple background prototypes per query image via k-means over high-level query features. The reported best mean mIoU occurs at 9 class-agnostic prototypes on PASCAL-50, and the loss weight 1 is reported as optimal, whereas 2 collapses performance to 3 because training then uses only query-query comparisons without support guidance (Chen et al., 2021).
Generalized referring expression segmentation pushes the idea further by binding queries to spatial regions rather than to global classes. MABP treats queries as region-bound prototypes: each patch is supervised only by its corresponding query, and the mixed-modal decoder updates queries using
4
Its Regional Supervision Head makes different queries specialize to different instances, different categories, or different parts of the same instance. On gRefCOCO validation, the reported scores are cIoU 5 and gIoU 6, compared with ReLA’s cIoU 7 and gIoU 8 (Li et al., 2024).
Medical image segmentation adopts another decomposition. Self-supervised instance-adaptive prototype learning combines a Common Prototype Proposal, which is a soft-clustered prototype bank, with an Instance-Specific Prototype Proposal, which is built from the current image using confidence-weighted pooling. The reported ablations give the best mean DSC of 9 with three decoder scales, soft-cluster number 0, and threshold 1 on the 2-organ abdominal dataset (Liang et al., 10 Jul 2025).
Continual segmentation highlights a distinct adaptation target: not the query, but the evolving representation itself. Adapter introduces Adaptive Deviation Compensation, which measures the displacement between old and current sub-prototypes and updates a stored old-class prototype via
3
then blends it with the original prototype through an adaptive weight 4. The same framework adds an uncertainty-aware constraint and a compensation-based prototype similarity discriminative loss. On Pascal VOC in the challenging 5-6 overlapped setting, the reported all-class mIoU rises from STAR’s 7 to 8 (Zhu et al., 2024).
Detection uses analogous principles. ICPE conditions support features on the current query, dynamically aggregates salient regions within each support image and across support images, and then applies the resulting class prototype as channel-wise attention on query RoI features. On MS COCO, the paper reports AP 9 at 0-shot and AP 1 at 2-shot, and states that ICPE improves AP over Meta R-CNN by more than 3 points (Lu et al., 2022).
4. Cross-domain, unsupervised, and multimodal formulations
Cross-domain few-shot classification exposes a more subtle issue: adaptation can target the relation between prototypes and images rather than the prototype alone. CoPA argues that prototype embeddings and image instance embeddings extracted from a frozen backbone exhibit a natural gap analogous to a modality gap, and that a shared transformation head constrains the optimal representation. It therefore introduces separate heads,
4
and trains them with a symmetric cross-entropy contrastive objective. On Meta-Dataset in the train-on-all-datasets setting, the reported averages are 5 seen, 6 unseen, and 7 overall, compared with URL’s 8, 9, and 0 (Tian et al., 2024).
APPL modifies a different component of the few-shot pipeline. Rather than averaging support embeddings, it concatenates the support features of a class and passes them through a Prototype Calculator Network,
1
while meta-training adds inter-class and intra-class prototype regularizers,
2
At test time, APPL performs transductive refinement with a weighted-moving-average self-training rule on query distances. The resulting prototypes are not fully query-conditioned in a single forward pass, but they are adapted to the target query distribution across iterations (Heidari et al., 2023).
Unsupervised and zero-shot settings replace supervision by prompting, clustering, or semantics. In ConPVP, each sentence is paired with prompt-derived positive and negative prototypes, and the anchor is pulled toward its own positive prototype rather than merely toward an augmented view (Zeng et al., 2022). In novel category discovery, adaptive prototype learning maintains unit-norm class prototypes on a hypersphere, assigns unlabeled instances by cosine similarity,
3
updates prototypes online by EMA,
4
and adds pair-wise angular separation. The reported unlabeled training accuracies are 5 on CIFAR10, 6 on CIFAR100, 7 on OmniGlot, and 8 on the ImageNet 9-class split (Zhang et al., 2022). In generalized zero-shot learning, PCE-GZSL uses learnable class prototypes in projection space together with instance-adaptive pair scaling, and reports harmonic means of $\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$0 on AWA1, $\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$1 on AWA2, $\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$2 on CUB, and $\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$3 on SUN (Paul et al., 2023).
The same motif appears in robotics and multimodal skill transfer. UniPrototype represents demonstrations with a learnable matrix of prototype skill primitives,
$\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$4
computes soft assignments by
$\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$5
and selects the number of prototypes by monitoring
$\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$6
Its ablations report that soft row-normalized assignments outperform exclusive Sinkhorn normalization, and that adaptive selection of $\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$7 outperforms a fixed $\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$8 (Hu et al., 27 Sep 2025). This suggests that instance adaptivity can also mean compositional co-activation rather than a single personalized centroid.
5. Memory, multiple-instance learning, and interpretability
Long before the recent segmentation and few-shot literature, multiple instance learning already contained continuous and bag-adaptive prototype formulations. MIS-Boost learns discriminative instance prototypes in a boosting framework, does not restrict prototypes to a discrete set of training instances, and evaluates a bag by its minimum distance to each prototype,
$\vec p^s = \frac {\sum_{i,j}{\vec F_{i,j}^s \odot M_{i,j}^s}{\sum_{i,j}{M_{i,j}^s},$9
The overall classifier is
$\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$0
Because the nearest instance within each bag changes as the prototype moves, the selection is adaptive at the bag level even though the prototype itself is global (Akbas et al., 2011).
“Learning Maximally Predictive Prototypes in Multiple Instance Learning” formalizes a closely related idea with a permutation-invariant bag embedding,
$\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$1
followed by layer normalization and logistic prediction
$\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$2
Here, instance adaptivity again resides in the bag-specific instance that activates each prototype (Yuksekgonul et al., 2019).
Memory-augmented point cloud segmentation adds another perspective. MPNet stores a matrix of memory items shared across all instances and classes, computes attention weights
$\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$3
and retrieves an adaptive feature
$\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$4
For semantic segmentation it derives class summaries by averaging contiguous memory segments. On S3DIS Area-5, the ablation reports mPrec/mRec improving from $\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$5 in the baseline to $\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$6 in full MPNet (He et al., 2020).
Interpretability-oriented work makes the adaptive relation explicit at explanation time. The “alike parts” framework defines an importance-aligned overlap between an instance $\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$7 and its nearest prototype $\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$8 by computing normalized squared SHAP scores $\vec p_{k}^{q}= \frac {\sum_{i, j}{\vec {F_{i, j}^q} \odot \bar M_{i,j,k}^q}{\sum_{i, j}{ \bar M_{i,j,k}^q}.$9 and 0, the per-feature product
1
and the above-mean subset
2
The same signal is added to the prototype selection objective through
3
Across all reported datasets and algorithms, the feature-importance-informed prototypes maintain or improve 4-NN surrogate accuracy relative to the raw objective (Karolczak et al., 8 May 2025).
6. Empirical behavior, limitations, and recurring design tensions
Across domains, adaptive prototypes are reported to improve accuracy most clearly when a fixed prototype would be mismatched to the current query or representation. DPA’s ablation on S3DIS 5-way 6-shot rises from 7 for the ProtoNet baseline to 8 with Prototype Rectification, 9 after Prototype-to-Query Attention, and 00 after Prototype Distillation (Liu et al., 2024). APANet reports ResNet-101 results of 01/02 on PASCAL-503 and 04/05 on COCO-2006 for 07-shot/08-shot, while removing the class-agnostic branch at test time keeps inference complexity identical to standard prototype segmentation (Chen et al., 2021). MABP reports improved no-target recognition on gRefCOCO, with N-acc 09 versus ReLA 10 (Li et al., 2024). Adapter shows its largest gains in long multi-step continual settings, including the VOC 11-12 and 13-14 regimes (Zhu et al., 2024).
At the same time, the literature repeatedly records sensitivity to adaptation strength and to noise in the conditioning signal. In DPA, larger 15 can destabilize training, and increasing the number of decoder blocks improves performance only up to a point and then overfits (Liu et al., 2024). In APANet, the number of class-agnostic prototypes 16 and the loss weight 17 are both sensitive, with performance dropping when 18 is too small or too large and collapsing when 19 (Chen et al., 2021). ICPE notes that noisy or atypical query features can misguide coupling and that per-query conditioning lowers FPS and increases FLOPs relative to static prototypes (Lu et al., 2022). CoPA reports that simply adding a second head without contrastive regularization can overfit, indicating that more flexible prototype/image parameterizations need explicit alignment constraints (Tian et al., 2024).
Several papers also distinguish between fully query-conditioned prototypes and weaker forms of adaptivity. CoPA learns separate image and prototype transformations but does not construct query-conditioned prototypes in the strict sense (Tian et al., 2024). APPL learns parametric support-conditioned prototypes and adapts them transductively to the query distribution, but not as 20 in a single forward pass (Heidari et al., 2023). This suggests that the field contains a spectrum: from hard query-conditioned prototype construction, through episode-level or representation-level adaptation, to explanation-time or memory-time adaptation.
A plausible implication is that future work will continue to combine multiple adaptation axes rather than choosing only one. The existing papers already point in that direction: DPA combines rectification, attention, and distillation; APANet combines class-specific and class-agnostic branches; Adapter combines deviation compensation, uncertainty-aware filtering, and similarity discrimination; UniPrototype combines soft co-activation with adaptive prototype cardinality (Liu et al., 2024, Chen et al., 2021, Zhu et al., 2024, Hu et al., 27 Sep 2025). The common research problem is no longer merely how to compute a prototype, but how to decide which statistics should remain stable and which should change with the instance.