Class-Specific Prototype Anchoring

• Class-specific prototype anchoring is the design principle of using reference prototypes—centroids, manifolds, or distributions—to enforce intra-class compactness and inter-class separability.
• It is applied in diverse domains such as few-shot learning, semantic segmentation, and federated learning by leveraging fixed, learnable, or data-derived anchors.
• Empirical results demonstrate that integrating prototype anchors improves accuracy, calibration, and robustness while managing stability-plasticity trade-offs in complex models.

Class-specific prototype anchoring denotes the explicit use of class-conditioned reference entities—prototypes, semantic anchors, manifolds, latent distributions, or anchor boxes—to organize representation learning and decision making around class-specific targets. Across the cited works, these anchors may be pre-defined and independent of training features, as in Semantic Anchor Regularization (Ge et al., 2023); maintained as compact class-wise memories for contrastive learning, as in COARSE3D (Li et al., 2022); learned end-to-end as class-level conditioning vectors, as in SkillMoV (Bianchi et al., 16 Jun 2026); represented by class-specific autoencoder manifolds, as in CSSR (Huang et al., 2022); or instantiated as clustered 3D box priors, as in CAP (Raffiee et al., 2019). The common objective is compactness within a class and well-separability between classes, but the operational meaning of “prototype” varies substantially by task and modality.

1. Conceptual scope and taxonomy

In its most direct form, a class-specific prototype is the mean of support or class features. Prototype classifiers in few-shot learning form a class prototype

$p_c=\frac{1}{|S_c|}\sum_{x\in S_c} f(x),$

and classify a query by nearest prototype using a softmax over negative distances (Hou et al., 2021). Class-incremental segmentation methods such as Cs2K also store old-class prototypes as averages of pixel features assigned to each class, using them later for prototype-guided pseudo labeling and class adaptation (Cong et al., 2024).

Other formulations decouple the anchor from the empirical class mean. SAR introduces pre-defined, class-specific vectors that are generated before training and kept fixed; only their embedded versions are EMA-aggregated, and no gradient flows into the original anchor bank (Ge et al., 2023). FedSA makes a closely related move in federated learning by using one semantic anchor per class to decouple prototype generation from local representation learning, then updating anchors by EMA from aggregated client prototypes (Zhou et al., 9 Jan 2025). FedHPro extends the idea further: class anchors on the server are not single vectors but small sets of learnable global class-wise hyper-prototypes, whose averages serve as anchors and are optimized by gradient matching against class-wise gradients distilled from clients’ real samples (Wang et al., 13 May 2026).

Several works broaden “prototype” beyond a point centroid. CSSR replaces point prototypes with manifolds represented by class-specific autoencoders and measures class belongingness through reconstruction error in semantic feature space (Huang et al., 2022). Prototype Latent World Model Replay represents each class as a Gaussian mixture over several latent prototypes with class-specific variances, then samples replay states from that mixture during later increments (Nie et al., 28 Jun 2026). GeoProto anchors matching inside a class-specific diffusion manifold rather than in raw Euclidean feature space, using per-class diffusion maps and differentiable Nyström interpolation (Jia et al., 21 Sep 2025).

This range suggests that class-specific prototype anchoring is better understood as a design principle than as a single algorithmic template. A plausible implication is that the “anchor” can be a centroid, a distribution, a manifold, a cache, or even a geometry prior, provided that it acts as a class-conditioned reference for attraction, calibration, replay, or proposal generation.

2. Mathematical forms of the anchor

The simplest anchor form is a class mean in feature space. Few-shot prototype classifiers use the mean support feature as the class center and classify by nearest prototype; the main finding of “A Closer Look at Prototype Classifier for Few-shot Image Classification” is that L2 normalization and transformations minimizing the ratio of within-class variance to between-class variance can make such prototype anchors effective without retraining or meta-learning (Hou et al., 2021).

SAR formalizes a different regime. For classification, with feature $f(x)\in\mathbb{R}^d$, label $y$, and semantic anchor $\hat a_y$, the anchor attraction term is

$L_{\text{anchor}}=\|f(x)-\hat a_y\|_2^2,$

and the full training objective is

$$L_{\text{total}}=L_{\text{ce}}(x,y)+\lambda_1 L_{\text{aux-ce}}+\lambda_2 \|f(x)-\hat a_y\|_2^2.$$

The anchor itself is pre-defined, embedded by a trainable mapping $h_\psi$, and stabilized by EMA; inter-class separability of anchors is enforced by a classifier-aware auxiliary cross-entropy with confidence-based reweighting (Ge et al., 2023).

COARSE3D formulates class-specific prototype anchoring as an InfoNCE-style contrastive problem. For an anchor $a$ with predicted class $y$, the loss uses all prototypes of class $y$ as positives and all prototypes of other classes as negatives: $f(x)\in\mathbb{R}^d$0 Here the prototypes are keys from a class-wise memory bank, while low-entropy predicted pixels or points serve as anchors (Li et al., 2022).

Stochastic Prototype Embeddings treats both embeddings and prototypes as random variables. The support-derived class anchor is a Gaussian posterior

$f(x)\in\mathbb{R}^d$1

with posterior mean and covariance obtained by precision-weighted aggregation of support embeddings. Classification then uses a Gaussian predictive density rather than a deterministic nearest-centroid rule (Scott et al., 2019). Prototype Latent World Model Replay uses a related but task-distinct construction, modeling each class as

$f(x)\in\mathbb{R}^d$2

so that replay samples are drawn from prototype-centered latent distributions rather than from stored images (Nie et al., 28 Jun 2026).

In CSSR, the class anchor is a manifold $f(x)\in\mathbb{R}^d$3 realized by a class-specific autoencoder $f(x)\in\mathbb{R}^d$4, and the per-class score is derived from reconstruction error,

$f(x)\in\mathbb{R}^d$5

Class assignment is then based on softmax over $f(x)\in\mathbb{R}^d$6, and unknown detection can be based on thresholding reconstruction-derived scores (Huang et al., 2022).

SkillMoV uses one learnable prototype vector per class, $f(x)\in\mathbb{R}^d$7, not as a nearest-prototype classifier, but as conditioning anchors. After cross-view attention and view pooling to $f(x)\in\mathbb{R}^d$8, it computes

$f(x)\in\mathbb{R}^d$9

and uses the similarity vector $y$0 to gate a projection that produces the final skill embedding (Bianchi et al., 16 Jun 2026).

These forms are mathematically heterogeneous, but they all encode class membership relative to a class-conditioned reference geometry. That geometry may be Euclidean, contrastive, probabilistic, reconstructive, or manifold-based.

3. Construction, update, and optimization regimes

A major fault line in the literature concerns whether anchors are fixed, learned, or data-derived. SAR fixes the original anchor matrix $y$1, embeds it through $y$2, updates only the semantic anchors $y$3 by EMA, and uses them only when auxiliary classification confidence exceeds a threshold $y$4 (Ge et al., 2023). This design explicitly avoids prototype drift induced by minibatch feature statistics.

COARSE3D moves in the opposite direction. It initializes a prototype bank randomly, updates it every iteration from labelled pixels only, assigns labelled embeddings to class-specific prototypes through Sinkhorn-balanced optimal transport on cosine distance, then applies EMA with momentum $y$5 followed by $y$6-normalization (Li et al., 2022). The anchors in that setting are not fixed references but online class-wise keys constrained by labelled data and balanced assignment.

Federated methods expose another update regime. FedSA broadcasts semantic anchors to clients, uses anchor-based regularization and classifier calibration locally, aggregates local prototypes by class-count-weighted averaging, and updates anchors by

$y$7

The anchors are thus neither purely local nor directly trained from client gradients; they are centrally maintained reference signals (Zhou et al., 9 Jan 2025). FedHPro instead updates global class-wise hyper-prototypes by minimizing cosine dissimilarity between aggregated client feature gradients and server-side “virtual” gradients computed on the hyper-prototype variables themselves (Wang et al., 13 May 2026).

Prototype calibration during deployment yields still another regime. CPL-NC freezes the visual encoder, updates only textual prototypes at test time, and anchors those textual updates on a class-aware visual prototype cache. The cache admits only low-entropy visual features, dynamically scales per-class capacity based on frequency, and applies inactivity-aware rejuvenation for inactive classes; alignment and negative contrast losses are then used to refine the textual side against these stable visual anchors (Qiao et al., 22 Oct 2025).

Incremental segmentation methods use prototypes less as static targets and more as calibration signals. Cs2K computes old-class prototypes from the previous dataset and previous model, uses feature-to-prototype proximity to rectify pseudo labels, and generates self-augmented and inter-prototype augmented features for prototype-guided class adaptation (Cong et al., 2024). Prototype-Guided Calibration Distillation and Dual-Aligned Prototype Distillation for class-incremental medical image segmentation maintain global prototypes by cumulative moving average, compute batch-local prototypes, then align current local prototypes with both old global prototypes and old-model local prototypes (Zhu et al., 11 Nov 2025).

Taken together, these constructions show that “anchoring” may be enforced by EMA, balanced memory-bank updates, class-wise gradient matching, cumulative moving averages, or asymmetric test-time refinement. A plausible implication is that the anchor update rule is inseparable from the intended stability–plasticity trade-off of the application.

4. Representative instantiations across domains

The literature spans a wide range of tasks, and the anchor object changes with the domain.

Setting	Anchor form	Representative paper
Semantic segmentation	Pre-defined semantic anchors or stored old-class prototypes	(Ge et al., 2023, Cong et al., 2024, Zhu et al., 11 Nov 2025)
Weakly-supervised 3D segmentation	Class-wise prototype memory bank with entropy-selected anchors	(Li et al., 2022)
Multi-view proficiency estimation	One learnable prototype per class used for conditioning	(Bianchi et al., 16 Jun 2026)
Federated learning	Semantic anchors or global hyper-prototypes on the server	(Zhou et al., 9 Jan 2025, Wang et al., 13 May 2026)
Open-set and fine-grained recognition	Class-specific AE manifolds or diffusion-space prototypes	(Huang et al., 2022, Jia et al., 21 Sep 2025)
Incremental and few-shot learning	Condensed prototypes, latent prototype mixtures, or tuned prototypes	(Kong et al., 2023, Nie et al., 28 Jun 2026, Jiang et al., 5 Feb 2026, Huang et al., 29 Oct 2025)
RF and 3D detection	Capture-local pseudo-prototypes or clustered 3D box priors	(Yao et al., 6 Jul 2026, Raffiee et al., 2019)

In few-shot object detection, PDA maintains support-only prototypes in a learnable identity-initialized projection space, optionally updates them by EMA during fine-tuning using labelled foreground RoIs, applies best-of-$y$8 matching to handle intra-class multi-modality, and fuses prototype-based logits with detector logits as a calibrated “second opinion” (Huang et al., 29 Oct 2025). In few-shot class-incremental learning, Efficient Prototype Tuning freezes the feature extractor and instead fine-tunes class prototypes through a decomposition

$y$9

where $\hat a_y$0 is a class-specific offset and $\hat a_y$1 is a task-aware offset (Jiang et al., 5 Feb 2026).

In class-incremental learning without exemplar storage, YONO stores one condensed prototype per class, learns it by an attentional mean-shift update, and uses ArcFace both to pull samples toward their class prototype and to separate classes by an angular margin (Kong et al., 2023). Prototype Latent World Model Replay replaces stored exemplars with class-conditional prototype mixtures in a frozen latent space, using replayed latent samples and supervised contrastive learning in adapter space to preserve old decision regions (Nie et al., 28 Jun 2026).

Several methods use anchoring to compensate for domain shift rather than data scarcity alone. The digital-twin fault-diagnosis method performs bi-directional twin-domain prototype anchoring: physical-space features are attracted to digital-twin prototypes, and digital-twin features are attracted to prototypes estimated from few-shot physical support data (Xia et al., 7 Mar 2026). PISA-CAPC separates source-domain physics-informed structure anchoring from target-domain fixed-backbone calibration, then uses unlabeled, capture-local pseudo-prototypes to recalibrate target scores without updating the backbone (Yao et al., 6 Jul 2026).

Even 3D anchor generation has been recast in prototype terms. CAP clusters ground-truth 3D box dimensions $\hat a_y$2 separately for cars, pedestrians, and cyclists, then uses the cluster means as class-specific anchor prototypes for AVOD’s proposal generation (Raffiee et al., 2019).

5. Empirical patterns and quantitative evidence

The empirical record is heterogeneous, but several recurring effects are explicit. In semantic segmentation, SAR improves single-scale, no-TTA mIoU on Cityscapes, ADE20K, and Pascal-Context across FCN, DeepLabV3, HRNet, OCRNet, SegFormer, and UPerNet backbones. On Cityscapes, for example, HRNet improves from 79.9 to 81.4 and OCRNet from 80.7 to 81.7; on ADE20K, HRNet improves from 42.0 to 42.8; on Pascal-Context, FCN improves from 48.4 to 49.7 (Ge et al., 2023). The same study reports strong rare-class gains, including Cityscapes “Train” IoU 75.5 → 83.9 and Pascal-Context “Mouse” IoU 34.7 → 40.3, while adding approximately 0.03 GFLOPs and approximately 1.56M parameters on HRNet with $\hat a_y$3 input (Ge et al., 2023). Cs2K similarly reports large gains over prototype-agnostic baselines in incremental semantic segmentation, such as VOC 10-1 “all” mIoU 30.5 → 61.5 over PLOP and ADE20K 100-5 “all” mIoU 25.9 → 34.2 over MiB (Cong et al., 2024).

Under sparse supervision, COARSE3D reports consistent gains over its weakly supervised baselines. On SemanticKITTI hidden test with SalsaNext, the mIoU gain is +5.6 at 0.1% labels and +3.6 at 0.01% labels; on SemanticPOSS validation, gains are +4.1 and +3.7; and on SemKITTI validation at 1% labels it reaches 58.30 mIoU, close to its 100% label result of 58.39 (Li et al., 2022). This suggests that prototype-bank anchoring can remain effective even when annotations are extremely sparse.

In multi-view proficiency estimation, prototype anchoring in SkillMoV is not the dominant ablation, but it is substantial: removing prototype anchoring reduces Exos accuracy by 4.07 percentage points, while the full model reaches 50.17% in the Exos setting and 47.63% in Ego+Exos (Bianchi et al., 16 Jun 2026). In federated learning, FedHPro reports 84.80% average accuracy on Digits versus 82.54% for FedSA and 78.82% for FedAvg, and 64.52% on Office-Caltech versus 60.57% for FedSA (Wang et al., 13 May 2026). FedSA, under model heterogeneity on CIFAR-100 HtFE8, reports 35.68% versus 16.31% for FedProto and 32.04% for FedTGP (Zhou et al., 9 Jan 2025).

Anchor-based calibration also yields strong task-specific gains outside standard classification. PISA-CAPC reaches 0.9257 target-domain mean Macro-F1 under a balanced transductive setting on a ten-transmitter multi-antenna WiFi benchmark, while ablating capture grouping reduces Macro-F1 to 0.8993 and removing the class-balance prior yields 0.9097 (Yao et al., 6 Jul 2026). CAP improves KITTI 3D detection AP by +7.19%, +8.13%, and +8.8% on Easy, Moderate, and Hard pedestrian settings, by +2.19%, +2.17%, and +1.27% on car, and by +12.1% on cyclist Easy (Raffiee et al., 2019).

In incremental settings, the memory-free latent replay method raises Split CIFAR-100 LastAcc from 4.55% to 31.64% in Inc5, from 9.06% to 37.06% in Inc10, and from 16.96% to 43.10% in Inc20 (Nie et al., 28 Jun 2026). YONO+ reaches 60.98% average accuracy with 16.87% forgetting on CIFAR-100 b0-10 and 58.42% accuracy with 22.66% forgetting on TinyImageNet b0-10 (Kong et al., 2023). These findings do not establish a single universal advantage, but they do indicate that class-specific anchors are repeatedly associated with improved margin control, retention, or calibration when compared with less class-structured baselines.

6. Limitations, failure modes, and open directions

Several limitations recur explicitly. SAR notes that a poorly spread anchor basis, such as an unlucky standard-normal draw, can slow auxiliary separation, that very low feature dimension makes separability among many class anchors harder, and that extreme class imbalance may still require complementary re-weighting or re-sampling techniques (Ge et al., 2023). COARSE3D reports that if labels are too sparse per class, prototype clustering can become unreliable, as seen on nuScenes at 0.01% labels, where 10 of 16 classes have fewer than 20 labels and performance drops (Li et al., 2022). SkillMoV emphasizes that its prototypes are not ordinally calibrated: figure-level analysis shows that one prototype can receive the highest average cosine similarity across all classes, so the learned geometry is not monotonic along the skill axis (Bianchi et al., 16 Jun 2026).

Domain-dependent anchoring brings its own sensitivities. PISA-CAPC depends on capture grouping and can degrade when capture identifiers are unavailable; it is also sensitive to topology or CFO misestimation (Yao et al., 6 Jul 2026). CAP improves anchor coverage but can under-handle unusual-size outliers when the number of clusters is too small, particularly for cyclist Moderate and Hard settings (Raffiee et al., 2019). CSSR scales per class because each known class has a dedicated autoencoder, increasing parameters and per-class compute at inference (Huang et al., 2022). GeoProto notes that the quality of each class manifold depends on sufficient and representative per-class samples, and that inference cost grows linearly with the number of class manifolds (Jia et al., 21 Sep 2025).

Incremental and federated settings add stability and privacy issues. FedHPro points out that class-wise gradient signals may be noisy when clients have very few samples for certain classes and that sharing average feature-space gradients raises privacy considerations, motivating secure aggregation or differential privacy (Wang et al., 13 May 2026). FedSA observes that anchors for rare classes adapt more slowly when clients lack those classes locally and that feature-dimension alignment is required under model heterogeneity (Zhou et al., 9 Jan 2025). YONO’s single condensed prototype can under-represent highly multimodal classes, and its synthetic replay may become less faithful if the feature extractor drifts substantially across long task sequences (Kong et al., 2023).

The future directions proposed in the source material are correspondingly diverse. SAR suggests adaptive anchor refinement, hybrid learned-frozen anchors, and margin-based variants (Ge et al., 2023). COARSE3D proposes adaptive $\hat a_y$4, consistency regularization, and richer uncertainty-based anchor selection (Li et al., 2022). FedHPro points to hierarchical class grouping and semi-supervised extensions (Wang et al., 13 May 2026). PISA-CAPC raises online capture segmentation and open-set extensions (Yao et al., 6 Jul 2026). Prototype Latent World Model Replay suggests more expressive latent densities beyond diagonal Gaussian mixtures (Nie et al., 28 Jun 2026). Collectively, these proposals indicate that the next stage of class-specific prototype anchoring is likely to focus less on whether anchors should exist and more on how anchors should be parameterized, updated, regularized, and scaled under long-tail, multimodal, federated, and nonstationary conditions.