Prototype Grounding Strategies

• Prototype grounding is a modeling strategy that employs explicit prototypes as intermediates between raw inputs and predictions, ensuring evidence-based and interpretable results.
• It encompasses methods from nonparametric clustering to learnable prototype banks, serving as localized cues for object parts, concepts, and affordances.
• Research demonstrates that grounding prototypes improve performance and generalization in cross-view, multimodal, and generative tasks by aligning internal representations with concrete evidence.

Prototype grounding is a family of modeling strategies in which prototypes serve as explicit intermediates between raw inputs and grounded predictions. Recent work uses prototypes as source-derived object-part centroids for affordance localization, orthonormal evidence directions in class-specific subspaces, localized visual patterns with paired concept semantics, dynamic banks of affordance concepts, shared class-level anchors in multimodal embedding spaces, and projected real signal segments that support case-based explanations (Li et al., 2023, Jia et al., 9 Mar 2026, Colamonaco et al., 17 Apr 2026, Gou et al., 18 Mar 2026, Huang et al., 22 Sep 2025, Song et al., 7 May 2026). This suggests that prototype grounding is less a single architecture than a recurring design principle: constrain an otherwise opaque representation by forcing it to align with compact, inspectable, or transferable units of evidence.

1. Prototype forms and grounding targets

Across the recent literature, prototype grounding varies along two main axes. First, prototypes may be nonparametric cluster centroids extracted on the fly, learnable banks updated during training, orthonormal subspace bases, or projected exemplars from real training data (Li et al., 2023, Xie et al., 8 Sep 2025, Jia et al., 9 Mar 2026, Song et al., 7 May 2026). Second, the grounding target may be an affordance heatmap, a localized evidence patch, a concept bottleneck, a point-wise 3D mask, a bounding box, or a denoising condition (Gou et al., 18 Mar 2026, Tang et al., 2023, Faye et al., 13 Aug 2025).

Setting	Prototype form	Grounding target
LOCATE	k-means cluster centroids / latent concepts extracted on the fly	object-part affordance heatmap (Li et al., 2023)
AMP	orthonormal basis in a class-specific subspace on the Stiefel manifold	localized, non-overlapping part evidence and nearest training patch (Jia et al., 9 Mar 2026)
PGCM	learned visual prototypes with image and concept representations	part-level concepts and concept alignment table (Colamonaco et al., 17 Apr 2026)
APA for 3D affordance	dynamic learnable prototype bank of affordance concepts	per-point affordance probability mask (Gou et al., 18 Mar 2026)
TransCP / PAML / ViewRefer	cluster-level visual embeddings, multi-neighbor semantic prototypes, or learnable multi-view prototypes	open-vocabulary visual or 3D grounding (Tang et al., 2023, Xie et al., 8 Sep 2025, Guo et al., 2023)
ProtoSSL / MVCL-DAF++ / PDM	projected real signal segments, batch-derived class centroids, or internal codebook entries	case-based explanation, class-level semantic anchoring, or denoising guidance (Song et al., 7 May 2026, Huang et al., 22 Sep 2025, Faye et al., 13 Aug 2025)

These differences also change what “grounded” means. In inherently interpretable recognition, grounding means that each prototype is tied to concrete, human-verifiable evidence and can be associated with a localized image patch or training exemplar (Jia et al., 9 Mar 2026). In concept bottlenecks, grounding means that concepts are inferred through prototypes that have both image realizations and concept probabilities (Colamonaco et al., 17 Apr 2026). In open-vocabulary grounding, prototypes often operate as transferable semantic anchors or memory-bank entries rather than as fixed class templates (Tang et al., 2023, Xie et al., 8 Sep 2025).

2. Cross-view weakly supervised affordance grounding

A canonical prototype-grounding formulation appears in LOCATE, which formulates affordance grounding as the localization of the object region that supports a queried action in a cross-view weakly supervised setting. The source domain is a set of exocentric third-person interaction images, the target domain is an egocentric inactive object image, training uses only image-level affordance labels and object labels, and test-time input is only an egocentric image plus an affordance label (Li et al., 2023).

The method first extracts dense features $\mathcal{F}\in\mathbb{R}^{D\times H\times W}$ with a frozen DINO-ViT-S/16 backbone and predicts class-aware localization maps with a CAM-style head,

$$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$

For the ground-truth affordance class, LOCATE thresholds the normalized source localization map, collects the corresponding feature tokens, concatenates them across $N$ exocentric images into $f_{exo}\in\mathbb{R}^{L\times D}$, and clusters them with k-means into $\boldsymbol{p}\in\mathbb{R}^{K\times D}$. These prototypes are explicitly interpreted as latent concepts extracted from weakly localized interaction regions rather than as learnable memory slots. When $K=3$, the visualizations often align with the semantic decomposition human, object part, and background (Li et al., 2023).

Prototype selection is driven by cross-view correspondence. Each prototype is matched to the egocentric target feature map by dense cosine similarity,

$$S^{k,u,v}=\frac{\boldsymbol{p}_k \cdot \mathcal{F}_{ego}^{u, v}}{\left\| \boldsymbol{p}_k \right\|\left\| \mathcal{F}_{ego}^{u, v} \right\|},$$

and the resulting similarity maps are compared against a target-image saliency prior derived from DINO-ViT self-attention. The PartIoU heuristic then selects the prototype whose similarity map behaves like a part of the salient object rather than the whole object or background, with

$k^*=\arg\max_k \gamma_k.$

If $\max_k \gamma_k > \mu$, PartSelect outputs the selected object-part prototype $f_{op}$; otherwise no prototype is used for that sample (Li et al., 2023).

Grounding in the target image is enforced by making the target’s predicted part embedding match the source-derived prototype. LOCATE aggregates the egocentric feature map under the predicted localization map with masked average pooling to obtain $$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$0 and applies a cosine embedding loss

$$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$1

Training uses

$$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$2

with a one-epoch warm-up without $$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$3 because early CAMs are too noisy (Li et al., 2023).

The empirical case is direct. On AGD20K, which contains 20,061 exocentric images, 3,755 egocentric images, and 36 affordances, LOCATE achieves Seen: KLD 1.226, SIM 0.401, NSS 1.177, and Unseen: KLD 1.405, SIM 0.372, NSS 1.157. Compared with Cross-view-AG+, the paper reports unseen improvements of 20.4% KLD, 33.3% SIM, and 31.2% NSS. The ablations isolate the role of the prototype mechanism: replacing global knowledge transfer with regional knowledge transfer improves unseen KLD from 1.971 to 1.823, but adding PartSelect plus $$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$4 yields the major jump to KLD 1.439, SIM 0.358, NSS 1.130; adding concentration loss gives the final 1.405 / 0.372 / 1.157 (Li et al., 2023). In this formulation, prototype grounding is specifically cross-domain, cross-view part matching under weak supervision.

3. Interpretable recognition and concept alignment

In interpretable recognition, prototype grounding is defined more stringently: a prototype should correspond to concrete evidence and should not collapse into redundant copies of the same cue. AMP studies this failure mode explicitly and ties it to Neural Collapse under cross-entropy training. If class-specific prototypes are unconstrained Euclidean vectors, then near the terminal phase of training the paper argues that within-class variance vanishes, all prototypes converge toward the class mean, and the class prototype matrix becomes effectively rank-1. AMP replaces such free vectors with an orthonormal basis $$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$5 on the Stiefel manifold, scores local features by projection energy

$$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$6

and augments this with a nonnegative capacity vector $$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$7 to learn class-specific effective rank (Jia et al., 9 Mar 2026).

The geometric claim is that rank-1 prototype collapse becomes infeasible by construction because the basis vectors must remain orthonormal. AMP then addresses the residual rotational ambiguity of subspace models with two spatial regularizers. The basis-specific response map

$$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$8

is normalized into a spatial distribution $$\mathcal{P} = \psi_{cam}(\mathcal{F}) \in\mathbb{R}^{C\times H\times W}, \qquad \boldsymbol{z}={\rm GAP} (\mathcal{P}) \in\mathbb{R}^{C}.$$9; Spatial Entropy Minimization encourages each active direction to be local, and the Spatial Overlap Penalty discourages multiple active directions from focusing on the same region. At inference, the method forms weighted evidence maps, localizes the most contributing region, and retrieves the same-class training patch with maximal energy, grounding each active basis direction in both a test-image region and a supporting exemplar (Jia et al., 9 Mar 2026).

The reported gains are both predictive and interpretive. On CUB-200-2011, AMP achieves Consistency 76.80, Stability 49.20, OIRR 28.10, and DAUC 3.45; on Stanford Cars, Consistency 50.20, Stability 76.40, OIRR 33.50, and DAUC 5.65. Removing the Stiefel constraint causes the largest drop: on CUB, accuracy falls from 88.4 to 85.2, Consistency from 76.8 to 65.0, and DAUC worsens from 3.45 to 4.95 (Jia et al., 9 Mar 2026).

PGCM takes a different but complementary route. It modifies the concept bottleneck architecture from

$N$0

to

$N$1

where concepts are inferred through prototype assignments. Each prototype has a learnable embedding $N$2, an image representation $N$3, and a concept representation $N$4. For each segmented image part $N$5, the selector computes

$N$6

and each prototype carries Bernoulli concept probabilities $N$7 (Colamonaco et al., 17 Apr 2026).

Because prototypes have both visual and symbolic semantics, PGCM can expose a concept alignment table: each row shows a prototype’s image patch and the concepts active for that prototype. The model further enforces verifiability by re-encoding prototype images,

$N$8

and halfway through training it swaps each prototype to the closest real image part from the training set and freezes those images for the remainder of training. Prototype-level editing is then possible by changing the concept values in the table, removing spurious prototypes, or adding new ones. In noisy-label ColorMNIST+, removing wrong prototypes improves concept accuracy from $N$9 to $f_{exo}\in\mathbb{R}^{L\times D}$0, and editing them improves it from $f_{exo}\in\mathbb{R}^{L\times D}$1 to $f_{exo}\in\mathbb{R}^{L\times D}$2 (Colamonaco et al., 17 Apr 2026). The contrast with AMP is instructive: AMP grounds subspace directions through localized evidence and exemplar retrieval, whereas PGCM grounds concept semantics through explicit prototype images and prototype-level intervention.

4. Open-vocabulary, multi-view, and 3D grounding

Open-vocabulary grounding papers use prototypes primarily as transferable semantic anchors. TransCP learns a prototype bank from disentangled visual features rather than raw image tokens. After cross-modal visual context disentangling and language phrase attention, each disentangled visual feature is assigned to its nearest prototype by Euclidean distance, quantized with a stop-gradient trick, and fused with disentangled language features through a Hadamard product before box regression (Tang et al., 2023). The prototypes are not category labels; they are latent cluster-level visual embeddings intended to transfer from seen to unseen categories. The bank size matters mainly for transfer: too few prototypes make concepts too coarse, too many make them too fine, and the best tradeoff is 2048 prototypes. Empirically, TransCP reaches 84.25 / 87.38 / 79.78 on RefCOCO val/testA/testB and improves substantially over TransVG in both standard and cross-dataset open-vocabulary settings (Tang et al., 2023).

PAML also treats prototypes as transferable visual-semantic anchors, but its prototype bank is updated online with an EMA-style memory mechanism. Discriminative visual features are first refined by text-guided context suppression, then flattened into queries for a bank $f_{exo}\in\mathbb{R}^{L\times D}$3. For each token, PAML retrieves its top-$f_{exo}\in\mathbb{R}^{L\times D}$4 nearest prototypes by squared Euclidean distance, computes soft weights

$f_{exo}\in\mathbb{R}^{L\times D}$5

and inherits a weighted aggregate

$f_{exo}\in\mathbb{R}^{L\times D}$6

The final prototype-enhanced representation is a gated mixture of inherited prototype information and original local visual detail (Xie et al., 8 Sep 2025). The multi-neighbor design is central: on RefCOCO, $f_{exo}\in\mathbb{R}^{L\times D}$7 gives 84.31 / 87.42 / 80.02, while $f_{exo}\in\mathbb{R}^{L\times D}$8 gives the best 85.68 / 88.07 / 83.47; larger $f_{exo}\in\mathbb{R}^{L\times D}$9 degrades performance. In the open-vocabulary setting train-on-RefCOCO, test-on-ReferIt, the module ablation shows the largest gain when prototype inheritance is combined with the visual discriminative encoder, reaching 41.70 / 42.78 (Xie et al., 8 Sep 2025).

ViewRefer specializes prototype guidance to multi-view 3D visual grounding. It introduces learnable multi-view prototypes

$\boldsymbol{p}\in\mathbb{R}^{K\times D}$0

one per view, described as scene-agnostic 3D-text view-consistent knowledge. These prototypes are used twice: first to generate a view-guided textual context,

$\boldsymbol{p}\in\mathbb{R}^{K\times D}$1

and then to weight per-view predictions by

$\boldsymbol{p}\in\mathbb{R}^{K\times D}$2

In the main experiments, $\boldsymbol{p}\in\mathbb{R}^{K\times D}$3, so the model uses four prototypes. The full method surpasses the second-best by +2.8% on Sr3D, +1.5% on Nr3D, and +1.35% on ScanRefer (Guo et al., 2023). Here, prototype grounding is a form of global view prior rather than part or class grounding.

Language-driven 3D affordance grounding adds yet another variant. The APA module defines a dynamic learnable prototype bank

$\boldsymbol{p}\in\mathbb{R}^{K\times D}$4

or, in the supplement, $\boldsymbol{p}\in\mathbb{R}^{K\times D}$5 with dynamic $\boldsymbol{p}\in\mathbb{R}^{K\times D}$6 initialized to 17 and expanded when new affordances are encountered. A masked average pooled region embedding $\boldsymbol{p}\in\mathbb{R}^{K\times D}$7 from the fused point features is classified against these prototypes by cosine similarity,

$\boldsymbol{p}\in\mathbb{R}^{K\times D}$8

using the temperature-scaled loss $\boldsymbol{p}\in\mathbb{R}^{K\times D}$9 (Gou et al., 18 Mar 2026). Prototype association is training-only and does not affect inference efficiency. On OpenAfford, removing APA drops open-set full-view aIoU from 18.38 to 14.31, open-set partial-view from 15.85 to 14.76, closed-set seen from 19.18 to 17.37, and closed-set unseen from 17.81 to 16.05 (Gou et al., 18 Mar 2026). In this setting, prototypes encode cross-object geometric consistency for a given affordance rather than localized evidence patches.

5. Sequence models, time series, and generative conditioning

Prototype grounding also appears in multimodal classification, self-supervised time-series learning, generative modeling, and language-plan execution. MVCL-DAF++ introduces prototype-aware contrastive alignment as class-level semantic anchoring in multimodal intent recognition. For each class $K=3$0, the prototype is the batch mean

$K=3$1

with $K=3$2, followed by L2 normalization. Each instance embedding $K=3$3 is then aligned by a class-conditioned InfoNCE objective against $K=3$4 and all other class prototypes (Huang et al., 22 Sep 2025). The formulation is dynamic, batch-derived, and shared across modalities. The reported gains are strongest under noisy or long-tailed conditions: the abstract reports rare-class improvements of +1.05% WF1 on MIntRec and +4.18% WF1 on MIntRec2.0, and the loss ablation shows classification + contrastive + prototype outperforming either component alone (Huang et al., 22 Sep 2025).

ProtoSSL separates motif discovery from label alignment and grounding in unlabeled time-series data. During pretraining it learns a prototype bank $K=3$5 by applying SimCLR-style self-supervision directly to prototype activations rather than to the encoder embedding. For ECG, activations are local:

$K=3$6

where $K=3$7 is a sliding 1-second window. Downstream, prototypes are assigned to labels by solving a rectangular linear assignment problem and then grounded by projection to the actual labeled sample or local segment that maximally activates them (Song et al., 7 May 2026). The final grounded prototypes are therefore real examples rather than free latent vectors. The human evaluation is notable: prototype acceptability is 91.4% for ProtoSSL versus 66.4% for SupProto Direct, and explanation acceptability is 82.9% versus 67.9% (Song et al., 7 May 2026). In this line of work, grounding is explicitly case-based.

PDM moves prototype grounding into generation. It learns a bank of internal visual prototypes $K=3$8 from clean image features $K=3$9 and assigns each image to its nearest prototype

$$S^{k,u,v}=\frac{\boldsymbol{p}_k \cdot \mathcal{F}_{ego}^{u, v}}{\left\| \boldsymbol{p}_k \right\|\left\| \mathcal{F}_{ego}^{u, v} \right\|},$$0

The denoiser then predicts noise conditioned on the selected prototype and time embedding,

$$S^{k,u,v}=\frac{\boldsymbol{p}_k \cdot \mathcal{F}_{ego}^{u, v}}{\left\| \boldsymbol{p}_k \right\|\left\| \mathcal{F}_{ego}^{u, v} \right\|},$$1

Training combines a diffusion loss, an InfoNCE-like contrastive prototype loss, a direct alignment loss, and a prototype diversity penalty (Faye et al., 13 Aug 2025). Unlike retrieval-augmented diffusion, PDM uses no external memory; prototypes are internal learned codebook entries updated jointly with the encoder and denoiser. On CIFAR-10, DDPM obtains FID 18.45, ProtoDiffusion 11.70, PDM 8.10, and supervised s-PDM 6.58; similar trends hold on STL-10, EuroSAT, and Tiny ImageNet (Faye et al., 13 Aug 2025).

A broader abstraction appears in GLiDE, which treats mode families as grounding primitives between language plans and low-level trajectories. The mode sequence $$S^{k,u,v}=\frac{\boldsymbol{p}_k \cdot \mathcal{F}_{ego}^{u, v}}{\left\| \boldsymbol{p}_k \right\|\left\| \mathcal{F}_{ego}^{u, v} \right\|},$$2 is supplied by language, while a classifier $$S^{k,u,v}=\frac{\boldsymbol{p}_k \cdot \mathcal{F}_{ego}^{u, v}}{\left\| \boldsymbol{p}_k \right\|\left\| \mathcal{F}_{ego}^{u, v} \right\|},$$3 grounds low-level states into those discrete modes using only trajectory-level success/failure supervision from counterfactual perturbations. In 2D navigation, the learned mode classifier reaches 0.989 accuracy in both the 3-mode and 4-mode settings, but removing counterfactual failures reduces the corresponding numbers to 0.701 and 0.669 (Wang et al., 2024). Although this paper does not use prototype terminology in the same strict sense as ProtoSSL or PDM, it explicitly interprets mode families as prototype-like execution abstractions