---
title: 'PGCMs: Prototype-Grounded Concept Models'
url: https://www.emergentmind.com/topics/prototype-grounded-concept-models-pgcms
type: topic
---

# PGCMs: Prototype-Grounded Concept Models

Prototype-Grounded Concept Models (PGCMs) are a family of models in which concepts are represented explicitly by prototypes rather than only by abstract latent coordinates or scalar bottleneck activations. Across the literature, the term has been used in several closely related ways: as a concept-learning framework with discrete prototype slots and latent swapping in Interactive Concept Swapping Networks (iCSNs) [2112.02290]; as a multi-modal architecture in which each perceptual category is represented by a single prototype vector that drives both description generation and zero-shot classification [2303.04053]; as a “gray-box” classifier family in which predictions are made by matching unseen inputs against learned prototypes [2205.15769]; and as a concept-bottleneck-style architecture in which each concept is tied to learned visual prototypes, enabling direct inspection of concept semantics and targeted intervention at the prototype level [2604.16076]. In all of these formulations, the central claim is not merely that prototypes aid prediction, but that they make concept semantics concrete, inspectable, and, in some settings, editable.

## 1. Definitions and conceptual scope

The 2023 vision–language formulation defines a PGCM as “a multi-modal neural architecture that represents each perceptual category $\mathcal{C}$ by a single prototype vector $\mathbf{p}_{\mathcal{C}\in\mathbb{R}^d}$,” with both generation of natural-language descriptions and zero-shot classification driven by these prototype vectors [2303.04053]. In that account, prototype-only representations are contrasted with exemplar-based models that store one or more specific example embeddings per category, and the stated motivation is that a single abstract prototype emphasizes “the central tendency of the category,” yielding “more compact representations” and “(often) better generalisation in zero-shot settings” [2303.04053].

A broader machine-learning definition appears in work on ProtoPNets and concept-level debugging, where Prototype-Grounded Concept Models are described as a family of “gray-box” classifiers built on two core ideas: concepts are represented explicitly as a finite set of learned prototypes in latent space, and predictions are made by matching unseen inputs against these concept prototypes [2205.15769]. In that usage, interpretability and transparency are co-equal design goals: every prototype corresponds to a concrete part or example drawn from the training data, and the decision procedure is a distance-based matching stage followed by a simple aggregation [2205.15769].

The 2026 concept-alignment formulation narrows the emphasis further. It presents PGCMs as revisiting the familiar “input→concepts→task” pipeline of Concept Bottleneck Models while removing “the unspoken—and uncheckable—assumption that an abstract scalar concept ‘means what its name says’” [2604.16076]. There, each concept is explicitly tied to a finite bank of learned visual prototypes, and this grounding is presented as the mechanism that enables “verifiable alignment” and prototype-level intervention [2604.16076].

Outside contemporary deep learning, a much earlier cognitive formalization represents prototypes and features as vectors and subspaces in a complex Hilbert space, using state vectors, prototype vectors, and entangled measurements to model typicality, graded membership, context dependence, and concept combination [1303.2430]. This suggests that the phrase “prototype-grounded” spans both interpretable neural architectures and formal theories in which prototype structure is treated as fundamental to concept representation.

## 2. Architectural patterns

Despite substantial variation, several recurring architectural patterns appear across PGCM instantiations.

In iCSNs, the model builds on a deterministic autoencoder with encoder $f_\theta$ and decoder $g_\phi$, but inserts concept read-out modules and prototype slots into the latent space [2112.02290]. The encoded latent vector $z=f_\theta(x)\in\mathbb{R}^Z$ is transformed by category-specific read-out encoders $\phi_j=m_j(z)\in\mathbb{R}^Q$, and each category $j$ has a codebook of learnable prototype vectors $\mathrm{P}_j=[p_j^1,\dots,p_j^{K_j}]$ [2112.02290]. A scaled dot-product followed by two nested softmaxes yields softened but sharpenable assignments $\Pi_j$, which are concatenated into a joint discrete code $y=[\Pi_1|\Pi_2|\dots|\Pi_J]$ and decoded to reconstruct the input [2112.02290].

In the multi-modal perceptual-description setting, the architecture is explicitly split into two separately trained networks, Generator and Interpreter [2303.04053]. Both share a vision encoder and a label embedding matrix $\mathbf{V}\in\mathbb{R}^{N\times D}$; in the Generator, each row $\mathbf{v}_\ell$ acts as the prototype $\mathbf{p}_\ell$, and in the Interpreter the same matrix contains the target prototypes for classification [2303.04053]. The Generator uses a Transformer decoder conditioned on the prototype, while the Interpreter uses a BERT-based text encoder plus a projection layer intended to match one of the prototypes [2303.04053].

In ProtoPNets, the architecture comprises three stages: a pre-trained convolutional feature extractor mapping an image to a latent tensor, a part-prototype matching stage over $1\times1\times d'$ latent patches, and a linear aggregation stage with class-specific weights [2205.15769]. Prototype activations are computed by either a “difference of logs” function or an exponential form, both monotonically decreasing in prototype–patch distance, followed by max-pooling over latent patches [2205.15769]. The resulting explanation template is explicit: a prediction is justified because the image strongly matches specific part-prototypes that are themselves image patches from training examples [2205.15769].

The 2026 PGCM architecture is closer to a latent-variable concept bottleneck model [2604.16076]. It uses a shared encoder $f_{\mathrm{enc}}$, a bank of learnable prototype embeddings $\{e_1,\dots,e_m\}$, an image decoder $f_{\mathrm{image}}$, a concept decoder $f_{\mathrm{concept}}$, and a transparent task predictor $g(c)$ [2604.16076]. Each image part $x_i$ is assigned a latent variable $S_i\in\{1,\dots,m\}$ with a uniform prior, and the posterior selector is a softmax over prototype similarities [2604.16076]. Conditioned on a selected prototype, the model simultaneously reconstructs the image part and predicts its concept vector [2604.16076].

Other variants retain the prototype-centered logic while changing the surrounding machinery. Concept-Guided Prototype Networks store prototypes in a learnable matrix partitioned into positive and negative pools for each binary concept, compute each prototype’s activation by negative Euclidean distance to the nearest patch, and feed the resulting similarity vector into a linear concept-prediction head [2410.18705]. The Locality-aware Concept Bottleneck Model associates one prototype with each concept, computes prototype–patch cosine similarities over spatial features, and uses CLIP-guided masking plus attention-based aggregation to improve localization [2508.14562].

## 3. Grounding mechanisms and training objectives

The distinguishing feature of PGCMs is not only the presence of prototypes, but the mechanism by which they are semantically bound to concepts.

In iCSNs, the key grounding mechanism is latent swapping between paired inputs that share a specific category [2112.02290]. For two inputs that share exactly the basic concept in category $v$, the model swaps only the $v$-th category’s prototype assignments between their codes, then requires the swapped decodings to reconstruct the original images [2112.02290]. The stated effect is that the $v$-th prototype slot is forced to encode only the category-$v$ concept, because that is the only information being exchanged [2112.02290]. Training uses a single paired reconstruction loss, with no explicit KL, mutual-information, clustering, or sparsity terms; the report states that “the swapping plus the sharpened softmax” suffice to bind category information to prototype slots [2112.02290].

In ProtoPNets, grounding is enforced through cluster and separation regularizers together with projection of prototypes onto actual latent patches from training examples [2205.15769]. The loss combines the negative log-likelihood of the class label with $L_{\mathrm{clst}}$ and $L_{\mathrm{sep}}$, where the former encourages each training example to be close to a same-class prototype and the latter repels it from prototypes of other classes [2205.15769]. Projection then ensures that prototypes correspond to actual image regions rather than unconstrained latent points [2205.15769].

In the 2026 concept-alignment PGCM, grounding is probabilistic and joint: prototypes are responsible both for reconstructing image parts and for predicting concept vectors [2604.16076]. The total loss is written as
$$
L_{\mathrm{total}} = L_{\mathrm{task}} + L_{\mathrm{concept}} + \lambda_{\mathrm{recon}}\cdot L_{\mathrm{recon}} + \beta\cdot L_{\mathrm{KL}},
$$
where the reconstruction term forces the prototype bank to “faithfully cover the visual variability of parts,” and the concept loss forces each prototype to carry semantically correct concept labels $\pi_j$ [2604.16076]. This is presented as a VAE-style ELBO objective on the joint of parts, concepts, and task labels [2604.16076].

In Concept-Guided Prototype Networks, grounding is reinforced by prototype-conformity losses and, optionally, orthogonality losses [2410.18705]. The cluster cost pushes at least one patch of an example toward a same-class prototype, the separation cost repels patches from opposite-class prototypes, and the total loss may combine classification, prototype conformity, diffusion, and orthogonality terms [2410.18705]. In Locality-aware CBM, grounding is spatial: a locality loss matches concept influence distributions to masked prototype-similarity distributions via per-concept KL divergence, while an auxiliary classification head regularizes aggregated prototype representations [2508.14562].

A different grounding mechanism appears in the multi-modal PGCM for category descriptions. There, prototypes are not attached to local image parts but to categories themselves, and both language generation and interpretation are conditioned on the same prototype vectors [2303.04053]. The interpreter’s text projection is trained with a cosine-embedding loss with margin $\delta=0.1$, and the Generator combines a visual classification term with an autoregressive language loss [2303.04053]. This formulation grounds prototypes at the category level rather than at the part level.

## 4. Interpretability, alignment, and intervention

A persistent theme across PGCM work is that prototype grounding should make concept semantics inspectable rather than merely interpretable in principle.

The 2026 formulation makes this explicit by defining the semantics of concept $k$ as
$$
C_k \text{ is active } \Longleftrightarrow \exists \text{ prototype } j \text{ with } \pi_{j,k}=1 \text{ such that “part looks like } \mu_j\text{.”}
$$
It proposes a “Concept Alignment Table” in which each row shows the image prototype $\mu_j$ and its concept vector $\pi_j$, so that a human can directly verify whether the visual evidence matches the concept assignment [2604.16076]. Once a misaligned prototype is identified, the stated interventions include removing the prototype, editing its concept labels, or adding a new prototype sourced from a real image part [2604.16076]. The same report also notes that prototype-selection interventions—fixing $q(S_i|x_i)$ to prefer or avoid certain prototypes—provide an additional level of low-level control [2604.16076].

Interactive repair is also central to iCSNs. At test time, the model can project an input through its prototype assignment pipeline, present the user with prototypical reconstructions of each concept slot by decoding one-hot codes, and accept logical constraints that merge or prune prototype slots for curated subsets of inputs [2112.02290]. These constraints are implemented as simple $L_2$ penalties on relevant coordinates of the discrete code during fine-tuning [2112.02290].

ProtoPDebug extends the intervention idea to concept-level debugging of ProtoPNets [2205.15769]. Users inspect the top-activating training images for each prototype and label a prototype as confounded or high-quality; the corresponding image cut-outs are then stored in forbidden or valid concept sets [2205.15769]. Fine-tuning introduces a forgetting loss
$$
L_f(\theta) = \frac{1}{v}\sum_{y=1}^v \max_{p\in\mathcal{P}^y,\;c\in\mathcal{F}_y} a(p,c)
$$
and a remembering loss
$$
L_r(\theta) = -\frac{1}{v}\sum_{y=1}^v \min_{p\in\mathcal{P}^y,\;c\in\mathcal{V}_y} a(p,c),
$$
which are designed respectively to discourage activation on confounders and preserve valid concepts [2205.15769]. The paper frames this as concept-level rather than pixel-level debugging.

The ProtoPNet and concept-guided prototype literature also emphasizes faithful visualization. ProtoPNet explanations are case-based: each prediction can be traced to specific learned part-prototypes drawn from training data [2205.15769]. Concept-Guided ProtoPNet and ProtoPools report that each learned prototype can be visualized via its nearest training patch and overlaid on new inputs to show “this looks like that” explanations [2410.18705]. LCBM likewise treats the reshaped prototype–patch similarity map $A_k$ as a localization map for concept $k$, with thresholding or GradCAM used to visualize spatial support [2508.14562].

A plausible implication is that PGCMs are best understood not as a single architecture class, but as a family of mechanisms for making concept evidence inspectable at the level of prototypes, patches, or prototype-conditioned generations.

## 5. Representative empirical results

Reported results vary substantially because the models address different tasks, supervision regimes, and evaluation protocols.

For iCSNs on the Elementary Concept Reasoning benchmark, the reported quantitative findings are unusually strong for a weakly supervised concept learner [2112.02290]. The model attains near-zero latent-code variance for all concepts and outperforms $\beta$-VAE, Ada-VAE, Gaussian VAE with pair-wise averaging, and a categorical (Gumbel-softmax) VAE on that metric [2112.02290]. On linear probing of the learned one-hot-like code, iCSN achieves “≈ 100 % accuracy,” while all VAE baselines plateau at “70–98 %” [2112.02290]. Additional ablations report that replacing prototype discretization with categorical Gumbel-softmax “reduces performance drastically,” and replacing latent swapping by simply averaging pair encodings “also collapses performance” [2112.02290].

In the same iCSN study, online addition of novel concepts is explicitly quantified [2112.02290]. Introducing a new basic concept, half-circle, and binding it to an empty shape-slot via user feedback raises probing accuracy from “≈ 79 %” to “≈ 98 %,” while adding a new superordinate concept, spot/no-spot, by inserting a new read-out encoder and codebook pushes spot classification accuracy from “≈ 67 %” to “≈ 98.3 %” [2112.02290].

The 2023 multi-modal PGCM yields more mixed results [2303.04053]. On CUB-200–2011 with five zero-shot folds, the interpreter taught by one ground-truth description achieves acc@1 = 19.1%, acc@5 = 44.1%, mean rank = 30.0, whereas when taught by Gen-Ex descriptions it reaches acc@1 = 8.6%, acc@5 = 25.1%, mean rank = 46.0 [2303.04053]. In the direct comparison among generator variants, prototype-only Gen-Prot with beam search obtains BLEU-1 = 0.61, BLEU-4 = 0.55, CIDEr = 1.80, mean rank = 73.0, acc@1 = 2.7%, and acc@5 = 13.6%, while exemplar-based Gen-Ex with beam search achieves BLEU-1 = 0.64, BLEU-4 = 0.58, CIDEr = 1.92, mean rank = 46.0, acc@1 = 8.6%, and acc@5 = 25.1% [2303.04053]. The paper’s stated conclusion is that prototype-grounded generation can produce plausible category descriptions but “struggle[s] to convey fine-grained distinctions unless supported by exemplar information” [2303.04053].

For concept prediction with interpretable prototypes, Concept-Guided ProtoPNet achieves up to $\mathrm{Acc}_{\rm CUB}\approx0.880$ and $\mathrm{Acc}_{\rm AwA2}\approx0.885$, while Concept-Guided ProtoPools reaches up to $\mathrm{Acc}_{\rm CUB}\approx0.878$ and $\mathrm{Acc}_{\rm AwA2}\approx0.892$; the cited black-box oracle from Koh et al. (2020) is $\mathrm{Acc}_{\rm CUB}=0.961$ and $\mathrm{Acc}_{\rm AwA2}=0.901$ [2410.18705]. The discussion characterizes this as performance “close to black-box oracles (within ~5 %)” [2410.18705].

The 2026 PGCM for verifiable concept alignment reports dataset-specific comparisons against CBM-style baselines [2604.16076]. On ColorMNIST+, it achieves 99.2% concept accuracy and 99.7% task accuracy, described as “identical to CBM’s 99.2/99.6” [2604.16076]. On CelebA, it reports 78.5% concept accuracy versus 81.3% for CBM, and 83.0% task accuracy versus 84.0% for CBM [2604.16076]. On a noisy-label version of ColorMNIST+, model editing by dropping or relabeling 30% misaligned prototypes improves test concept accuracy from “≈93%” to “≈97%” [2604.16076].

LCBM focuses on concept localization as well as classification [2508.14562]. On CUB-200-2011, the reported numbers are: accuracy 75.2% versus 76.5% for VLG-CBM and 73.2% for LfCBM; explanation precision@10 of 76.6% versus 68.2% and 44.7%; inclusion 72.4 versus 65.3 and 53.8; mIoU 29.7 versus 28.3 and 22.7; REP 51.2 versus 37.6 and 30.6; and deletion ratio/difference 79.3%/0.177 versus baselines “≈ 87–90%/0.086–0.11” [2508.14562]. The conclusion given is that localization and explanation faithfulness improve while downstream accuracy is maintained [2508.14562].

For concept-level debugging, ProtoPDebug reports on three settings [2205.15769]. On a synthetic confounded dataset, ProtoPDebug matches the clean baseline’s test $F_1$, approximately 0.90, while IAIA-BL reaches approximately 0.76 even with 100% masks [2205.15769]. On a natural-confounder bird dataset, average test $F_1$ improves from 0.56 to 0.69 and Attribution Precision rises from 0.68 to 0.90 after two debugging rounds [2205.15769]. On confounded COVID-19 chest X-rays, test $F_1$ increases from approximately 0.26 to approximately 0.54 after four debugging rounds [2205.15769].

## 6. Limitations, debates, and extensions

The literature presents PGCMs as interpretable and intervenable, but it also records clear limitations.

In iCSNs, three limitations are stated directly [2112.02290]. First, a reconstruction-only loss “may struggle when concepts are very fine-grained or when pixel-perfect fidelity is not aligned with concept fidelity.” Second, the temperature schedule $\tau\downarrow0$ “requires careful tuning to avoid premature discretization or slot collapse.” Third, the Elementary Concept Reasoning benchmark is “synthetic and object-centric,” so real-world scenes with backgrounds, occlusions, or multiple objects require additional preprocessing such as “object proposals or slot-attention modules” [2112.02290].

The 2023 multi-modal PGCM raises a different issue: prototypes may be too abstract for fine-grained communicative grounding [2303.04053]. The paper identifies failure modes in description generation as “overly generic templates, omission of distinguishing attributes, and distributional shift from human-written descriptions,” and argues that there is “a clear mismatch between intrinsic NLG metrics (BLEU, CIDEr) and task-based communicative success” [2303.04053]. This is an explicit caution against equating fluent prototype-conditioned generation with effective concept communication.

Prototype-based concept predictors also inherit annotation and representation trade-offs. Concept-Guided Prototype Networks require concept annotations at train time, note that prototype projection ties prototypes to actual patches and “may not capture abstract shape variations,” and describe diffusion-based concept-pure image generation as “computationally intensive” [2410.18705]. The 2026 PGCM for concept alignment states a “capacity vs. interpretability trade-off”: more prototypes cover more nuance but increase cognitive load, and extending the framework to “continuous or relational concepts (beyond binary flags)” remains future work [2604.16076]. LCBM likewise depends on a concept vocabulary and uses foundation-model guidance to localize concepts, which suggests a hybrid regime rather than fully annotation-free concept grounding [2508.14562].

A further extension appears in one-shot concept learning with Abstracted Gaussian Prototypes [2408.17251]. There, a concept is represented by a GMM-derived set of topological subparts whose parameters support both one-shot classification and generation. The framework is described as “uniquely low in theoretical and computational complexity” and “completely standalone,” but it is also stated to be “not state-of-the-art” in classification accuracy and “restricted to 2-D binary stroke images” [2408.17251]. This suggests that prototype grounding can also be formulated generatively without large-scale pre-training, though current evidence is domain-specific.

Taken together, these limitations indicate that PGCMs do not solve concept alignment or interpretability in a single step. Rather, they shift the problem into a form in which concept semantics are grounded in prototypes, and therefore become inspectable, localizable, and, in several of the reported systems, directly modifiable.

Source: https://www.emergentmind.com/topics/prototype-grounded-concept-models-pgcms