ProtoPool: Interpretable Prototype-Based Classifier

• ProtoPool is a prototype-based interpretable neural network that globally shares prototypes via differentiable slot assignments.
• It employs a focal similarity measure and Gumbel-Softmax mechanism to enhance part localization and reduce redundant activations.
• Empirical evaluations reveal competitive accuracy on fine-grained image classification with an order-of-magnitude fewer prototypes than prior models.

ProtoPool is a prototype-based interpretable neural network for image classification. Distinct from standard prototypical-part models, ProtoPool introduces a fully differentiable mechanism for sharing a compact set of prototypes globally across all classes via class-specific “slots,” using Gumbel-Softmax prototype assignment and a focal similarity measure. This approach reduces prototype redundancy, eliminates post-hoc pruning, and achieves competitive or state-of-the-art accuracy—most notably on fine-grained recognition tasks—while providing explanations rooted in the presence of recognizable, learned parts. Its interpretability, training protocol, and empirical performance have been assessed comprehensively in both the original presentation and follow-up evaluations (Rymarczyk et al., 2021, Schlinge et al., 9 Jul 2025).

1. Motivation and Conceptual Foundations

ProtoPool was proposed to resolve inefficiencies and interpretability limitations in prior prototype models, particularly ProtoPNet. In the classic ProtoPNet framework, every class is allocated its own fixed set of prototypes, resulting in $C \times K$ total prototypes for $C$ classes and $K$ prototypes per class, with no mechanism for part-sharing. This design leads to scalability issues and significant redundancy, as many fine-grained classes (e.g., bird species) share similar visual parts. Furthermore, existing non-differentiable assignment and max-pooling similarity mechanisms hinder end-to-end optimization, force architectural workarounds like pruning or merging, and bias activations toward large background regions rather than foreground features.

ProtoPool's two major contributions are (a) the use of a single, global pool of prototypes, with class-specific “slots” that select from the pool through a fully differentiable assignment mechanism, and (b) the introduction of a focal similarity function to encourage the localization and distinctiveness of prototype activations. Unlike approaches relying on “negative reasoning” (prototypes indicating the absence of class-specific parts, as in ProtoTree), ProtoPool enforces strictly positive, part-based reasoning (“this looks like that”).

2. Architecture and Mechanisms

2.1 Model Structure

A ProtoPool network consists of:

1. Feature extractor: A convolutional backbone (e.g., ResNet-50 or ResNet-34) yielding spatial feature maps $f(x) \in \mathbb{R}^{H \times W \times D}$; the set of feature vectors is $Z_x = \{ z_i \in \mathbb{R}^D \}$.
2. Prototype pool layer: A set of $M$ global prototypes $P = \{ p_1, \dots, p_M \}$ and, for each class, $K$ “slots”. Each slot is associated with a learnable distribution $q_k \in \Delta^{M-1}$ (the simplex over prototypes), determining a (soft) allocation of prototypes to slots.
3. Classification layer: A linear classifier $h$ mapping the concatenated slot scores $C$0 to logits $C$1 for each class.

2.2 Prototype-Image Matching and Slot Aggregation

For an image $C$2, prototype-to-image similarity for prototype $C$3 is computed over all patches in $C$4:

$C$5

with $C$6 (“focal similarity”). Each slot $C$7 pools similarities across the global pool according to its weights:

$C$8

A class $C$9’s logit is computed via a positively initialized weight matrix $K$0:

$K$1

Typically, $K$2 is positive only for class $K$3, encoding positive part-based reasoning.

3. Differentiable Prototype Assignment and Regularization

ProtoPool enforces a unique, unambiguous assignment of prototypes to each slot using Gumbel-Softmax. For each slot $K$4, the pre-assignment vector $K$5 is transformed into $K$6 via:

$K$7

where $K$8 is i.i.d. Gumbel(0,1) noise, and $K$9 is an annealed temperature. As $f(x) \in \mathbb{R}^{H \times W \times D}$0, assignments converge to one-hot, and each slot selects exactly one prototype.

An orthogonality loss among slots for the same class prevents multiple slots from selecting the same prototype:

$f(x) \in \mathbb{R}^{H \times W \times D}$1

This ensures slot assignments are distinct within each class.

The total training loss combines:

• Cross-entropy classification loss ($f(x) \in \mathbb{R}^{H \times W \times D}$2)
• Cluster loss ($f(x) \in \mathbb{R}^{H \times W \times D}$3): Each prototype is pulled toward similar-class patches
• Separation loss ($f(x) \in \mathbb{R}^{H \times W \times D}$4): Prototypes are pushed away from other-class patches (typically negative coefficient)
• Orthogonality loss ($f(x) \in \mathbb{R}^{H \times W \times D}$5)
• Optionally, an $f(x) \in \mathbb{R}^{H \times W \times D}$6 regularization on classifier weights for sparsity

Formally,

$f(x) \in \mathbb{R}^{H \times W \times D}$7

with recommended settings $f(x) \in \mathbb{R}^{H \times W \times D}$8, $f(x) \in \mathbb{R}^{H \times W \times D}$9, $Z_x = \{ z_i \in \mathbb{R}^D \}$0, $Z_x = \{ z_i \in \mathbb{R}^D \}$1.

4. Shared Prototypes and Scalability

By using a global prototype pool, ProtoPool achieves a significant reduction in prototype count. For CUB-200-2011 (200 classes), ProtoPool requires $Z_x = \{ z_i \in \mathbb{R}^D \}$2 prototypes (with $Z_x = \{ z_i \in \mathbb{R}^D \}$3 per class), in contrast to 1,655–2,000 prototypes in class-specific allocation schemes (Rymarczyk et al., 2021).

The assignment mechanism, enforced by Gumbel-Softmax and orthogonality loss, automatically discovers both reusable (shared) and class-specific prototypes, thus eliminating the need for post-training prototype pruning or merging. At inference, slot distributions $Z_x = \{ z_i \in \mathbb{R}^D \}$4 are thresholded to obtain hard assignments.

This compactness provides computational and memory benefits and, in practice, allows for positive reasoning grounded in “this part” explanations, as each class slot selects a prototype on-demand.

5. Empirical Performance and Interpretability

5.1 Accuracy and Compactness

ProtoPool delivers competitive classification accuracy with far fewer prototypes than prior methods:

Dataset	Model	#Prototypes	Accuracy (%)
CUB-200-2011	ProtoPNet/ResNet34	1,655	79.5
	ProtoPool/ResNet34	202	80.3 ± 0.2
	TesNet/ResNet34	2,000	82.7
	ProtoTree/ResNet50	202	82.2 ± 0.7 (pretrained)
	ProtoPool/ResNet50	202	85.5 ± 0.1 (pretrained)
Stanford Cars	ProtoPNet/ResNet34	1,960	86.1 ± 0.2
	ProtoPool/ResNet34	195	89.3 ± 0.1
	TesNet/ResNet34	1,960	92.6 ± 0.3

ProtoPool achieves state-of-the-art or near-state-of-the-art performance while using an order-of-magnitude fewer prototypes (Rymarczyk et al., 2021).

5.2 Interpretability and Saliency

A user study assessed the saliency of ProtoPool’s prototypes by presenting overlays to human annotators; the average saliency score for focal similarity activations was $Z_x = \{ z_i \in \mathbb{R}^D \}$5 versus $Z_x = \{ z_i \in \mathbb{R}^D \}$6 for non-focal similarity and ProtoTree. A Mann–Whitney U test yielded $Z_x = \{ z_i \in \mathbb{R}^D \}$7, confirming the statistical significance of improved saliency (Rymarczyk et al., 2021).

Follow-up quantitative analysis applied interpretability metrics, reporting on compactness (e.g., 205 global prototypes, 96% sparsity in weights, NPR of 0.54), contrastivity (mean activation overlap 15.6%, mean entropy 0.91), and complexity (object overlap 2.0%, background overlap 84.8%), as well as continuity under perturbations (activation location change 2.6, rank change across 3.7 classes) (Schlinge et al., 9 Jul 2025).

5.3 Analysis and Limitations

Ablation experiments demonstrate that:

• Removing Gumbel-Softmax for assignment causes assignment collapse and a large accuracy drop (e.g., 64.5% on Cars).
• Omitting focal similarity marginally affects accuracy but leads to decreased interpretability.
• Removing orthogonality loss slightly reduces accuracy.

However, comprehensive evaluation also finds that ProtoPool prototypes often fire on background regions (high background overlap), lack consistent semantic alignment with object parts (low object overlap, moderate continuity and contrastivity), and that the reduction in prototype number can impair prototype specificity relative to alternatives like PIPNet (Schlinge et al., 9 Jul 2025).

6. Theoretical Guarantees

ProtoPool’s design is underpinned by two simple but strong theoretical properties:

• Gumbel-Softmax convergence: As $Z_x = \{ z_i \in \mathbb{R}^D \}$8, slot distributions become one-hot, so each slot selects exactly one prototype.
• Orthogonality constraint: When orthogonality loss is minimized, slots within a class are forced to select distinct prototypes.

These principles ensure that every class is represented by a fixed, unique set of prototypes at test time, obviating both ambiguity and redundancy that arise in prior hard assignment or softmax-based approaches (Rymarczyk et al., 2021).

7. Comprehensive Evaluation and Future Prospects

Extensive evaluations highlight that ProtoPool represents an important advancement in the scalable, interpretable, prototype-based paradigm. Its main strengths are strong accuracy with drastically fewer prototypes, end-to-end differentiable training, and more salient explanations (by human judgment) than earlier approaches. However, objective metrics reveal challenges with semantic alignment: prototypes often highlight background features and lack strong contrastivity. This suggests that further innovations—potentially architectural or in the loss design—are needed to improve the semantic coherence and stability of learned prototypes for truly human-centred interpretability, especially in fine-grained and real-world scenarios (Rymarczyk et al., 2021, Schlinge et al., 9 Jul 2025).