Post-hoc Self-explanation of CNNs

Abstract: Although standard Convolutional Neural Networks (CNNs) can be mathematically reinterpreted as Self-Explainable Models (SEMs), their built-in prototypes do not on their own accurately represent the data. Replacing the final linear layer with a $k$-means-based classifier addresses this limitation without compromising performance. This work introduces a common formalization of $k$-means-based post-hoc explanations for the classifier, the encoder's final output (B4), and combinations of intermediate feature activations. The latter approach leverages the spatial consistency of convolutional receptive fields to generate concept-based explanation maps, which are supported by gradient-free feature attribution maps. Empirical evaluation with a ResNet34 shows that using shallower, less compressed feature activations, such as those from the last three blocks (B234), results in a trade-off between semantic fidelity and a slight reduction in predictive performance.

• The paper proposes a k-means-based replacement for classifier prototypes to reinterpret CNNs as self-explanable models, improving prototype representativeness (cosine similarity from ~0.5 to ~0.9).
• It introduces a novel gradient-free, spatially consistent attribution mechanism that aggregates feature importance without relying on backpropagation.
• Empirical results on datasets like CUB-200 show that the KMEx method maintains classification performance (78.6% vs 79.1% accuracy) while significantly enhancing model interpretability.

Post-hoc Self-explanation of CNNs: Formalizing and Implementing K-Means Prototype Explanations

Introduction

Convolutional Neural Networks (CNNs) underpin state-of-the-art image classification systems but have been widely critiqued for their lack of interpretability. Although the architecture of standard CNNs appears opaque, this work demonstrates that these models can be mathematically reinterpreted as Self-Explainable Models (SEMs) by exploiting the commutativity of average pooling and linear classification operations. However, the class prototypes induced by standard classifier weights are not representative or diverse enough to facilitate meaningful explanations. The paper "Post-hoc Self-explanation of CNNs" (2603.28466) proposes a rigorous, unified formalism for post-hoc, k-means-based explanations at three critical model locations: the classifier, the encoder's final output, and combinations of intermediate activation blocks. This reinterpretation yields highly expressive, concept-based explanation maps and features a novel, fully gradient-free, spatially consistent attribution mechanism.

Formalism and Mechanistic Insight

The mathematical foundation establishes that the final linear layer of a CNN acts as a set of $C$ class prototypes (the classifier's columns). The average pooled encoder output plays the role of a data embedding projected onto these prototype directions. However, empirical evidence reveals that the resulting prototypes are poorly situated in the embedding space—being aligned with class separation (as demanded by cross-entropy loss) rather than representing data diversity. This misalignment is evident in the distribution of class and out-of-class cosine similarities.

To address this, the paper introduces a k-means-based replacement for the classifier: for each class, $K/C$ k-means centroids are computed from the class-specific embedding vectors in training data. Classification is performed by matching incoming data to its nearest prototype, improving both representativeness and class coverage, and making the model's “concepts” more semantically faithful.

A generalized SEM formulation then extends naturally to intermediate layers. Feature activation tensors from arbitrary combinations of encoder blocks (e.g., B4 or B234 in ResNet34) are linearly upsampled and normalized, concatenated, and then clustered via k-means. This approach enables explanations grounded in the spatial structure of learned features and supports concept-level segmentation maps and attributions.

Figure 1: Interpretation process for B4 (left) and B234 (right) on a CUB-200 red-bellied woodpecker, showing low- and high-resolution feature importance maps and representative training patches per prototype.

Concept Prototype Alignment and UMAP Analysis

A critical empirical finding pertains to the (mis)alignment between classifier prototypes and actual data points in the embedding space. Quantitative results indicate that the cosine similarity between standard classifier prototypes and their corresponding class embeddings is ~0.5. In contrast, k-means prototypes achieve cosines near 0.9, reflecting much better representativeness. ProtoPNet, with its specifically trained prototype layer, yields virtually perfect alignment but comes at the cost of complex training and degraded performance. This discrepancy is not only revealed numerically but illustrated via UMAP on the CUB-200 dataset, where k-means prototypes are embedded within their respective data clusters, while classifier columns are scattered and seldom coincide with any real data.

Figure 2: UMAP projection of training and test (lower opacity) embeddings from the twenty sparrow classes of CUB-200, visualizing the distribution and alignment of classifier and KMEx prototypes.

Gradient-free Feature Attribution and Explanation Maps

The method further proposes a gradient-free, segment-aware attribution mechanism. Downstream feature importance is defined as a normalized dot-product between each feature patch and the associated prototype, yielding attribution maps analogous to CAM but without relying on gradients or class backpropagation. Instead, attribution can be aggregated spatially, respecting segmentation induced by prototype assignments. Incorporation of shallow and deep feature activations (e.g., B234) increases spatial resolution and fidelity, as shown in detailed explanation maps. Representative training patches associated with each prototype further substantiate the assignments and reinforce semantic interpretability.

Empirical Evaluation

On MNIST, STL10, and CUB-200, replacing the classifier with a k-means-based proxy (KMEx) results in no loss of classification performance relative to the original CNN (test accuracy on CUB-200: 79.1% for CNN, 78.6% for KMEx with $K/C=5$). The B4 model, relying solely on the encoder's output block (as in ProtoPNet), similarly matches backbone accuracy, demonstrating that post-hoc concept-based explanations need not sacrifice predictive power. Inclusion of more shallow blocks (B234) enables fine-grained explanations but causes a moderate drop in accuracy (52.8% on CUB-200 test), elucidating a trade-off between explanation granularity and classification strength. Comparisons with ProtoPNet and matrix-factorization models reinforce the efficiency and reliability of the post-hoc, k-means-based method.

Contextualization and Theoretical Implications

This framework unifies disparate concept-based explanation methodologies, avoiding user-defined concepts or reliance on gradients/perturbations. It provides a theoretical justification for the self-explainability of standard CNNs (via commutativity arguments) and supplies concrete post-hoc procedures that require no model retraining. Unlike prior work (e.g., TCAV, ACE, Network Dissection), explanations here are strictly data-driven, segmentation-aware, and entirely coherent within the learned representation space.

The results imply that the “black box” opacity of CNNs stems more from a lack of suitable post-hoc analysis than from architectural constraints. By revealing explicit concept attribution pathways and validating the empirical fidelity of alternatives to classifier-based prototypes, this work points to practical routes for explainable deep learning that do not require architectural interventions or retraining.

Implications, Limitations, and Future Directions

Practically, this approach offers scalable, model-agnostic interpretability mechanisms that can be applied post-hoc to any existing CNN, immediately boosting transparency for high-stakes domains without perturbing model deployment pipelines. The primary scalability issue concerns k-means clustering over exceedingly large training datasets; preliminary results suggest that prototype quality is maintained on representative subsets, but further work on efficient, distributed clustering is warranted.

Theoretically, the commutative reinterpretation bridges modern deep vision models and the long-standing study of prototype-based learning, suggesting new avenues for theory-informed explanation research. Empirically, the framework enables systematic studies on the fidelity and compactness of concept activation spaces, as well as their links to model robustness and generalization.

Future developments could explore adaptive selection of prototype density, integration with active learning or curriculum strategies, or extensions to more complex, non-visual data modalities.

Conclusion

This paper establishes a robust, formal connection between CNN architectures and self-explainable prototype models, introducing a comprehensive post-hoc framework for concept-based explanation. Utilizing k-means clustering to obtain representative prototypes for both classifier and intermediate features, the approach delivers high-fidelity, spatially resolved explanation maps and faithful, gradient-free attribution. Empirical results validate that interpretability gains do not compromise predictive accuracy for backbone-aligned models. These findings challenge the notion of CNNs as irreducibly black-box and position post-hoc, k-means-based interpretability as a practical, theoretically grounded solution for transparent AI in vision and beyond.