This Looks Like That: Deep Learning for Interpretable Image Recognition (1806.10574v5)

Abstract: When we are faced with challenging image classification tasks, we often explain our reasoning by dissecting the image, and pointing out prototypical aspects of one class or another. The mounting evidence for each of the classes helps us make our final decision. In this work, we introduce a deep network architecture -- prototypical part network (ProtoPNet), that reasons in a similar way: the network dissects the image by finding prototypical parts, and combines evidence from the prototypes to make a final classification. The model thus reasons in a way that is qualitatively similar to the way ornithologists, physicians, and others would explain to people on how to solve challenging image classification tasks. The network uses only image-level labels for training without any annotations for parts of images. We demonstrate our method on the CUB-200-2011 dataset and the Stanford Cars dataset. Our experiments show that ProtoPNet can achieve comparable accuracy with its analogous non-interpretable counterpart, and when several ProtoPNets are combined into a larger network, it can achieve an accuracy that is on par with some of the best-performing deep models. Moreover, ProtoPNet provides a level of interpretability that is absent in other interpretable deep models.

• The paper presents ProtoPNet, which leverages prototypical parts for transparent decision-making in image classification while preserving strong performance.
• It combines a CNN for feature extraction with a prototype layer that computes similarity scores via squared L2 distances and a projection step for interpretability.
• Empirical results on datasets like CUB-200-2011 and Stanford Cars reveal competitive accuracy, underscoring its potential for trustworthy AI applications.

Interpretability in Image Classification: An Analysis of Prototypical Part Networks

This paper introduces a new deep learning architecture termed Prototypical Part Network (ProtoPNet), optimized for interpretable image classification tasks. Provably comparable in accuracy to its non-interpretable counterparts, ProtoPNet aims to elucidate the rationale behind the classification decisions it makes. Through the lens of prototypes, it draws analogies to how humans such as ornithologists or radiologists classify images by examining prototypical parts.

Core Approach and Methodology

ProtoPNet consists of several foundational components: a conventional convolutional neural network (CNN), a prototype layer, and a fully connected (FC) layer. This architecture is driven by prototypes that are latently derived from training data images. The essential innovation lies in the way the network utilizes these prototypes.

1. Feature Extraction: The CNN component, initialized with pre-trained models like VGG, ResNet, or DenseNet, processes and extracts intricate features from the image data.
2. Prototype Layer: Following feature extraction, the prototype layer computes squared $L^2$ distances between learned prototypes and the patches of the feature map. The process involves inverting these distances to deduce similarity scores which are crucial for the final prediction. The latent space is shaped by clustering similar patches around the prototypes, significantly influencing how new examples are classified based on these learned representations.
3. Projection and Optimization: A critical step involves projecting prototypes onto the nearest latent training patches to ensure they are interpretable. This projection, combined with convex optimization of the FC layer weights, allows the network to maintain high interpretability without sacrificing accuracy.

Numerical Results and Accuracy

The experimentation on datasets such as CUB-200-2011 for bird species identification and the Stanford Cars dataset showed promising results.

• On the CUB-200-2011 dataset, ProtoPNet combined with various CNN backbones yielded accuracies ranging from 76.1% to 80.2%, with the best-performing combined network (VGG19, ResNet34, DenseNet121-based) achieving 84.8% on cropped images.
• ProtoPNet's performance on the Stanford Cars dataset was on par with state-of-the-art models, reaching an accuracy of 91.4%.

Implications and Interpretability Comparison

ProtoPNet's approach to deep learning interpretability is distinct from post hoc methods and attention-based models. While methods such as class activation maps (CAM) offer object-level or part-level interpretability by highlighting regions of interest, ProtoPNet adds another dimension by pointing to prototypical parts learned during training. This built-in case-based reasoning is a significant step forward, empowering users with more granular and faithful explanations of the model's decision-making process.

Future Directions

Future work might delve into several avenues:

• Hyperparameter Optimization: While the paper discusses hyperparameters such as the number of prototypes per class and the architecture configuration, further exploration could optimize these settings across varied datasets.
• Expanding Application Domains: Extending ProtoPNet to other classification tasks, including medical imaging or biometric identification, could validate the model's adaptability and robustness.
• Enhanced Training Techniques: Integrating advanced training protocols such as self-supervised learning could improve the efficiency and accuracy of the model.

Conclusion

Prototypical Part Networks represent a noteworthy advancement in interpretable AI. By integrating human-like reasoning into the classification process—this looks like that—ProtoPNet not only maintains robust predictive performance but also offers transparent, interpretable insights into its decision-making. These strides hold significant potential for applications demanding both accuracy and interpretability, underpinning the evolving landscape of reliable AI systems.

The full paper detailing ProtoPNet, including source code, is available here.