Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps (1312.6034v2)

Abstract: This paper addresses the visualisation of image classification models, learnt using deep Convolutional Networks (ConvNets). We consider two visualisation techniques, based on computing the gradient of the class score with respect to the input image. The first one generates an image, which maximises the class score [Erhan et al., 2009], thus visualising the notion of the class, captured by a ConvNet. The second technique computes a class saliency map, specific to a given image and class. We show that such maps can be employed for weakly supervised object segmentation using classification ConvNets. Finally, we establish the connection between the gradient-based ConvNet visualisation methods and deconvolutional networks [Zeiler et al., 2013].

• The paper introduces visualization techniques that generate both class-representative images and saliency maps to elucidate ConvNet decision-making.
• It optimizes a class score with L2 regularization and computes input gradients to pinpoint influential regions in the image.
• Empirical results on the ILSVRC-2013 dataset demonstrate that these methods enable effective weakly supervised object localization with competitive performance.

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

This essay examines the paper "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps," authored by Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, affiliated with the Visual Geometry Group at the University of Oxford. This investigation primarily explores visualizing image classification models trained with deep Convolutional Networks (ConvNets) and explores two specific visualization techniques: generating class-representative images and computing image-specific saliency maps.

Visualisation Techniques

Class Model Visualisation

The first visualization technique aims to generate images that are representative of specific classes as understood by the ConvNet. This method extends upon previous work by Erhan et al., optimizing the input image to maximize the score for a particular class. Unlike earlier scenarios involving unsupervised deep architectures, the supervised nature of the ConvNet training enables clear associations between specific neurons in the final fully connected layer and corresponding classes.

Formally, the class score function $S_c(I)$ for a given input image $I$ and class $c$ is maximized along with an $L_2$-regularization term to generate a representative image: $\arg\max_I S_c(I) - \lambda \|I\|_2^2,$ where $\lambda$ is a regularization parameter. This approach yields images that capture various aspects of class appearances as encoded by the ConvNet, providing insightful visual representations of the class-specific feature learning.

Image-Specific Class Saliency Maps

The second technique calculates image-specific class saliency maps, which highlight the spatial regions in an image that most influence the ConvNet's classification decision for a given class. By computing the gradient of the class score with respect to the input image, the method exploits the first-order Taylor expansion about the image: $$w = \left . \frac{\partial S_c}{\partial I} \right|_{I_0},$$ where $w$ represents the derivative of the class score $S_c$ with respect to image $I$ at a specific image $I_0$. This derivative serves as a linear approximation to identify influential pixels, offering a mechanism to generate saliency maps without additional annotation beyond image labels.

The saliency maps derived via back-propagation can be further utilized for weakly supervised object localization. By initializing GraphCut segmentation based on high saliency regions, it is feasible to segment objects within an image, a capability validated by the authors on the ILSVRC-2013 dataset.

Implementation and Empirical Validation

The visualizations were conducted using a deep ConvNet architecture similar to that of AlexNet, trained on the ILSVRC-2013 dataset. The network configuration and training procedures bear resemblance to standard practices but incorporate additional image jittering and a less wide architecture resulting in a top-1/top-5 validation error of 39.7%/17.7%.

Through qualitative evaluation, class model visualizations and image-specific class saliency maps provided interpretable insights into the types of features and spatial regions deemed important by the ConvNet. Moreover, quantitative evaluations in weakly supervised object localization demonstrated competitive performance, achieving a 46.4% top-5 error rate on the ILSVRC-2013 test set.

Theoretical and Practical Implications

The paper establishes a connection between gradient-based visualization and the DeconvNet architecture proposed by Zeiler et al. While DeconvNets reconstruct the input of each layer from its output, this paper demonstrates that similar reconstructions can be perceived through the lens of computing gradients, underscoring the linkage and potential generalizations to any layer within the ConvNet, including fully connected layers.

Practically, these visualization techniques present opportunities to enhance interpretability in deep learning models, providing researchers and practitioners with tools to probe the inner workings of ConvNets. The saliency maps, in particular, enable weakly supervised object localization without necessitating dedicated segmentation models, offering a path to more efficient and scalable AI systems.

Conclusion and Future Directions

The paper contributes valuable methods to the visualization and understanding of deep ConvNets in the context of image classification. By generating class-representative images and image-specific saliency maps, it enriches the toolkit available for probing ConvNet behavior and explicating their decision-making processes.

Future research may explore integrating image-specific saliency maps more directly into learning formulations, potentially yielding improved training regimes and more robust models. Further investigation into extending these visualization techniques to other neural network architectures and applications could also provide broader insights and advancements in the field of deep learning.