Learning how to explain neural networks: PatternNet and PatternAttribution

Abstract: DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.

• The paper introduces two novel explanation methods that address theoretical flaws in existing neural network explanation techniques.
• It reveals that traditional methods misalign network weights with actual signal directions, challenging their reliability even in linear models.
• Empirical tests on VGG-16 with ImageNet show that PatternNet and PatternAttribution produce clearer signal visualizations and more accurate attributions.

Explaining Neural Networks: PatternNet and PatternAttribution

The paper "Learning how to explain neural networks: PatternNet and PatternAttribution" critically examines existing explanation techniques for deep neural networks and introduces two new methods aimed at providing theoretically sound explanations for these models. The main contributions include a critique of prevalent methods such as DeConvNet, Guided BackProp, and Layer-wise Relevance Propagation (LRP), alongside the introduction of PatternNet and PatternAttribution, which are posited to address identified shortcomings.

Overview

Existing methods for explaining neural networks, including saliency maps, DeConvNet, Guided BackProp, and LRP, operate on the premise that it is possible to trace back the output signal through the network to highlight relevant input features responsible for the model's decision. However, the authors argue that these methods may not generate theoretically correct explanations even for a simple linear model. They propose that a robust explanation method should reliably handle the simplest case: linear models.

Theoretical Foundation

The authors emphasize that many current explanation approaches assume the direction of the network weight vector aligns with the signal in the data. By scrutinizing the behavior of linear models, the authors demonstrate that the weight vector often appropriates a direction primarily aimed at filtering out distractors, rather than aligning with the signal direction. This indicates potential misalignment in deeper, non-linear models, raising concerns about the validity of contemporary explanation methods.

Proposed Methods

Based on theoretical analysis, the authors introduce:

1. PatternNet: This technique aims to rectify the limitations of DeConvNet and Guided BackProp by ensuring that visualizations approximate the actual signal detected by neurons. PatternNet estimates a signal direction vector for each neuron, supposed to be more indicative of the features the network detects.
2. PatternAttribution: This method builds upon PatternNet by focusing on the attributions, offering a refined mechanism for mapping the proportion of signal components contributing to the network's output.

Empirical Evaluation

The authors validate their propositions through rigorous empirical evaluations on the VGG-16 model with ImageNet data. Their proposed methods are compared against traditional approaches using several criteria, demonstrating improved performance both qualitatively and quantitatively. PatternNet provides clearer signal visualizations, while PatternAttribution yields significantly enhanced pixel-wise attributions. These enhancements are assessed through correlation measures and image degradation experiments, where PatternNet and PatternAttribution show superior performance in maintaining information fidelity and interpretability.

Implications and Future Directions

This analysis underlines the critical need to assess the underlying assumptions of explanation techniques for neural networks. By addressing the theoretical discrepancies, the paper sets a foundation for more reliable interpretation tools. Future work could extend these concepts to a wider variety of network architectures and explore using these improved explanations to debug and optimize neural networks.

In conclusion, PatternNet and PatternAttribution present a significant step forward in neural network interpretability by aligning theoretical validation with empirical assessments. These methods represent a crucial evolution towards robust models' transparency, which is paramount for AI systems' deployment in critical and high-stakes environments.