Understanding Neural Networks through Representation Erasure (1612.08220v3)

Abstract: While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

• The paper introduces representation erasure as a method to expose the contributions of individual neural network units in decision-making.
• It employs log-likelihood differences and reinforcement learning to identify both critical and counterproductive components across NLP tasks.
• The study demonstrates that targeted erasure not only clarifies model behavior but also informs improvements in neural architecture design.

Neural Network Interpretability via Representation Erasure

The paper, "Understanding Neural Networks through Representation Erasure" by Jiwei Li, Will Monroe, and Dan Jurafsky, introduces a methodological framework to enhance the interpretability of neural networks, specifically focusing on NLP tasks. This work confronts the challenges posed by the opaque nature of neural networks, often described as "black boxes," by dissecting the contributions of individual units within these networks to their overall decision-making processes.

Key Contributions

The paper proposes a systematic approach to interpret neural network decisions by eradicating certain components of the representation to observe how these changes impact model outcomes. This erasure is performed at different levels, including input word vectors, intermediate hidden units, and input phrases. By quantifying both the detrimental and beneficial impacts of such erasures, the researchers identify crucial aspects and potential pitfalls within neural architectures.

Methodology

Two primary approaches are presented for evaluating the impacts of erasure:

1. Calculation of Log-Likelihood Differences: The simplest method involves measuring the difference in log likelihoods on gold-standard labels when specific representations are removed.
2. Reinforcement Learning (RL) Model: A more advanced approach employs reinforcement learning to determine the smallest set of words necessary to reverse a model's decision.

These methodologies provide insights into how neural networks select input vector dimensions for linguistic features, prioritize words and phrases in sentiment analysis, and highlight the efficiencies of architectures like LSTMs over RNNs.

Empirical Analysis

The framework is extensively tested across multiple tasks, ranging from morphology to sentence-level sentiment analysis. The experiments reveal:

• Dimensional Importance: For tasks like POS tagging and chunking, specific vector dimensions are consistently more critical, with dropout techniques helping distribute importance across dimensions.
• Word-Level Insights: In sentiment analysis, models such as Bi-LSTM exhibit a higher sensitivity to sentiment-related terms than simpler models like RNNs.
• Negative Importance: Interestingly, some word removals improve decision accuracy, unveiling potential areas for model error analysis.

Results and Implications

• Numerical Performance: In investigating word embeddings (GloVe and word2vec), certain dimensions dominate task performance. With erasure, it's shown that models can maintain performance by redistributing focus across other dimensions.
• Word Sensitivity: Sentiment models, especially Bi-LSTMs, demonstrate heightened sensitivity to sentiment-indicative terms, outperforming tree-based models on certain datasets.
• Error Analysis: Identifying words with negative importance helps pinpoint common points of confusion for models, aiding in refining future model architectures.

Theoretical and Practical Implications

The methodologies introduced for representation erasure enable more transparent neural models, facilitating error analysis and model refinement. This can significantly enhance the effectiveness and reliability of NLP systems across various applications.

Future Developments

This work opens avenues for further extensions in interpretability studies. One promising direction is enhancing adversarial training techniques to improve model robustness without sacrificing interpretability. Moreover, exploring erasure effects in other domains beyond NLP could reinforce generalization abilities.

In conclusion, this paper presents a significant step forward in understanding and demystifying neural network behaviors through strategic component erasure, providing a promising toolkit for researchers aiming to decode the intricacies of complex neural architectures.