Red Teaming Deep Neural Networks with Feature Synthesis Tools (2302.10894v3)

Abstract: Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using \emph{feature synthesis} methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools on debugging tasks. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when an interpretation is correct. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation. A website for this paper and its code is at https://benchmarking-interpretability.csail.mit.edu/

• The paper introduces a benchmark task by implanting 12 engineered trojans in neural networks to evaluate interpretability tools.
• It demonstrates that state-of-the-art feature attribution methods struggle to detect flaws, often performing no better than simple baselines.
• The study shows that advanced feature synthesis techniques, including two novel variants, significantly enhance model debugging in OOD contexts.

Red Teaming Deep Neural Networks with Feature Synthesis Tools

In "Red Teaming Deep Neural Networks with Feature Synthesis Tools," the authors address a prevalent challenge in AI interpretability: understanding model behavior in out-of-distribution (OOD) contexts. While interpretability tools are purported to help identify model bugs, their practical success is limited by their dependency on datasets the user can sample. This paper evaluates such interpretability tools by benchmarking them against tasks involving human-interpretable trojans implanted in neural networks — a novel method for evaluating their efficacy in detecting flaws analogous to OOD bugs.

The paper begins by identifying a gap in the current understanding of interpretability tools, which often analyze model behavior using pre-existing datasets. This limitation constrains the user's ability to diagnose how a model reacts to unseen or unexpected features. The authors address this by proposing an evaluation methodology that involves implanting trojans into models and using interpretability tools to detect these intentional flaws. These trojans serve as a ground truth for validating interpretability methods.

The paper outlines four main contributions:

1. Introducing a benchmark task for interpretability tools centered on discovering trojans, involving 12 different trojans across three types.
2. Demonstrating the difficulty faced by 16 state-of-the-art feature attribution and saliency tools in identifying these trojans, even with data access to the trojan triggers illustrating a significant shortcoming.
3. Evaluating seven feature synthesis methods, highlighting their relative success compared to attribution tools.
4. Developing and assessing two new variants of the best-performing feature synthesis method, demonstrating their potential to enhance model debugging.

The experiments revealed that feature attribution and saliency tools struggle significantly with the task, often performing no better than an edge-detection baseline despite access to pertinent data. However, feature synthesis tools show more promise, suggesting they may be more effective in deploying targeted model debugging strategies.

The innovative approach of embedding human-interpretable trojans in neural networks provides a robust framework for benchmarking interpretability tools, pushing towards an understanding that transcends just performance metrics and ventures into the qualitative field of model errors and biases.

Several implications arise from this research. Practically, the findings denote a need for improved diagnostic tools that don't rely on pre-existing datasets, offering potential pathways for developing more comprehensive interpretability frameworks. Theoretically, it challenges the community to revisit the definitions and goals of interpretability in AI systems, pushing towards benchmarks that can accommodate diverse real-world deployment scenarios.

Future research could expand this work by integrating these findings with natural LLMs. Additionally, the exploration of automated diagnostics beyond human evaluation, perhaps integrating more advanced AI systems for assessment, could significantly fuel advancements in AI safety and reliability. The challenge remains to develop an interpretability toolkit rather than a singular solution, emphasizing the adaptability to multifaceted real-world applications and model architectures.