The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations (2106.00786v2)

Abstract: Feature importance (FI) estimates are a popular form of explanation, and they are commonly created and evaluated by computing the change in model confidence caused by removing certain input features at test time. For example, in the standard Sufficiency metric, only the top-k most important tokens are kept. In this paper, we study several under-explored dimensions of FI explanations, providing conceptual and empirical improvements for this form of explanation. First, we advance a new argument for why it can be problematic to remove features from an input when creating or evaluating explanations: the fact that these counterfactual inputs are out-of-distribution (OOD) to models implies that the resulting explanations are socially misaligned. The crux of the problem is that the model prior and random weight initialization influence the explanations (and explanation metrics) in unintended ways. To resolve this issue, we propose a simple alteration to the model training process, which results in more socially aligned explanations and metrics. Second, we compare among five approaches for removing features from model inputs. We find that some methods produce more OOD counterfactuals than others, and we make recommendations for selecting a feature-replacement function. Finally, we introduce four search-based methods for identifying FI explanations and compare them to strong baselines, including LIME, Anchors, and Integrated Gradients. Through experiments with six diverse text classification datasets, we find that the only method that consistently outperforms random search is a Parallel Local Search (PLS) that we introduce. Improvements over the second-best method are as large as 5.4 points for Sufficiency and 17 points for Comprehensiveness. All supporting code for experiments in this paper is publicly available at https://github.com/peterbhase/ExplanationSearch.

• The paper demonstrates how traditional feature removal methods create out-of-distribution inputs that can lead to misleading model explanations.
• It introduces a counterfactual training approach combined with novel search strategies to improve the fidelity of feature importance explanations.
• Comparative analysis shows that the Parallel Local Search method outperforms random search based on key metrics like sufficiency and comprehensiveness.

Addressing the Out-of-Distribution Problem in Explainability through Rigorous Evaluation and Search Methods

The paper "The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations" by Hase, Xie, and Bansal provides a critical examination of the feature importance (FI) methodologies used to explain model predictions, with a focus on the out-of-distribution (OOD) issue, which occurs when perturbing input features for explanation purposes. The authors highlight how this common practice can result in explanations that are misaligned with social expectations because these perturbations create inputs that the model has not been trained on. To address this, they propose a modification to the training procedure and introduce novel search-based methods for enhancing FI explanations.

Key Contributions

1. OOD Problem in FI Explanations: The authors begin by articulating the issue with removing input features during explanation generation or evaluation, emphasizing that the resulting counterfactual inputs do not match the distribution on which models were trained. This misalignment means that explanations could be misleading, as they reflect the random initialization and model priors rather than learned relationships from the data.
2. Proposed Solutions: To mitigate the OOD issue, the authors suggest an adjustment to the training process: exposing models to counterfactual inputs during training. This strategy, termed Counterfactual Training, aims to align models' prediction behavior with what will be encountered during explanation generation and hence reduce misalignment.
3. Comparative Analysis of Feature Removal Strategies: Several methods for feature removal are evaluated across various metrics to determine their impact on creating OOD inputs. The authors compare techniques like using MASK tokens, attention masking, complete feature removal (Slice Out), zero-vector embeddings, and marginalizing predictions. Based on this analysis, recommendations are provided on when and how to use specific Replace functions.
4. Advanced Search Methods for FI Explanation: In addition to exploring the OOD issue, the paper innovatively contributes to developing search-based strategies for identifying optimal FI explanations. A novel Parallel Local Search (PLS) technique is introduced, which consistently outperformed random search techniques in experiments, offering improvements in Sufficiency and Comprehensiveness metrics across diverse text classification tasks.

Implications and Future Work

The implications of this paper are profound for both theoretical and practical aspects of AI interpretability. On a theoretical level, addressing the OOD problem ensures that explanations are genuinely indicative of the information learned by the model, thus improving the trustworthiness and utility of AI explanations in decision-making processes. Practically, improved search-based methods for FI explanations enhance the efficiency of identifying relevant explanations, reducing the time and computational overhead typically associated with model interpretability tasks.

Future research can explore the scalable implementation of Counterfactual Training methodologies, as well as the adaptation of similar strategies across different model architectures or datasets. There are also opportunities to investigate the integration of FI explanations with causal inference methods to further distill model behavior under real-world conditions.

Overall, by refining both the generation and evaluation of FI explanations, this work sets a critical foundation for advancing model transparency and interpretability in AI systems, aiding developers, users, and stakeholders in understanding and trusting complex machine learning models.