Do Feature Attribution Methods Correctly Attribute Features? (2104.14403v2)

Abstract: Feature attribution methods are popular in interpretable machine learning. These methods compute the attribution of each input feature to represent its importance, but there is no consensus on the definition of "attribution", leading to many competing methods with little systematic evaluation, complicated in particular by the lack of ground truth attribution. To address this, we propose a dataset modification procedure to induce such ground truth. Using this procedure, we evaluate three common methods: saliency maps, rationales, and attentions. We identify several deficiencies and add new perspectives to the growing body of evidence questioning the correctness and reliability of these methods applied on datasets in the wild. We further discuss possible avenues for remedy and recommend new attribution methods to be tested against ground truth before deployment. The code is available at https://github.com/YilunZhou/feature-attribution-evaluation

• The paper introduces a novel dataset modification technique that embeds ground truth to systematically assess feature attribution methods.
• The paper finds that saliency maps, attention mechanisms, and rationale models each exhibit limitations in reliably identifying manipulated features.
• The paper highlights the need for improved attribution frameworks to enhance model interpretability and address spurious correlations in ML.

Evaluation of Feature Attribution Methods in Machine Learning

The paper "Do Feature Attribution Methods Correctly Attribute Features?" by Zhou et al. rigorously investigates the adequacy of feature attribution methods in machine learning. These methods, which assign importance scores to input features relative to model predictions, are widely adopted in model interpretability, particularly in applications where understanding the model's decision process is critical. The paper addresses a significant gap in the literature by evaluating these methods systematically through ground truth attribution—a task notably complicated by the absence of explicit ground truth in practical datasets.

Methodology

The authors introduce a novel dataset modification technique that allows them to embed ground truth feature attributions into semi-natural datasets. This is achieved by modifying datasets with controlled input manipulations and label reassignments that ensure reliance on certain features for achieving high prediction accuracy. They then evaluate three popular attribution methods: saliency maps, attention mechanisms, and rationale models, against these modifications.

Key to the methodology is the use of various image and text datasets, modified to correlate specific features heavily with output labels. For instance, they perturb bird species images with watermarks or color shifts and alter reviews in the BeerAdvocate dataset by manipulating article words ('a' vs. 'the') based on reassigned labels. These controlled settings permit the authors to ascertain whether feature attribution methods can correctly identify the contributions of manipulated features, thus establishing ground truth for evaluating these methods.

Findings

Zhou et al. identify several deficiencies in existing feature attribution methods through a series of experiments:

1. Saliency Maps: The evaluation reveals that while methods such as SHAP show relatively better performance, none of the saliency map methods reliably allocate high attribution scores to the manipulated regions. Furthermore, these maps often fail to reflect increasing reliance on manipulated features as models' accuracy improves.
2. Attention Mechanisms: Contrary to expectations, attention weights do not significantly highlight correlating features and tend to exhibit high variance across different training runs. This questions their reliability as attribution values, indicating that attention may not consistently focus on semantically important features reflective of the model's decision process.
3. Rationale Models: While rationale models ensure a causal link between input features and predictions, they can include spurious features, thereby complicating model interpretability. The reinforcement learning-based rationale models are particularly prone to selecting misleading non-correlating features.

Implications and Future Directions

The implications of these findings are multifold. Practically, the paper casts doubt on the robustness of current attribution methods in real-world settings, where models might exploit spurious correlations not immediately apparent. Theoretically, it challenges the assumptions underlying these methods, urging researchers to develop new attribution frameworks that can be rigorously validated against known ground truths.

In terms of future directions, the paper advocates for the integration of generative modeling techniques in dataset augmentation to create more naturalistic test cases. These could help bridge the gap between semi-natural datasets with known manipulations and genuinely novel features that a model might exploit, thereby enhancing the utility of attribution methods in aiding scientific discovery and debugging machine learning models.

Conclusion

This paper contributes significantly to the ongoing discourse on interpretable AI by providing a structured approach to attribute evaluation. It underscores the necessity of developing new methodologies that can better quantify feature importance in models, especially when these are critical to understanding the model's predictions in high-stakes applications.