A Benchmark for Interpretability Methods in Deep Neural Networks (1806.10758v3)

Abstract: We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.

An Evaluation Framework for Interpretability in Deep Neural Networks

The paper "A Benchmark for Interpretability Methods in Deep Neural Networks" introduces a novel empirical framework, termed ROAR (RemOve And Retrain), designed to evaluate interpretability methods that estimate feature importance in deep neural networks. The primary focus is on determining the accuracy of feature importance estimates through extensive experimentation on large-scale datasets. The paper identifies significant issues with existing interpretability methods and provides insights into the performance of various approaches.

Summary

The authors tackle a critical challenge in machine learning: reliably measuring feature importance in model predictions, especially when ground truth is unavailable. Traditional approaches often face drawbacks due to distribution shifts when features are removed directly from datasets without subsequent model retraining. The ROAR methodology ameliorates this issue by retraining models after perturbing input features deemed important by interpretability methods, thus ensuring the evaluation is distribution-consistent.

Methodology

ROAR involves ranking input features by estimated importance and systematically replacing those deemed most significant. New models are then trained on these modified datasets, allowing for direct comparison of prediction accuracies to determine the reliability of the feature importance estimates. The use of large datasets such as ImageNet, Food 101, and Birdsnap enables the authors to provide robust and generalizable conclusions.

Key Findings

1. Resilience to Input Removal: Models trained on modified datasets exhibited resilience, maintaining reasonable accuracy even with up to 90% of inputs removed. This indicates that only a subset of features significantly influences decision-making.
2. Comparison of Interpretability Methods: Base methods like Gradient Heatmaps and Integrated Gradients performed no better than random assignments in determining core informative features. Surprisingly, ensemble methods such as SmoothGrad provided little improvement and sometimes resulted in worse performance.
3. Effective Ensemble Methods: The paper highlights the exceptional performance of VarGrad and SmoothGrad-Squared, which significantly outperformed other methods in terms of accuracy after important feature removal. These methods combine estimates to construct more reliable explanations, suggesting ensemble approaches could be crucial for effective interpretable AI.
4. Limitations of Simple Approaches: Methods relying solely on single or non-ensemble estimates are generally less effective, emphasizing the need for more sophisticated techniques to capture feature importance accurately.

Implications and Future Work

The results call for a reevaluation of commonly used interpretability techniques given their performance relative to ROAR. For practical applications, especially those involving critical decision-making like healthcare or autonomous driving, relying on less effective interpretability could lead to misleading insights. The authors recommend further work on developing methods that harness ensemble strategies effectively and encourage exploration into why specific ensemble techniques demonstrate superior performance.

The adaptability of ROAR to diverse model architectures and feature types also presents a promising avenue for future research. Extending this framework to other machine learning paradigms could lead to breakthroughs in understanding model interpretability across a broad spectrum of use cases.

Conclusion

This comprehensive paper underscores the significant gap between existing interpretability methods and their assumed effectiveness. VarGrad and SmoothGrad-Squared emerge as promising paths forward, suggesting the potential benefits of ensembling strategies in feature attribution tasks. The rigorous evaluation through ROAR provides an empirical basis for advancing interpretability research and sets a precedent for future inquiries into the functional transparency of machine learning systems.