Evaluating the Utility of Model Explanations for Model Development (2312.06032v1)
Abstract: One of the motivations for explainable AI is to allow humans to make better and more informed decisions regarding the use and deployment of AI models. But careful evaluations are needed to assess whether this expectation has been fulfilled. Current evaluations mainly focus on algorithmic properties of explanations, and those that involve human subjects often employ subjective questions to test human's perception of explanation usefulness, without being grounded in objective metrics and measurements. In this work, we evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. We conduct a mixed-methods user study involving image data to evaluate saliency maps generated by SmoothGrad, GradCAM, and an oracle explanation on two tasks: model selection and counterfactual simulation. To our surprise, we did not find evidence of significant improvement on these tasks when users were provided with any of the saliency maps, even the synthetic oracle explanation designed to be simple to understand and highly indicative of the answer. Nonetheless, explanations did help users more accurately describe the models. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
- Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
- Debugging tests for model explanations. arXiv preprint arXiv:2011.05429, 2020.
- Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2021.
- Use-case-grounded simulations for explanation evaluation. Advances in Neural Information Processing Systems, 35:1764–1775, 2022.
- Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772–782, 2020.
- Are visual explanations useful? a case study in model-in-the-loop prediction. arXiv preprint arXiv:2007.12248, 2020.
- Multi-objective counterfactual explanations. In International Conference on Parallel Problem Solving from Nature, pages 448–469. Springer, 2020.
- Towards automatic concept-based explanations. Advances in neural information processing systems, 32, 2019.
- Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? arXiv preprint arXiv:2005.01831, 2020.
- A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
- Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR, 2018.
- Co-mixup: Saliency guided joint mixup with supermodular diversity. arXiv preprint arXiv:2102.03065, 2021.
- Concept bottleneck models. In International conference on machine learning, pages 5338–5348. PMLR, 2020.
- Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32–73, 2017.
- An evaluation of the human-interpretability of explanation. arXiv preprint arXiv:1902.00006, 2019.
- Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018.
- A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
- Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421, 2018.
- Evaluating explanations: How much do explanations from the teacher aid students? Transactions of the Association for Computational Linguistics, 10:359–375, 2022.
- " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
- Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660–2673, 2016.
- Human interpretation of saliency-based explanation over text. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 611–636, 2022.
- Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
- Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713, 2016.
- Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017.
- Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017.
- Actionable recourse in linear classification. In Proceedings of the conference on fairness, accountability, and transparency, pages 10–19, 2019.
- Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017.
- Do feature attribution methods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9623–9633, 2022a.
- Exsum: From local explanations to model understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, July 2022b.
- Shawn Im (7 papers)
- Jacob Andreas (116 papers)
- Yilun Zhou (28 papers)