Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis (2309.11987v1)

Published 21 Sep 2023 in cs.LG, cs.AI, and cs.HC

Abstract: Post-hoc explainability methods aim to clarify predictions of black-box machine learning models. However, it is still largely unclear how well users comprehend the provided explanations and whether these increase the users ability to predict the model behavior. We approach this question by conducting a user study to evaluate comprehensibility and predictability in two widely used tools: LIME and SHAP. Moreover, we investigate the effect of counterfactual explanations and misclassifications on users ability to understand and predict the model behavior. We find that the comprehensibility of SHAP is significantly reduced when explanations are provided for samples near a model's decision boundary. Furthermore, we find that counterfactual explanations and misclassifications can significantly increase the users understanding of how a machine learning model is making decisions. Based on our findings, we also derive design recommendations for future post-hoc explainability methods with increased comprehensibility and predictability.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 275–285. https://doi.org/10.1145/3377325.3377519
  2. Machine learning interpretability: A survey on methods and metrics. Electronics 8, 8 (2019), 832.
  3. Discovering statistics using R. SAGE publications, London, England. 361–365 pages.
  4. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.10820 (2018).
  5. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2018), 1–42.
  6. David Harrison Jr and Daniel L Rubinfeld. 1978. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management 5, 1 (1978), 81–102.
  7. SG Hart et al. 1988. Development of NASA-TLX: Results of empirical and theoretical research.” inP. A. Hancock and N. Meshkati (eds.), Human Mental Workload.
  8. Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? Association for Computational Linguistics (ACL) (2020).
  9. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87 (2014), 96–110.
  10. Robert R Hoffman. 2017. A taxonomy of emergent trusting in the human–machine relationship. Cognitive systems engineering: The future for a changing world (2017), 137–164.
  11. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in AI. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 624–635.
  12. Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376219
  13. Himabindu Lakkaraju and Osbert Bastani. 2020. ”How Do I Fool You?”: Manipulating User Trust via Misleading Black Box Explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (New York, NY, USA) (AIES ’20). Association for Computing Machinery, New York, NY, USA, 79–85. https://doi.org/10.1145/3375627.3375833
  14. Zachary C Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (2018), 31–57.
  15. Scott M. Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (Long Beach, California, USA) (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 4768–4777.
  16. David W. Martin. 2007. Doing Psychology Experiments. (2007), 148–170.
  17. Philipp Mayring. 2004. Qualitative content analysis. A companion to qualitative research 1, 2 (2004), 159–176.
  18. Sina Mohseni. 2019. Toward Design and Evaluation Framework for Interpretable Machine Learning Systems. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 553–554.
  19. Sina Mohseni and Eric D Ragan. 2020. A human-grounded evaluation benchmark for local explanations of machine learning. arXiv preprint arXiv:1801.05075 (2020).
  20. A multidisciplinary survey and framework for design and evaluation of explainable AI systems. ACM Transactions on Interactive Intelligent Systems (TiiS) 11, 3-4 (2021), 1–45.
  21. Interpretable Machine Learning–A Brief History, State-of-the-Art and Challenges. Koprinska I. et al. (eds) ECML PKDD Workshops. ECML PKDD 2020. Communications in Computer and Information Science, vol 1323. Springer, Cham. (2020).
  22. Ishani Mondal and Debasis Ganguly. 2020. ALEX: Active Learning based Enhancement of a Classification Model’s EXplainability. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 3309–3312.
  23. The effects of meaningful and meaningless explanations on trust and perceived system accuracy in intelligent systems. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 97–105.
  24. How model accuracy and explanation fidelity influence user trust. AI IJCAI Workshop on Explainable Artificial Intelligence (2019).
  25. Qualtrics. [n.d.]. Copyright Year: 2021, Location: Provo, Utah, USA. https://www.qualtrics.com
  26. ”Why should I trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1135–1144.
  27. Anchors: High-Precision Model-Agnostic Explanations.. In Proceedings of the 32nd AAAI International Conference on Artificial Intelligence, Vol. 18. 1527–1535.
  28. Stefan Rüping. 2006. Learning interpretable models. (2006). http://dx.doi.org/10.17877/DE290R-8863
  29. Philipp Schmidt and Felix Biessmann. 2019. Quantifying interpretability and trust in machine learning systems. AAAI-19 Workshop on Network Interpretability for Deep Learning (2019).
  30. Donghee Shin. 2021. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies 146 (2021), 102551.
  31. Learning Important Features through Propagating Activation Differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML’17). JMLR.org, Sydney, NSW, Australia, 3145–3153.
  32. Kacper Sokol and Peter Flach. 2020. Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT* ’20). Association for Computing Machinery, New York, NY, USA, 56–67. https://doi.org/10.1145/3351095.3372870
  33. explAIner: A visual analytics framework for interactive and explainable machine learning. IEEE transactions on visualization and computer graphics 26, 1 (2019), 1064–1074.
  34. The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models. (2020), 107–118. https://www.aclweb.org/anthology/2020.emnlp-demos.15
  35. What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. In Proceedings of the 4th Machine Learning for Healthcare Conference (Proceedings of Machine Learning Research, Vol. 106), Finale Doshi-Velez, Jim Fackler, Ken Jung, David Kale, Rajesh Ranganath, Byron Wallace, and Jenna Wiens (Eds.). PMLR, Ann Arbor, Michigan, 359–380. http://proceedings.mlr.press/v106/tonekaboni19a.html
  36. Designing Theory-Driven User-Centric Explainable AI. Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3290605.3300831
  37. Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics 10, 5 (2021), 593.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Anahid Jalali (11 papers)
  2. Bernhard Haslhofer (35 papers)
  3. Simone Kriglstein (3 papers)
  4. Andreas Rauber (12 papers)
Citations (3)
View on Semantic Scholar