Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Evaluating Saliency Explanations in NLP by Crowdsourcing (2405.10767v1)

Published 17 May 2024 in cs.HC and cs.AI

Abstract: Deep learning models have performed well on many NLP tasks. However, their internal mechanisms are typically difficult for humans to understand. The development of methods to explain models has become a key issue in the reliability of deep learning models in many important applications. Various saliency explanation methods, which give each feature of input a score proportional to the contribution of output, have been proposed to determine the part of the input which a model values most. Despite a considerable body of work on the evaluation of saliency methods, whether the results of various evaluation metrics agree with human cognition remains an open question. In this study, we propose a new human-based method to evaluate saliency methods in NLP by crowdsourcing. We recruited 800 crowd workers and empirically evaluated seven saliency methods on two datasets with the proposed method. We analyzed the performance of saliency methods, compared our results with existing automated evaluation methods, and identified notable differences between NLP and computer vision (CV) fields when using saliency methods. The instance-level data of our crowdsourced experiments and the code to reproduce the explanations are available at https://github.com/xtlu/lreccoling_evaluation.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations.
  2. How to explain individual classification decisions. Journal of Machine Learning Research, 11:1803–1831.
  3. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  4. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
  5. Evaluating and characterizing human rationales. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9294–9307, Online. Association for Computational Linguistics.
  6. Karin de Langis and Dongyeop Kang. 2023. A comparative study on textual saliency of styles from eye tracking, annotations, and language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 108–121, Singapore. Association for Computational Linguistics.
  7. Ranking a stream of news. In Proceedings of the 14th international conference on World Wide Web, pages 97–106.
  8. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.
  9. ERASER: A benchmark to evaluate rationalized NLP models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4443–4458, Online. Association for Computational Linguistics.
  10. Shuoyang Ding and Philipp Koehn. 2021. Evaluating saliency methods for neural language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5034–5052, Online. Association for Computational Linguistics.
  11. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1.
  12. Learning to scaffold: Optimizing model explanations for teaching. In Advances in Neural Information Processing Systems, volume 35, pages 36108–36122. Curran Associates, Inc.
  13. New definitions and evaluations for saliency methods: Staying intrinsic, complete and sound. In Advances in Neural Information Processing Systems.
  14. A benchmark for interpretability methods in deep neural networks. Advances in Neural Information Processing Systems, 32:9737–9748.
  15. Neighboring words affect human interpretation of saliency explanations. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11816–11833, Toronto, Canada. Association for Computational Linguistics.
  16. Towards explainable NLP: A generative explanation framework for text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5570–5581, Florence, Italy. Association for Computational Linguistics.
  17. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  18. Crowdsourcing evaluation of saliency-based xai methods. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 431–446. Springer.
  19. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.
  20. Evaluating the faithfulness of importance measures in NLP by recursively masking allegedly important tokens and retraining. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1731–1751, Abu Dhabi, United Arab Emirates.
  21. Mary Ann Marcinkiewicz. 1994. Building a large annotated corpus of english: The penn treebank. Using Large Corpora, 273.
  22. Measuring and improving faithfulness of attention in neural machine translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2791–2802, Online. Association for Computational Linguistics.
  23. Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? Transactions of the Association for Computational Linguistics, 10:359–375.
  24. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144.
  25. Brian Riordan. 2020. Analyzing saliency in neural models for scoring content in science explanations. BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
  26. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660–2673.
  27. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626.
  28. Learning important features through propagating activation differences. In International conference on machine learning, pages 3145–3153. PMLR.
  29. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713.
  30. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
  31. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems, 34:9391–9404.
  32. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5):828–841.
  33. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR.
  34. Attention is all you need. Advances in neural information processing systems, 30.
  35. Crowdsourcing mechanism for trust evaluation in cpcs based on intelligent mobile edge computing. ACM Trans. Intell. Syst. Technol., 10(6).
  36. Toward improving the evaluation of visual attention models: a crowdsourcing approach. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8.
  37. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929.
  38. Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10(5):593.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Xiaotian Lu (7 papers)
  2. Jiyi Li (16 papers)
  3. Zhen Wan (42 papers)
  4. Xiaofeng Lin (13 papers)
  5. Koh Takeuchi (22 papers)
  6. Hisashi Kashima (63 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets