Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

RankSHAP: Shapley Value Based Feature Attributions for Learning to Rank (2405.01848v2)

Published 3 May 2024 in cs.IR and cs.LG

Abstract: Numerous works propose post-hoc, model-agnostic explanations for learning to rank, focusing on ordering entities by their relevance to a query through feature attribution methods. However, these attributions often weakly correlate or contradict each other, confusing end users. We adopt an axiomatic game-theoretic approach, popular in the feature attribution community, to identify a set of fundamental axioms that every ranking-based feature attribution method should satisfy. We then introduce Rank-SHAP, extending classical Shapley values to ranking. We evaluate the RankSHAP framework through extensive experiments on two datasets, multiple ranking methods and evaluation metrics. Additionally, a user study confirms RankSHAP's alignment with human intuition. We also perform an axiomatic analysis of existing rank attribution algorithms to determine their compliance with our proposed axioms. Ultimately, our aim is to equip practitioners with a set of axiomatically backed feature attribution methods for studying IR ranking models, that ensure generality as well as consistency.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. MS MARCO: Microsoft Machine Reading Comprehension. https://microsoft.github.io/msmarco/, 2016. Accessed: [3.5.2023].
  2. Explainable information retrieval: A survey. arXiv preprint arXiv:2211.02405, 2022.
  3. Finding inverse document frequency information in bert. arXiv preprint arXiv:2202.12191, 2022.
  4. Rank-LIME: Local model-agnostic feature attribution for learning to rank. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp.  33–37, 2023.
  5. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In Proceedings of the 37th IEEE Conference on Security and Privacy (Oakland), pp.  598–617, 2016.
  6. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp.  4171–4186, 2019.
  7. A formal study of information retrieval heuristics. In Proceedings of the 27th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp.  49–56, 2004.
  8. A study on the interpretability of neural retrieval models using DeepSHAP. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 1005–1008, 2019.
  9. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20(4):422–446, 2002.
  10. A unified approach to interpreting model predictions. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NeurIPS), pp.  4768–4777, 2017.
  11. Lower-bounding term frequency normalization. In Proceedings of the 20th ACM international conference on Information and Knowledge Management (CIKM), pp.  7–16, 2011.
  12. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085, 2019.
  13. Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting. arXiv preprint arXiv:2102.07831, 2021.
  14. A general approximation framework for direct optimization of information retrieval measures. Information retrieval, 13(4):375–397, 2010.
  15. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
  16. Explaining documents’ relevance to search queries. arXiv preprint arXiv:2111.01314, 2021.
  17. One word at a time: adversarial attacks on retrieval models. arXiv preprint arXiv:2008.02197, 2020.
  18. “why should i trust you?” explaining the predictions of any classifier. In Proceedings of the 22nd Special Interest Group on Knowledge Discovery in Data (SIGKDD), pp.  1135–1144, 2016.
  19. Simple, proven approaches to text retrieval. Technical report, University of Cambridge, Computer Laboratory, 1994.
  20. Shapley, L. S. A value for n-person games. Contributions to the Theory of Games, 2(28):307–317, 1953.
  21. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp.  3145–3153, 2017.
  22. Exs: Explainable search using local model agnostic interpretability. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM), pp.  770–773, 2019.
  23. Valid explanations for learning to rank models. arXiv preprint arXiv:2004.13972, 2020.
  24. Lirme: locally interpretable ranking model explanation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pp. 1281–1284, 2019.
  25. Towards axiomatic explanations for neural ranking models. In Proceedings of the 2021 ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR), pp.  13–22, 2021.
  26. Feature importance ranking for deep learning. Advances in Neural Information Processing Systems, 33:5105–5114, 2020.
  27. Prada: practical black-box adversarial attacks against neural ranking models. ACM Transactions on Information Systems, 41(4):1–27, 2023.
  28. Relation based term weighting regularization. In Proceedings of the 34th European Conference on Information Retrieval (ECIR), pp.  109–120, 2012.
  29. Young, H. P. Monotonic solutions of cooperative games. International Journal of Game Theory, 14(2):65–72, 1985.
  30. Explain and predict, and then predict again. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM), pp.  418–426, 2021.
  31. Interpretable learning-to-rank with generalized additive models. arXiv preprint arXiv:2005.02553, 2020.
  32. A decision theoretic framework for ranking using implicit feedback. In SIGIR 2008 workshop on Learning to Rank for Information Retrieval, 2008.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Tanya Chowdhury (8 papers)
  2. Yair Zick (36 papers)
  3. James Allan (28 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets