Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Confident Feature Ranking (2307.15361v2)

Published 28 Jul 2023 in stat.ML, cs.AI, and cs.LG

Abstract: Machine learning models are widely applied in various fields. Stakeholders often use post-hoc feature importance methods to better understand the input features' contribution to the models' predictions. The interpretation of the importance values provided by these methods is frequently based on the relative order of the features (their ranking) rather than the importance values themselves. Since the order may be unstable, we present a framework for quantifying the uncertainty in global importance values. We propose a novel method for the post-hoc interpretation of feature importance values that is based on the framework and pairwise comparisons of the feature importance values. This method produces simultaneous confidence intervals for the features' ranks, which include the ``true'' (infinite sample) ranks with high probability, and enables the selection of the set of the top-k important features.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. A study of machine learning in healthcare. In 2017 IEEE 41st annual computer software and applications conference (COMPSAC), volume 2, pages 236–241. IEEE, 2017.
  2. Machine learning for quantitative finance applications: A survey. Applied Sciences, 9(24):5574, 2019.
  3. Applications and techniques for fast machine learning in science. Frontiers in big Data, 5:787421, 2022.
  4. Machine learning in concrete science: applications, challenges, and best practices. npj Computational Materials, 8(1):127, 2022.
  5. Stakeholders in explainable ai. arXiv preprint arXiv:1810.00184, 2018.
  6. European union regulations on algorithmic decision-making and a “right to explanation”. AI magazine, 38(3):50–57, 2017.
  7. Leo Breiman. Random forests. Machine learning, 45(1):5–32, 2001.
  8. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874, 2017.
  9. Explainable ai for trees: From local explanations to global understanding. arXiv preprint arXiv:1905.04610, 2019.
  10. The explanation game: Explaining machine learning models using shapley values. In International Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages 17–38. Springer, 2020.
  11. Explaining by removing: A unified framework for model explanation. arXiv preprint arXiv:2011.14878, 2020.
  12. General pitfalls of model-agnostic interpretation methods for machine learning models. In International Workshop on Extending Explainable AI Beyond Deep Models and Classifiers, pages 39–68. Springer, 2020.
  13. But are you sure? an uncertainty-aware perspective on explainable ai. In International Conference on Artificial Intelligence and Statistics, pages 7375–7391. PMLR, 2023.
  14. Robust and stable black box explanations. In International Conference on Machine Learning, pages 5628–5638. PMLR, 2020.
  15. Rethinking stability for attribution-based explanations. arXiv preprint arXiv:2203.06877, 2022.
  16. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances in Neural Information Processing Systems, 34:9391–9404, 2021.
  17. Local model explanations and uncertainty without model access. arXiv preprint arXiv:2301.05761, 2023.
  18. Standard errors and confidence intervals for variable importance in random forest regression, classification, and survival. Statistics in medicine, 38(4):558–582, 2019.
  19. Understanding global feature contributions through additive importance measures. arXiv preprint arXiv:2004.00668, 2020.
  20. Relating the partial dependence plot and permutation feature importance to the data generating process. arXiv preprint arXiv:2109.01433, 2021.
  21. Sources of uncertainty in long-term global scenarios of solar photovoltaic technology. Nature Climate Change, 11(3):266–273, 2021.
  22. Early risk assessment for covid-19 patients from emergency department data using machine learning. Scientific reports, 11(1):4200, 2021.
  23. Justin Rising. Uncertainty in ranking. arXiv preprint arXiv:2107.03459, 2021.
  24. Simultaneous confidence intervals for ranks with application to ranking institutions. Biometrics, 78(1):238–247, 2022.
  25. False discovery rate–adjusted multiple confidence intervals for selected parameters. Journal of the American Statistical Association, 100(469):71–81, 2005.
  26. Jason Hsu. Multiple comparisons: theory and methods. CRC Press, 1996.
  27. Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer. Proceedings of the National Academy of Sciences, 103(15):5923–5928, 2006.
  28. Harry O Posten. The robustness of the one-sample t-test over the pearson system. Journal of Statistical Computation and Simulation, 9(2):133–149, 1979.
  29. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65–70, 1979.
  30. Juliet Popper Shaffer. Control of directional errors with stagewise multiple test procedures. The Annals of Statistics, 8(6):1342–1347, 1980.
  31. Juliet Popper Shaffer. Multiple hypothesis testing. Annual review of psychology, 46(1):561–584, 1995.
  32. Resampling-based multiple testing: Examples and methods for p-value adjustment, volume 279. John Wiley & Sons, 1993.
  33. John Wilder Tukey. The problem of multiple comparisons. Multiple comparisons, 1953.
  34. Rand R Wilcox. Introduction to robust estimation and hypothesis testing. Academic press, 2011.
  35. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794, 2016.
  36. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence, 2:113–127, 2014.
  37. Machine bias. propublica, may 23, 2016, 2016.
  38. Nomao. UCI Machine Learning Repository, 2012. DOI: https://doi.org/10.24432/C53G79.
  39. Ronaldo C Prati. Combining feature ranking algorithms through rank aggregation. In The 2012 international joint conference on neural networks (IJCNN), pages 1–8. Ieee, 2012.
  40. Robust feature selection using ensemble feature selection techniques. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2008, Antwerp, Belgium, September 15-19, 2008, Proceedings, Part II 19, pages 313–325. Springer, 2008.
  41. Uncertainty quantification of surrogate explanations: An ordinal consensus approach. arXiv preprint arXiv:2111.09121, 2021.
  42. A variable ranking method for machine learning models with correlated features: in-silico validation and application for diabetes prediction. Applied Sciences, 11(16):7740, 2021.
  43. An information theoretic approach to quantify the stability of feature selection and ranking algorithms. Knowledge-Based Systems, 195:105745, 2020.
  44. The stability of different aggregation techniques in ensemble feature selection. Journal of Big Data, 9(1):1–23, 2022.
  45. Shanti S Gupta. On some multiple decision (selection and ranking) rules. Technometrics, 7(2):225–245, 1965.
  46. Using ranking and selection to “clean up” after simulation optimization. Operations Research, 51(5):814–825, 2003.
  47. Ranking populations based on sample survey data. Statistics, page 12, 2014.
  48. Ranking with confidence for large scale comparison data. arXiv preprint arXiv:2202.01670, 2022.
  49. Revisiting subset selection. In 2020 Winter Simulation Conference (WSC), pages 2972–2983. IEEE, 2020.
  50. Confidence intervals for ranks of age-adjusted rates across states or counties. Statistics in Medicine, 33(11):1853–1866, 2014.
  51. A joint confidence region for an overall ranking of populations. Journal of the Royal Statistical Society: Series C (Applied Statistics), 69(3):589–606, 2020.
  52. An extension on” statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. Journal of machine learning research, 9(12), 2008.
  53. Sture Holm. Confidence intervals for ranks. 2013.
  54. Bradley Efron. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, volume 1. Cambridge University Press, 2012.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Bitya Neuhof (1 paper)
  2. Yuval Benjamini (11 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets