Papers
Topics
Authors
Recent
2000 character limit reached

Fast Calculation of Feature Contributions in Boosting Trees (2407.03515v2)

Published 3 Jul 2024 in stat.ML and cs.LG

Abstract: Recently, several fast algorithms have been proposed to decompose predicted value into Shapley values, enabling individualized feature contribution analysis in tree models. While such local decomposition offers valuable insights, it underscores the need for a global evaluation of feature contributions. Although coefficients of determination ($R2$) allow for comparative assessment of individual features, individualizing $R2$ is challenged by the underlying quadratic losses. To address this, we propose Q-SHAP, an efficient algorithm that reduces the computational complexity of calculating Shapley values for quadratic losses to polynomial time. Our simulations show that Q-SHAP not only improves computational efficiency but also enhances the accuracy of feature-specific $R2$ estimates.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. G. Stiglic, P. Kocbek, N. Fijacko, M. Zitnik, K. Verbert, and L. Cilar, “Interpretability of machine learning-based prediction models in healthcare,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 10, no. 5, p. e1379, 2020.
  2. N. Bussmann, P. Giudici, D. Marinelli, and J. Papenbrock, “Explainable machine learning in credit risk management,” Computational Economics, vol. 57, no. 1, pp. 203–216, 2021.
  3. S. M. Lundberg and S.-I. Lee, “Consistent feature attribution for tree ensembles,” Workshop on Human Interpretability in Machine Learning, 2017.
  4. H. Ishwaran, “Variable importance in binary regression trees and forests,” Elecctronic Journal of Statistics, vol. 1, pp. 519–537, 2007.
  5. L. S. Shapley, “A value for N-person games,” Contributions to the Theory of Games, vol. 2(28), p. 307–317, 1953.
  6. S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  7. S. L. Chau, R. Hu, J. Gonzalez, and D. Sejdinovic, “RKHS-SHAP: Shapley values for kernel methods,” Advances in Neural Information Processing Systems, vol. 35, pp. 13 050–13 063, 2022.
  8. I. Covert, S. M. Lundberg, and S.-I. Lee, “Understanding global feature contributions with additive importance measures,” Advances in Neural Information Processing Systems, vol. 33, pp. 17 212–17 223, 2020.
  9. S. Lipovetsky and M. Conklin, “Analysis of regression in game theory approach,” Applied Stochastic Models in Business and Industry, vol. 17, no. 4, pp. 319–330, 2001.
  10. A. B. Owen and C. Prieur, “On Shapley value for measuring importance of dependent inputs,” SIAM/ASA Journal on Uncertainty Quantification, vol. 5, no. 1, pp. 986–1002, 2017.
  11. E. Song, B. L. Nelson, and J. Staum, “Shapley effects for global sensitivity analysis: theory and computation,” SIAM/ASA Journal on Uncertainty Quantification, vol. 4, no. 1, pp. 1060–1083, 2016.
  12. B. Williamson and J. Feng, “Efficient nonparametric statistical inference on population feature importance using Shapley values,” in International Conference on Machine Learning.   PMLR, 2020, pp. 10 282–10 291.
  13. S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S.-I. Lee, “From local explanations to global understanding with explainable ai for trees,” Nature Machine Intelligence, vol. 2, no. 1, pp. 56–67, 2020.
  14. J. Yang, “Fast Treeshap: Accelerating SHAP value computation for trees,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  15. A. Bifet, J. Read, C. Xu et al., “Linear TreeShap,” Advances in Neural Information Processing Systems, vol. 35, pp. 25 818–25 828, 2022.
  16. C. Bénard, G. Biau, S. Da Veiga, and E. Scornet, “SHAFF: Fast and consistent SHApley eFfect estimates via random Forests,” in International Conference on Artificial Intelligence and Statistics.   PMLR, 2022, pp. 5563–5582.
  17. L. Breiman, “Random Forests,” Machine Learning, vol. 45, pp. 5–32, 2001.
  18. A. Karczmarz, T. Michalak, A. Mukherjee, P. Sankowski, and P. Wygocki, “Improved Feature Importance Computation for Tree Models based on the Banzhaf Value,” in Proceedings of the Thirty-Eight Conference on Uncertainty in Artificial Intelligence, 2022, pp. 969–979.
  19. J. H. Friedman, “Greedy function approximation: A gradient boosting machine,” Annals of Statistics, pp. 1189–1232, 2001.
  20. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 785–794.
  21. G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  22. L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin, “Catboost: unbiased boosting with categorical features,” Advances in Neural Information Processing Systems, vol. 31, 2018.
  23. G. M. Kurtzer, V. Sochat, and M. W. Bauer, “Singularity: Scientific containers for mobility of compute,” PLOS ONE, vol. 12, no. 5, p. e0177459, 2017.
  24. J. N. Weinstein, E. A. Collisson, G. B. Mills, K. R. Shaw, B. A. Ozenberger, K. Ellrott, I. Shmulevich, C. Sander, and J. M. Stuart, “The cancer genome atlas pan-cancer analysis project,” Nature Genetics, vol. 45, no. 10, pp. 1113–1120, 2013.
  25. A. Colaprico, T. C. Silva, C. Olsen, L. Garofano, C. Cava, D. Garolini, T. S. Sabedot, T. M. Malta, S. M. Pagnotta, I. Castiglioni et al., “TCGAbiolinks: an R/Bioconductor package for integrative analysis of TCGA data,” Nucleic Acids Research, vol. 44, no. 8, p. e71, 2016.
  26. M. J. Goldman, B. Craft, M. Hastie, K. Repečka, F. McDade, A. Kamath, A. Banerjee, Y. Luo, D. Rogers, A. N. Brooks et al., “Visualizing and interpreting cancer genomics data via the Xena platform,” Nature Biotechnology, vol. 38, no. 6, pp. 675–678, 2020.
  27. Z. Jiang, C. Chen, Z. Xu, X. Wang, M. Zhang, and D. Zhang, “SIGNET: Transcriptome-wide causal inference for gene regulatory networks,” Scientific Reports, vol. 13, no. 1, p. 19371, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up