Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
184 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VisRuler: Visual Analytics for Extracting Decision Rules from Bagged and Boosted Decision Trees (2112.00334v5)

Published 1 Dec 2021 in cs.LG, cs.HC, and stat.ML

Abstract: Bagging and boosting are two popular ensemble methods in ML that produce many individual decision trees. Due to the inherent ensemble characteristic of these methods, they typically outperform single decision trees or other ML models in predictive performance. However, numerous decision paths are generated for each decision tree, increasing the overall complexity of the model and hindering its use in domains that require trustworthy and explainable decisions, such as finance, social care, and health care. Thus, the interpretability of bagging and boosting algorithms, such as random forest and adaptive boosting, reduces as the number of decisions rises. In this paper, we propose a visual analytics tool that aims to assist users in extracting decisions from such ML models via a thorough visual inspection workflow that includes selecting a set of robust and diverse models (originating from different ensemble learning algorithms), choosing important features according to their global contribution, and deciding which decisions are essential for global explanation (or locally, for specific cases). The outcome is a final decision based on the class agreement of several models and the explored manual decisions exported by users. We evaluated the applicability and effectiveness of VisRuler via a use case, a usage scenario, and a user study. The evaluation revealed that most users managed to successfully use our system to explore decision rules visually, performing the proposed tasks and answering the given questions in a satisfying way.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (127)
  1. Zhou ZH. Ensemble learning. Boston, MA: Springer. ISBN 978-0-387-73003-5, 2009. pp. 270–273. 10.1007/978-0-387-73003-5_293.
  2. Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery 2018; 8(4): e1249. 10.1002/widm.1249.
  3. Breiman L. Stacked regressions. Machine Learning 1996; 24(1): 49–64. 10.1007/BF00117832.
  4. Experiments with a new boosting algorithm. In Proc. of ICML. Morgan Kaufmann Publishers Inc. ISBN 1558604197, p. 148–156. 10.5555/3091696.3091715.
  5. Schapire RE. The strength of weak learnability. Machine Learning 1990; 5(2): 197–227. 10.1007/BF00116037.
  6. Wolpert DH. Stacked generalization. Neural Networks 1992; 5(2): 241–259. 10.1016/S0893-6080(05)80023-1.
  7. What are decision trees? Nature Biotechnology 2008; 26(9): 1011–1013.
  8. Breiman L. Random forests. Machine Learning 2001; 45(1): 5–32. 10.1023/A:1010933404324.
  9. Freund Y, Schapire R and Abe N. A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 1999; 14(5): 771–780.
  10. Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Research 1999; 11(1): 169–198. 10.5555/3013545.3013549.
  11. Explaining the success of adaboost and random forests as interpolating classifiers. Journal of Machine Learning Research 2017; 18(1): 1558–1590.
  12. Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research 2014; 15(1): 3133–3181. 10.5555/2627435.2697065.
  13. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml. Accessed November 12, 2022.
  14. Breiman L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science 2001; 16(3): 199 – 231. 10.1214/ss/1009213726.
  15. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proc. of ACM KDD. ACM. ISBN 9781450336642, p. 1721–1730. 10.1145/2783258.2788613.
  16. Tam GKL, Kothari V and Chen M. An analysis of machine- and human-analytics in classification. IEEE TVCG 2017; 23(1): 71–80. 10.1109/TVCG.2016.2598829.
  17. 2D transparency space—Bring domain users and machine learning experts together. In Human and Machine Learning. Springer, 2018. pp. 3–19. 10.1007/978-3-319-90403-0_1.
  18. Ribeiro MT, Singh S and Guestrin C. “Why should I trust you?”: Explaining the predictions of any classifier. In Proc. of ACM KDD. ACM. ISBN 978-1-4503-4232-2, pp. 1135–1144. 10.1145/2939672.2939778.
  19. Hastie T, Tibshirani R and Friedman J. The elements of statistical learning. Springer Series in Statistics, Springer New York Inc., 2001. 10.1007/978-0-387-84858-7.
  20. Lakkaraju H, Bach SH and Leskovec J. Interpretable decision sets: A joint framework for description and prediction. In Proc. of ACM KDD. ACM. ISBN 9781450342322, p. 1675–1684. 10.1145/2939672.2939874.
  21. An explainable AI decision-support-system to automate loan underwriting. Expert Systems with Applications 2020; 144: 113100. 10.1016/j.eswa.2019.113100.
  22. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning 1999; 36(1): 105–139.
  23. Combining bagging and boosting, 2007. Accessed November 12, 2022.
  24. Local vs. global interpretability of machine learning models in type 2 diabetes mellitus screening. In Proc. of Artificial Intelligence in Medicine: Knowledge Representation and Transparent and Explainable Systems. Cham: Springer International Publishing. ISBN 978-3-030-37446-4, pp. 108–119. 10.1007/978-3-030-37446-4_9.
  25. Lipton ZC. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 2018; 16(3): 31–57. 10.1145/3236386.3241340.
  26. Du M, Liu N and Hu X. Techniques for interpretable machine learning. Communications of the ACM 2019; 63(1): 68–77. 10.1145/3359786.
  27. Carvalho DV, Pereira EM and Cardoso JS. Machine learning interpretability: A survey on methods and metrics. Electronics 2019; 8(8). 10.3390/electronics8080832.
  28. Weller A. Transparency: Motivations and challenges. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 2019. pp. 23–40. 10.1007/978-3-030-28954-6_2.
  29. Kim B, Rudin C and Shah J. The Bayesian case model: A generative approach for case-based reasoning and prototype classification. In Proc. of NIPS, vol. 2. MIT Press, p. 1952–1960.
  30. Task-based visual interactive modeling: Decision trees and rule-based classifiers. IEEE TVCG 2021; : 1–110.1109/TVCG.2020.3045560.
  31. iForest: Interpreting random forests via visual analytics. IEEE TVCG 2019; 25(1): 407–416. 10.1109/TVCG.2018.2864475.
  32. Explainable Matrix - Visualization for global and local interpretability of random forest classification ensembles. IEEE TVCG 2021; 27(2): 1427–1437. 10.1109/TVCG.2020.3030354.
  33. RfX: A design study for the interactive exploration of a random forest to enhance testing procedures for electrical engines. CGF 2022; https://doi.org/10.1111/cgf.14452. To appear.
  34. Colorful Trees: Visualizing random forests for analysis and interpretation. In Proc. of IEEE WACV. IEEE, pp. 294–302. 10.1109/WACV.2019.00037.
  35. Multivariate data explanation by Jumping Emerging Patterns visualization. ArXiv e-prints 2021; 2106.11112.
  36. Visual diagnosis of tree boosting methods. IEEE TVCG 2018; 24(1): 163–173. 10.1109/TVCG.2017.2744378.
  37. GBRTVis: Online analysis of gradient boosting regression tree. JOVI 2019; 22(1): 125–140. 10.1007/s12650-018-0514-2.
  38. Investigating the evolution of tree boosting models with visual analytics. In Proc. of IEEE PacificVis. IEEE, pp. 186–195. 10.1109/PacificVis52677.2021.00032.
  39. GBMVis: Visual analytics for interpreting gradient boosting machine. In Proc. of Cooperative Design, Visualization, and Engineering. Cham: Springer International Publishing. ISBN 978-3-030-88207-5, pp. 63–72. 10.1007/978-3-030-88207-5_7.
  40. Friedman JH. Greedy function approximation: A gradient boosting machine. Annals of statistics 2001; : 1189–1232.
  41. van der Maaten L and Hinton G. Visualizing data using t-SNE. Journal of Machine Learning Research 2008; 9: 2579–2605.
  42. van den Elzen S and van Wijk JJ. BaobabView: Interactive construction and analysis of decision trees. In Proc. of IEEE VAST. IEEE, pp. 151–160. 10.1109/VAST.2011.6102453.
  43. Nguyen T, Ho T and Shimodaira H. A visualization tool for interactive learning of large decision trees. In Proc. of IEEE ICTAI. IEEE, pp. 28–35. 10.1109/TAI.2000.889842.
  44. Lee T, Johnson J and Cheng S. An interactive machine learning framework, 2016. 1610.05463.
  45. Cavallo M and  CD. Clustrophile 2: Guided visual clustering analysis. IEEE TVCG 2019; 25(1): 267–276. 10.1109/TVCG.2018.2864477.
  46. Case study: Visualization for decision tree analysis in data mining. In Proc. of IEEE INFOVIS. IEEE, pp. 149–152. 10.1109/INFVIS.2001.963292.
  47. FFTrees: A toolbox to create, visualize, and evaluate fast-and-frugal decision trees. Judgment and Decision making 2017; 12(4): 344–368.
  48. Interactive visual comparison of multiple trees. In Proc. of IEEE VAST. IEEE, pp. 31–40. 10.1109/VAST.2011.6102439.
  49. Song H, Curran EP and Sterritt R. Multiple foci visualisation of large hierarchies with FlexTree. Inf Vis 2004; 3(1): 19–35. 10.1057/palgrave.ivs.9500065.
  50. TreeJuxtaposer: Scalable tree comparison using focus+context with guaranteed visibility. ACM Transactions on Graphics 2003; 22(3): 453–462. 10.1145/882262.882291.
  51. Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In Proc. of IEEE VAST. IEEE, pp. 43–52. 10.1109/VAST.2014.7042480.
  52. TreePOD: Sensitivity-aware selection of Pareto-optimal decision trees. IEEE TVCG 2018; 24(1): 174–183. 10.1109/TVCG.2017.2745158.
  53. Visualizing change over time using dynamic hierarchies: TreeVersity2 and the StemView. IEEE TVCG 2013; 19(12): 2566–2575. 10.1109/TVCG.2013.231.
  54. Interactive exploration of parameter space in data mining: Comprehending the predictive quality of large decision tree collections. Computers & Graphics 2014; 41: 99–113. 10.1016/j.cag.2014.02.004.
  55. Ankerst M, Ester M and Kriegel HP. Towards an effective cooperation of the user and the computer for classification. In Proc. of ACM KDD. ACM. ISBN 1581132336, p. 179–188. 10.1145/347090.347124.
  56. StarClass: Interactive visual classification using star coordinates. In Proc. of 2003 SIAM International Conference on Data Mining. SIAM, pp. 178–185. 10.1137/1.9781611972733.16.
  57. PaintingClass: Interactive construction, visualization and exploration of decision trees. In Proc. of ACM KDD. ACM. ISBN 1581137370, p. 667–672. 10.1145/956750.956837.
  58. Do TN. Towards simple, easy to understand, an interactive decision tree algorithm. College of Information Technology, Cantho University, Cantho, Vietnam, Technical Report 2007; : 06–01.
  59. Visual exploration of machine learning model behavior with hierarchical surrogate rule sets. ArXiv e-prints 2022; 2201.07724.
  60. DRIL: Descriptive rules by interactive learning. In Proc. of 2020 IEEE Visualization Conference (VIS). pp. 256–260. 10.1109/VIS47514.2020.00058.
  61. Surrogate decision tree visualization interpreting and visualizing black-box classification models with surrogate decision tree. In Proc. of CEUR Workshop, volume 2327. CEUR-WS.
  62. RuleViz: A model for visualizing knowledge discovery process. In Proc. of ACM KDD. ACM. ISBN 1581132336, p. 244–253. 10.1145/347090.347139.
  63. RISSAD: Rule-based interactive semi-supervised anomaly detection. In Proc. of EuroVis 2021 - Short Papers. The Eurographics Association. ISBN 978-3-03868-143-4. 10.2312/evs.20211050.
  64. Interactive machine learning: Letting users build classifiers. International Journal of Human-Computer Studies 2001; 55(3): 281–292. 10.1006/ijhc.2001.0499.
  65. Yuan J, Nov O and Bertini E. An exploration and validation of visual factors in understanding classification rule sets, 2021. 2109.09160.
  66. Eisemann M, Albuquerque G and Magnor M. A nested hierarchy of localized scatterplots. In 2014 27th SIBGRAPI Conference on Graphics, Patterns and Images. pp. 80–86. 10.1109/SIBGRAPI.2014.14.
  67. ExplorerTree: A focus+context exploration approach for 2D embeddings. Big Data Research 2021; 25: 100239. 10.1016/j.bdr.2021.100239.
  68. Ming Y, Qu H and Bertini E. RuleMatrix: Visualizing and understanding classifiers with rules. IEEE TVCG 2019; 25(1): 342–352. 10.1109/TVCG.2018.2864812.
  69. Visualizing surrogate decision trees of convolutional neural networks. JOVI 2020; 23(1): 141–156. 10.1007/s12650-019-00607-z.
  70. Thomas LV, Deng J and Brown ET. FacetRules: Discovering and describing related groups. In Proc. of 2021 IEEE Workshop on Machine Learning from User Interactions (MLUI). pp. 21–26. 10.1109/MLUI54255.2021.00008.
  71. Deep sequencing of the vaginal microbiota of women with HIV. PLOS One 2010; 5(8): 1–9. 10.1371/journal.pone.0012078.
  72. Improving melanoma classification by integrating genetic and morphologic features. PLOS Medicine 2008; 5(6): e120. 10.1371/journal.pmed.0050120.
  73. A visual analytics system for multi-model comparison on clinical data predictions. Visual Informatics 2020; 4(2): 122–131. https://doi.org/10.1016/j.visinf.2020.04.005.
  74. Learning and inspecting classification rules from longitudinal epidemiological data to identify predictive features on hepatic steatosis. Expert Systems with Applications 2014; 41(11): 5405–5415. 10.1016/j.eswa.2014.02.040.
  75. Phylogenetic dependency networks: Inferring patterns of CTL escape and codon covariation in HIV-1 Gag. PLOS Computational Biology 2008; 4(11): e1000225. 10.1371/journal.pcbi.1000225.
  76. RuleVis: Constructing patterns and rules for rule-based models. In Proc. of 2019 IEEE Visualization Conference (VIS). pp. 191–195. 10.1109/VISUAL.2019.8933596.
  77. Structure-based prediction of asparagine and aspartate degradation sites in antibody variable regions. PLOS One 2014; 9(6): e100736. 10.1371/journal.pone.0100736.
  78. Visualization of actionable knowledge to mitigate DRDoS attacks. In Proc. of 2016 IEEE Symposium on Visualization for Cyber Security (VizSec). pp. 1–8. 10.1109/VIZSEC.2016.7739577.
  79. Social influence and the collective dynamics of opinion formation. PLOS One 2013; 8(11): e78433. 10.1371/journal.pone.0078433.
  80. EnsembleLens: Ensemble-based visual exploration of anomaly detection algorithms with multidimensional data. IEEE TVCG 2019; 25(1): 109–119. 10.1109/TVCG.2018.2864825.
  81. Integrating data and model space in ensemble learning by visual analytics. IEEE Transactions on Big Data 2018; 10.1109/TBDATA.2018.2877350.
  82. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proc. of ACM CHI. ACM. ISBN 978-1-60558-246-7, pp. 1283–1292. 10.1145/1518701.1518895.
  83. Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE TVCG 2019; 25(1): 364–373. 10.1109/TVCG.2018.2864499.
  84. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE TVCG 2017; 23(1): 61–70. 10.1109/TVCG.2016.2598828.
  85. Boxer: Interactive comparison of classifier results. CGF 2020; 39(3): 181–193. 10.1111/cgf.13972.
  86. QUESTO: Interactive construction of objective functions for classification tasks. CGF 2020; 39(3): 153–165. 10.1111/cgf.13970.
  87. PipelineProfiler: A visual analytics tool for the exploration of AutoML pipelines. IEEE TVCG 2021; 27(2): 390–400. 10.1109/TVCG.2020.3030361.
  88. StackGenVis: Alignment of data, algorithms, and models for stacking ensemble learning using performance metrics. IEEE TVCG 2021; 10.1109/TVCG.2020.3030352.
  89. VisEvol: Visual analytics to support hyperparameter search through evolutionary optimization. CGF 2021; 40(3): 201–214. 10.1111/cgf.14300.
  90. AutoML — Google Cloud AutoML. URL https://cloud.google.com/automl/. Accessed November 12, 2022.
  91. A partition-based framework for building and validating regression models. IEEE TVCG 2013; 19(12): 1962–1971. 10.1109/TVCG.2013.125.
  92. Visual predictive analytics using iFuseML. In Proc. of EuroVA. The Eurographics Association. ISBN 978-3-03868-064-2. 10.2312/eurova.20181106.
  93. LoVis: Local pattern visualization for model refinement. CGF 2014; 33(3): 331–340. 10.1111/cgf.12389.
  94. BEAMES: Interactive multi-model steering, selection, and inspection for regression tasks. IEEE CG&A 2019; 39(9). 10.1109/MCG.2019.2922592.
  95. Genuer R, Poggi JM and Tuleau-Malot C. Variable selection using random forests. Pattern recognition letters 2010; 31(14): 2225–2236.
  96. Combining bagging, boosting and dagging for classification problems. In Proc. of Knowledge-Based Intelligent Information and Engineering Systems. Berlin, Heidelberg: Springer. ISBN 978-3-540-74827-4, pp. 493–500.
  97. Kotsiantis S. Combining bagging, boosting, rotation forest and random subspace methods. Artificial Intelligence Review 2011; 35(3): 223–240.
  98. A survey of surveys on the use of visualization for interpreting machine learning models. Inf Vis 2020; 19(3): 207–233. 10.1177/1473871620904671.
  99. The state of the art in enhancing trust in machine learning models with the use of visualizations. CGF 2020; 39(3): 713–756. 10.1111/cgf.14034.
  100. Contrastive learning using spectral methods. In Proc. of Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
  101. VisRulerCode, 2022. URL https://github.com/angeloschatzimparmpas/VisRuler. Accessed November 12, 2022.
  102. Random search for hyper-parameter optimization. Journal of Machine Learning Research 2012; 13: 281–305. 10.5555/2188385.2188395.
  103. John F H, Richard L and Jeffrey D S. World happiness report 2019. New York: Sustainable Development Solutions Network 2019; .
  104. World happiness report, 2019. URL https://www.kaggle.com/unsdsn/world-happiness. Accessed November 12, 2022.
  105. Regression as classification. In Proc. of 2012 IEEE Southeastcon. pp. 1–6. 10.1109/SECon.2012.6196887.
  106. Identifying feature relevance using a random forest. In Proc. of Subspace, Latent Structure and Feature Selection. Berlin, Heidelberg: Springer. ISBN 978-3-540-34138-3, pp. 173–184.
  107. Wang R. Adaboost for feature selection, classification and its relation with SVM, a review. Physics Procedia 2012; 25: 800–807. https://doi.org/10.1016/j.phpro.2012.03.160. International Conference on Solid State Devices and Materials Science, April 1-2, 2012, Macao.
  108. McInnes L, Healy J and Melville J. UMAP: Uniform manifold approximation and projection for dimension reduction. ArXiv e-prints 2018; 1802.03426.
  109. Toward a quantitative survey of dimension reduction techniques. IEEE TVCG 2021; 27(3): 2153–2173. 10.1109/TVCG.2019.2944182.
  110. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proc. of KDD. AAAI Press, p. 226–231. 10.5555/3001460.3001507.
  111. Mannor S, Peleg D and Rubinstein R. The cross entropy method for classification. In Proc. of ICML. ACM. ISBN 1595931805, p. 561–568. 10.1145/1102351.1102422.
  112. Fisher RA. The use of multiple measurements in taxonomic problems. Annals of Eugenics 1936; 7(2): 179–188. 10.1111/j.1469-1809.1936.tb02137.x.
  113. A heuristic approach to value-driven evaluation of visualizations. IEEE TVCG 2019; 25(1): 491–500. 10.1109/TVCG.2018.2865146.
  114. Geurts P, Ernst D and Wehenkel L. Extremely randomized trees. Machine Learning 2006; 63(1): 3–42.
  115. LightGBM: A highly efficient gradient boosting decision tree. In Proc. of NIPS. Curran Associates Inc. ISBN 9781510860964, pp. 3149–3157.
  116. XGBoost: A scalable tree boosting system. In Proc. of ACM KDD. ACM. ISBN 9781450342322, pp. 785–794. 10.1145/2939672.2939785.
  117. Song C. Research of association rule algorithm based on data mining. In Proc. of IEEE ICBDA. IEEE, pp. 1–4. 10.1109/ICBDA.2016.7509789.
  118. Ware C. Information visualization: Perception for design. Morgan Kaufmann, 2019.
  119. Scale and complexity in visual analytics. Inf Vis 2009; 8(4): 247–253.
  120. Hyperparameter search in machine learning. In Proc. of MIC.
  121. Easy hyperparameter search using Optunity. ArXiv e-prints 2014; 1412.1114.
  122. HyperTuner: Visual analytics for hyperparameter tuning by professionals. In Proc. of IEEE MLUI.
  123. Splatterplots: Overcoming overdraw in scatter plots. IEEE TVCG 2013; 19(9): 1526–1538. 10.1109/TVCG.2013.65.
  124. Overlap removal of dimensionality reduction scatterplot layouts. ArXiv e-prints 2019; 1903.06262.
  125. Making many-to-many parallel coordinate plots scalable by asymmetric biclustering. In Proc. of IEEE PacificVis. IEEE, pp. 305–309. 10.1109/PACIFICVIS.2017.8031609.
  126. Stolper CD, Perer A and Gotz D. Progressive visual analytics: User-driven visual exploration of in-progress analytics. IEEE TVCG 2014; 20(12): 1653–1662. 10.1109/TVCG.2014.2346574.
  127. Progressive data science: Potential and challenges. CoRR 2018; abs/1812.08032. 1812.08032.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets