Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MetaStackVis: Visually-Assisted Performance Evaluation of Metamodels (2212.03539v3)

Published 7 Dec 2022 in cs.LG, cs.HC, and stat.ML

Abstract: Stacking (or stacked generalization) is an ensemble learning method with one main distinctiveness from the rest: even though several base models are trained on the original data set, their predictions are further used as input data for one or more metamodels arranged in at least one extra layer. Composing a stack of models can produce high-performance outcomes, but it usually involves a trial-and-error process. Therefore, our previously developed visual analytics system, StackGenVis, was mainly designed to assist users in choosing a set of top-performing and diverse models by measuring their predictive performance. However, it only employs a single logistic regression metamodel. In this paper, we investigate the impact of alternative metamodels on the performance of stacking ensembles using a novel visualization tool, called MetaStackVis. Our interactive tool helps users to visually explore different singular and pairs of metamodels according to their predictive probabilities and multiple validation metrics, as well as their ability to predict specific problematic data instances. MetaStackVis was evaluated with a usage scenario based on a medical data set and via expert interviews.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (32)
  1. On the surprising behavior of distance metrics in high dimensional space. In Proceedings of the International Conference on Database Theory (ICDT), pp. 420–434. Springer Berlin Heidelberg, 2001.
  2. A paper tiger? An empirical analysis of majority voting. Journal of Corporate Finance, 21:119–135, 2013. doi: 10 . 1016/j . jcorpfin . 2013 . 01 . 002
  3. Density-based clustering based on hierarchical density estimates. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 160–172. Springer, 2013.
  4. Scatterplot matrix techniques for large N. Journal of the American Statistical Association, 82(398):424–436, 1987.
  5. The state of the art in enhancing trust in machine learning models with the use of visualizations. Computer Graphics Forum, 39(3):713–756, June 2020. doi: 10 . 1111/cgf . 14034
  6. Empirical study: Visual analytics for comparing stacking to blending ensemble learning. In Proceedings of the 23rd International Conference on Control Systems and Computer Science (CSCS), pp. 1–8. IEEE, 2021.
  7. StackGenVis: Alignment of data, algorithms, and models for stacking ensemble learning using performance metrics. IEEE Transactions on Visualization and Computer Graphics, 27(2):1547–1557, Feb. 2021. doi: 10 . 1109/TVCG . 2020 . 3030352
  8. VisEvol: Visual analytics to support hyperparameter search through evolutionary optimization. Computer Graphics Forum, 40(3):201–214, June 2021. doi: 10 . 1111/cgf . 14300
  9. FeatureEnVi: Visual analytics for feature engineering using stepwise selection and semi-automatic extraction approaches. IEEE Transactions on Visualization and Computer Graphics, 28(4):1773–1791, 2022. doi: 10 . 1109/TVCG . 2022 . 3141040
  10. D. Dua and C. Graff. UCI Machine Learning Repository, 2017.
  11. M. Feurer and F. Hutter. Hyperparameter optimization. In Automated Machine Learning: Methods, Systems, Challenges, pp. 3–33. Springer International Publishing, 2019. doi: 10 . 1007/978-3-030-05318-5_1
  12. Jupyter — A web-based interactive computing platform. https://jupyter.org, 2014. Accessed December 10, 2022.
  13. R. Kohavi and D. Wolpert. Bias plus variance decomposition for zero-one loss functions. In Proceedings of the International Conference on Machine Learning, ICML ’96, pp. 275–283. Morgan Kaufmann Publishers Inc., 1996.
  14. Improving the accuracy of prediction of heart disease risk based on ensemble classification techniques. Informatics in Medicine Unlocked, 16:100203, 2019. doi: 10 . 1016/j . imu . 2019 . 100203
  15. P4: Portable parallel processing pipelines for interactive information visualization. IEEE Transactions on Visualization and Computer Graphics, 26(3):1548–1561, 2020. doi: 10 . 1109/TVCG . 2018 . 2871139
  16. Y. Liu and J. Heer. Somewhere over the rainbow: An empirical assessment of quantitative colormaps. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pp. 598:1–598:12. ACM, 2018. doi: 10 . 1145/3173574 . 3174172
  17. Explaining vulnerabilities to adversarial machine learning through visual analytics. IEEE Transactions on Visualization and Computer Graphics, 26(1):1075–1085, Jan. 2020. doi: 10 . 1109/TVCG . 2019 . 2934631
  18. UMAP: Uniform manifold approximation and projection for dimension reduction. ArXiv e-prints, 1802.03426, Feb. 2018.
  19. MetaStackVis code. http://bit.ly/MetaStackVis-code, 2022. Accessed December 10, 2022.
  20. Density-based clustering validation, pp. 839–847. 2014. doi: 10 . 1137/1 . 9781611973440 . 96
  21. Stacked generalization: An introduction to super learning. European Journal of Epidemiology, 33(5):459–464, 2018.
  22. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, Nov. 2011. doi: 10 . 5555/1953048 . 2078195
  23. Plotly — Python open source graphing library. https://plot.ly, 2013. Accessed December 10, 2022.
  24. O. Sagi and L. Rokach. Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery, 8(4):e1249, July–Aug. 2018. doi: 10 . 1002/widm . 1249
  25. Optimizing ensemble weights and hyperparameters of machine learning models for regression problems. Machine Learning with Applications, 7:100251, 2022. doi: 10 . 1016/j . mlwa . 2022 . 100251
  26. Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Annual Symposium Computer Application in Medical Care, pp. 261–265. American Medical Informatics Association, 1988.
  27. StackGenVis code. http://bit.ly/StackGenVis-code, 2021. Accessed December 10, 2022.
  28. Streamlit — The fastest way to build and share data apps. https://streamlit.io, 2020. Accessed December 10, 2022.
  29. Stacked generalization: When does it work? In Proceedings of the Fifteenth International Joint Conference on Artifical Intelligence — Volume 2, IJCAI ’97, pp. 866–871. Morgan Kaufmann Publishers Inc., 1997.
  30. D. H. Wolpert. Stacked generalization. Neural networks, 5(2):241–259, 1992.
  31. EnsembleLens: Ensemble-based visual exploration of anomaly detection algorithms with multidimensional data. IEEE Transactions on Visualization and Computer Graphics, 25(1):109–119, Jan. 2019. doi: 10 . 1109/TVCG . 2018 . 2864825
  32. SliceTeller : A data slice-driven approach for machine learning model validation. IEEE Transactions on Visualization and Computer Graphics, pp. 1–11, 2022. doi: 10 . 1109/TVCG . 2022 . 3209465

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets