Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization (2012.01205v4)

Published 2 Dec 2020 in cs.LG, cs.HC, and stat.ML

Abstract: During the training phase of ML models, it is usually necessary to configure several hyperparameters. This process is computationally intensive and requires an extensive search to infer the best hyperparameter set for the given problem. The challenge is exacerbated by the fact that most ML models are complex internally, and training involves trial-and-error processes that could remarkably affect the predictive result. Moreover, each hyperparameter of an ML algorithm is potentially intertwined with the others, and changing it might result in unforeseeable impacts on the remaining hyperparameters. Evolutionary optimization is a promising method to try and address those issues. According to this method, performant models are stored, while the remainder are improved through crossover and mutation processes inspired by genetic algorithms. We present VisEvol, a visual analytics tool that supports interactive exploration of hyperparameters and intervention in this evolutionary procedure. In summary, our proposed tool helps the user to generate new models through evolution and eventually explore powerful hyperparameter combinations in diverse regions of the extensive hyperparameter space. The outcome is a voting ensemble (with equal rights) that boosts the final predictive performance. The utility and applicability of VisEvol are demonstrated with two use cases and interviews with ML experts who evaluated the effectiveness of the tool.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (102)
  1. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2019), KDD ’19, ACM, pp. 2623–2631. doi:10.1145/3292500.3330701.
  2. AutoML — Google Cloud AutoML. Accessed February 26, 2021. URL: https://cloud.google.com/automl/.
  3. BayesOpt — Bayesian optimization for science and engineering. Accessed February 26, 2021. URL: https://bayesopt.github.io/.
  4. Bergstra J., Bengio Y.: Random search for hyper-parameter optimization. Journal of Machine Learning Research 13 (Feb. 2012), 281–305. doi:10.5555/2188385.2188395.
  5. Algorithms for hyper-parameter optimization. In Proceedings of the 24th International Conference on Neural Information Processing Systems (2011), NIPS ’11, Curran Associates Inc., pp. 2546–2554.
  6. Collaborative hyperparameter tuning. In Proceedings of Machine Learning Research (2013), vol. 28(2), PMLR, pp. 199–207.
  7. Automated optimized parameters for t-distributed stochastic neighbor embedding improve visualization and analysis of large datasets. Nature Communications 10, 5415 (2019). doi:10.1038/s41467-019-13055-y.
  8. Evaluation measures for models assessment over imbalanced data sets. Journal of Information Engineering and Applications 3, 10 (2013).
  9. Hyperopt: A Python library for model selection and hyperparameter optimization. Computational Science & Discovery 8, 1 (July 2015), 014008. doi:10.1088/1749-4699/8/1/014008.
  10. Breiman L.: Random forests. Machine Learning 45 (Oct. 2001), 5–32. doi:10.1023/A:1010933404324.
  11. LDA ensembles for interactive exploration and categorization of behaviors. IEEE Transactions on Visualization and Computer Graphics (2019). doi:10.1109/TVCG.2019.2904069.
  12. Claesen M., De Moor B.: Hyperparameter search in machine learning. In Proceedings of the 11th Metaheuristics International Conference (2015), MIC ’15.
  13. Chen T., Guestrin C.: XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016), KDD ’16, ACM, pp. 785–794. doi:10.1145/2939672.2939785.
  14. A paper tiger? An empirical analysis of majority voting. Journal of Corporate Finance 21 (2013), 119–135. doi:10.1016/j.jcorpfin.2013.01.002.
  15. Chicco D., Jurman G.: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21 (Jan. 2020), 6. doi:10.1186/s12864-019-6413-7.
  16. Cantu-Paz E., Kamath C.: An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 35, 5 (Oct. 2005), 915–927. doi:10.1109/TSMCB.2005.847740.
  17. The state of the art in enhancing trust in machine learning models with the use of visualizations. Computer Graphics Forum 39, 3 (June 2020), 713–756. doi:10.1111/cgf.14034.
  18. A survey of surveys on the use of visualization for interpreting machine learning models. Information Visualization 19, 3 (July 2020), 207–233. doi:10.1177/1473871620904671.
  19. t-viSNE: Interactive assessment and interpretation of t-SNE projections. IEEE Transactions on Visualization and Computer Graphics 26, 8 (Aug. 2020), 2696–2714. doi:10.1109/TVCG.2020.2986996.
  20. StackGenVis: Alignment of data, algorithms, and models for stacking ensemble learning using performance metrics. IEEE Transactions on Visualization and Computer Graphics (2021). doi:10.1109/TVCG.2020.3030352.
  21. Comet.ML — Build better models faster. Accessed February 26, 2021. URL: https://comet.ml/.
  22. Easy hyperparameter search using Optunity. ArXiv e-prints (Dec. 2014). arXiv:1412.1114.
  23. D3 — Data-driven documents, 2011. Accessed February 26, 2021. URL: https://d3js.org/.
  24. DataRobot — Empowering the human heroes of the intelligence revolution. Accessed February 26, 2021. URL: https://www.datarobot.com/.
  25. BEAMES: Interactive multi-model steering, selection, and inspection for regression tasks. IEEE Computer Graphics and Applications 39, 9 (Sept. 2019). doi:10.1109/MCG.2019.2922592.
  26. Genetic algorithms for hyperparameter optimization in predictive business process monitoring. Information Systems 74, Part 1 (May 2018), 67–83. doi:10.1016/j.is.2018.01.003.
  27. Davis J., Goadrich M.: The relationship between precision-recall and ROC curves. In Proceedings of the 23rd International Conference on Machine Learning (2006), ICML ’06, ACM, pp. 233–240. doi:10.1145/1143844.1143874.
  28. Dua D., Graff C.: UCI machine learning repository, 2017. Accessed February 26, 2021. URL: http://archive.ics.uci.edu/ml.
  29. Finding optimal neural network architecture using genetic algorithms. Research in Computing Science 27 (2007), 15–24.
  30. Feurer M., Hutter F.: Hyperparameter optimization. In Automated Machine Learning: Methods, Systems, Challenges. Springer International Publishing, 2019, pp. 3–33. doi:10.1007/978-3-030-05318-5_1.
  31. An experimental comparison of performance measures for classification. Pattern Recognition Letters 30, 1 (Jan. 2009), 27–38. doi:10.1016/j.patrec.2008.08.010.
  32. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of Machine Learning Research (2018), vol. 80, PMLR, pp. 1437–1446.
  33. Flask — A micro web framework written in Python, 2010. Accessed February 26, 2021. URL: https://flask.palletsprojects.com/.
  34. A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 14, 5 (Sept. 1999), 771–780.
  35. Google Vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2017), KDD ’17, ACM, pp. 1487–1495. doi:10.1145/3097983.3098043.
  36. Visual ensemble analysis to study the influence of hyper-parameters on training deep neural networks. In Proceedings of the EuroVis Workshop on Machine Learning Methods in Visualisation for Big Data (2019), MLVis ’19, The Eurographics Association. doi:10.2312/mlvis.20191160.
  37. Sequential model-based optimization for general algorithm configuration. In Proceedings of the International Conference on Learning and Intelligent Optimization (2011), LION ’11, Springer Berlin Heidelberg, pp. 507–523. doi:10.1007/978-3-642-25566-3_40.
  38. An efficient approach for assessing parameter importance in bayesian optimization. In Proceedings of the NIPS workshop on Bayesian Optimization in Theory and Practice (2013), BayesOpt ’13.
  39. An efficient approach for assessing hyperparameter importance. In Proceedings of Machine Learning Research (2014), vol. 32(1), PMLR, pp. 754–762.
  40. ParamILS: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research 36, 1 (Oct. 2009), 267–306. doi:10.1613/jair.2861.
  41. Inselberg A., Dimsdale B.: Parallel coordinates for visualizing multi-dimensional geometry. In Proceedings of the 5th International Conference on Computer Graphics (1987), CG ’87, Springer Japan, pp. 25–44. doi:10.1007/978-4-431-68057-4_3.
  42. Population based training of neural networks. ArXiv e-prints (Nov. 2017). arXiv:1711.09846.
  43. Visual analysis of the impact of neural network hyper-parameters. In Proceedings of the EGEV International Workshop on Machine Learning Methods in Visualisation for Big Data (2020), MLVis ’20, The Eurographics Association. doi:10.2312/mlvis.20201101.
  44. QIM: Quantifying hyperparameter importance for deep learning. In Proceedings of the IFIP International Conference on Network and Parallel Computing (2016), NPC 16, Springer International Publishing, pp. 180–188. doi:10.1007/978-3-319-47099-3_15.
  45. Kobak D., Berens P.: The art of using t-SNE for single-cell transcriptomics. Nature Communications 10, 5416 (Nov. 2019). doi:10.1038/s41467-019-13056-x.
  46. Kerren A., Egger T.: EAVis: A visualization tool for evolutionary algorithms. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (2005), VL/HCC ’05, IEEE, pp. 299–301. doi:10.1109/VLHCC.2005.33.
  47. Kerren A.: Improving strategy parameters of evolutionary computations with interactive coordinated views. In Proceedings of the IASTED International Conference on Visualization, Imaging, and Image Processing (2006), VIIP ’06, ACTA Press, pp. 88–93.
  48. Autotune: A derivative-free optimization framework for hyperparameter tuning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2018), KDD ’18, ACM, pp. 443–452. doi:10.1145/3219819.3219837.
  49. CHOPT: Automated hyperparameter optimization framework for cloud-based machine learning platforms. ArXiv e-prints (Oct. 2018). arXiv:1810.03527.
  50. LightGBM: A highly efficient gradient boosting decision tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems (2017), NIPS ’17, Curran Associates Inc., pp. 3149–3157.
  51. Kruskal J. B.: Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika 29, 1 (Mar. 1964), 1–27. doi:10.1007/BF02289565.
  52. HyperTuner: Visual analytics for hyperparameter tuning by professionals. In Proceedings of the IEEE VIS Workshop on Machine Learning from User Interaction for Visualization and Analytics (2018), MLUI ’18.
  53. Liu Y., Heer J.: Somewhere over the rainbow: An empirical assessment of quantitative colormaps. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (2018), CHI ’18, ACM, pp. 598:1–598:12. doi:10.1145/3173574.3174172.
  54. Latha C. B. C., Jeeva S. C.: Improving the accuracy of prediction of heart disease risk based on ensemble classification techniques. Informatics in Medicine Unlocked 16 (2019), 100203. doi:10.1016/j.imu.2019.100203.
  55. Hyperband: A novel bandit-based approach to hyperparameter optimization. Jounal of Machine Learning Research 18, 1 (Jan. 2017), 6765–6816.
  56. AUC: A misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography 17, 2 (Mar. 2008), 145–151. doi:10.1111/j.1466-8238.2007.00358.x.
  57. Tune: A research platform for distributed model selection and training. In Proceedings of the ICML/IJCAI-ECAI International Workshop on Automatic Machine Learning (2018), AutoML ’18.
  58. Auptimizer — An extensible, open-source framework for hyperparameter tuning. In Proceedings of the IEEE International Conference on Big Data (2019), Big Data ’19, IEEE, pp. 339–348. doi:10.1109/BigData47090.2019.9006330.
  59. Visual diagnosis of tree boosting methods. IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan. 2018), 163–173. doi:10.1109/TVCG.2017.2744378.
  60. UMAP: Uniform manifold approximation and projection for dimension reduction. ArXiv e-prints 1802.03426 (Feb. 2018). arXiv:1802.03426.
  61. Quantitative structure–activity relationship models for ready biodegradability of chemicals. Journal of Chemical Information and Modeling 53, 4 (2013), 867–878. doi:10.1021/ci4000213.
  62. Being accurate is not enough: How accuracy metrics have hurt recommender systems. In CHI ’06 Extended Abstracts on Human Factors in Computing Systems (2006), CHI EA ’06, ACM, pp. 1097–1101. doi:10.1145/1125451.1125659.
  63. Explaining vulnerabilities to adversarial machine learning through visual analytics. IEEE Transactions on Visualization and Computer Graphics 26, 1 (Jan. 2020), 1075–1085. doi:10.1109/TVCG.2019.2934631.
  64. NNI — Microsoft Neural Network Intelligence. Accessed February 26, 2021. URL: https://github.com/microsoft/nni.
  65. Neto M. P., Paulovich F. V.: Explainable Matrix — Visualization for global and local interpretability of random forest classification ensembles. IEEE Transactions on Visualization and Computer Graphics (2020). doi:10.1109/TVCG.2020.3030354.
  66. Tunability: Importance of hyperparameters of machine learning algorithms. Journal of Machine Learning Research 20, 53 (2019), 1–32.
  67. VisualHyperTuner: Visual analytics for user-driven hyperparameter tuning of deep neural networks. In Proceedings of the 2nd SysML Conference (2019), SysML 19.
  68. Plotly — JavaScript open source graphing library, 2010. Accessed February 26, 2021. URL: https://plotly.com.
  69. Pereira L., Nunes N.: A comparison of performance metrics for event classification in non-intrusive load monitoring. In Proceedings of the IEEE International Conference on Smart Grid Communications (2017), SmartGridComm ’17, IEEE, pp. 159–164. doi:10.1109/SmartGridComm.2017.8340682.
  70. HyperTendril: Visual analytics for user-driven hyperparameter optimization of deep neural networks. IEEE Transactions on Visualization & Computer Graphics (2021). doi:10.1109/TVCG.2020.3030380.
  71. Powers D. M. W.: Evaluation: From precision, recall and F-measure to ROC, informedness, markedness & correlation. Journal of Machine Learning Technologies 2, 1 (2011), 37–63.
  72. Scikit-Learn: Machine learning in Python. Journal of Machine Learning Research 12 (Nov. 2011), 2825–2830. doi:10.5555/1953048.2078195.
  73. ATM: A distributed, collaborative, scalable system for automated machine learning. In Proceedings of the IEEE International Conference on Big Data (2017), Big Data ’17, IEEE, pp. 151–162. doi:10.1109/BigData.2017.8257923.
  74. Integrating data and model space in ensemble learning by visual analytics. IEEE Transactions on Big Data (2018). doi:10.1109/TBDATA.2018.2877350.
  75. NSML: A machine learning platform that enables you to focus on your models. In Proceedings of the NIPS Workshop on ML Systems (2017), ML-Sys ’17.
  76. Sokolova M., Lapalme G.: A systematic analysis of performance measures for classification tasks. Information Processing & Management 45, 4 (July 2009), 427–437. doi:10.1016/j.ipm.2009.03.002.
  77. Practical Bayesian optimization of machine learning algorithms. In Proceedings of the 25th International Conference on Neural Information Processing Systems (2012), vol. 2 of NIPS ’12, Curran Associates Inc., pp. 2951–2959.
  78. Progressive visual analytics: User-driven visual exploration of in-progress analytics. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 1653–1662. doi:10.1109/TVCG.2014.2346574.
  79. Saito T., Rehmsmeier M.: The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE 10, 3 (Mar. 2015), e0118432. doi:10.1371/journal.pone.0118432.
  80. Sagi O., Rokach L.: Ensemble learning: A survey. WIREs Data Mining and Knowledge Discovery 8, 4 (July–Aug. 2018), e1249. doi:10.1002/widm.1249.
  81. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE 104, 1 (2016), 148–175. doi:10.1109/JPROC.2015.2494218.
  82. Sturm B. L.: Classification accuracy is not enough. Journal of Intelligent Information Systems 41, 3 (Dec. 2013), 371–406. doi:10.1007/s10844-013-0250-y.
  83. Takagi H.: Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation. Proceedings of the IEEE 89, 9 (Sept. 2001), 1275–1296. doi:10.1109/5.949485.
  84. Oríon: Experiment version control for efficient hyperparameter optimization. In Proceedings of the ICML Workshop on Reproducibility in Machine Learning (2018), RML ’18.
  85. Tharwat A.: Classification assessment methods. Applied Computing and Informatics (2018). doi:10.1016/j.aci.2018.08.003.
  86. Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2013), KDD ’13, ACM, pp. 847–855. doi:10.1145/2487575.2487629.
  87. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2009), CHI ’09, ACM, pp. 1283–1292. doi:10.1145/1518701.1518895.
  88. Runway: Machine learning model experiment management tool. In Proceedings of the 1st SysML Conference (2018), SysML ’18.
  89. Progressive data science: Potential and challenges. CoRR abs/1812.08032 (2018). arXiv:1812.08032.
  90. van der Maaten L., Hinton G.: Visualizing data using t-SNE. Journal of Machine Learning Research 9 (2008), 2579–2605.
  91. van Rijn J. N., Hutter F.: An empirical study of hyperparameter importance across datasets. In Proceedings of the ECML-PKDD International Workshop on Automatic Machine Learning (2017), AutoML ’17.
  92. van Rijn J. N., Hutter F.: Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2018), KDD ’18, ACM, pp. 2367–2376. doi:10.1145/3219819.3220058.
  93. Vue.js — The progressive JavaScript framework, 2014. Accessed February 26, 2021. URL: https://vuejs.org/.
  94. ATMSeer: Increasing transparency and controllability in automated machine learning. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019), CHI ’19, ACM, pp. 681:1–681:12. doi:10.1145/3290605.3300911.
  95. Wolpert D. H.: Stacked generalization. Neural Networks 5, 2 (1992), 241–259. doi:10.1016/S0893-6080(05)80023-1.
  96. AutoAI: Automating the end-to-end ai lifecycle with humans-in-the-loop. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion (2020), IUI ’20, ACM, pp. 77–78. doi:10.1145/3379336.3381474.
  97. AutoAIViz: Opening the blackbox of automated artificial intelligence with conditional parallel coordinates. In Proceedings of the 25th International Conference on Intelligent User Interfaces (2020), IUI ’20, ACM, pp. 308–312. doi:10.1145/3377325.3377538.
  98. EnsembleLens: Ensemble-based visual exploration of anomaly detection algorithms with multidimensional data. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan. 2019), 109–119. doi:10.1109/TVCG.2018.2864825.
  99. Yogatama D., Mann G.: Efficient transfer learning method for automatic hyperparameter tuning. In Proceedings of Machine Learning Research (Apr. 2014), vol. 33, PMLR, pp. 1077–1085.
  100. Optimizing deep learning hyper-parameters through an evolutionary algorithm. In Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments (2015), MLHPC ’15, ACM. doi:10.1145/2834892.2834896.
  101. iForest: Interpreting random forests via visual analytics. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan. 2019), 407–416. doi:10.1109/TVCG.2018.2864475.
  102. Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan. 2019), 364–373. doi:10.1109/TVCG.2018.2864499.
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets